clearpilot/CLAUDE.md

# ClearPilot — CLAUDE.md

## Project Overview

ClearPilot is a custom fork of **FrogPilot** (itself a fork of comma.ai's openpilot), purpose-built for Brian Hanson's **Hyundai Tucson** (HDA2 equipped). The vehicle's HDA2 system has specific quirks around how it synchronizes driving state with openpilot that require careful handling.

The fork was previously in a state where many features were layered on top but the driving model behavior had regressed in ways that couldn't be traced. On **2026-05-03** the working tree was reset back to a known-clean baseline so features can be re-introduced one at a time with proper testing.

## Current State (post-reset)

This tree currently contains:

- **The pristine pre-modification baseline** (commit `c2ab0fa`, "reset to pre-modification baseline").
- **Startup-chain customizations** (commit `5624898`): launch scripts, on_start/provision flow, OpenVPN auto-connect, nice-monitor, build helpers, custom logo + spinner, encrypted dev SSH keys.
- **Build/launch fixes** (commit `b287fd0`, tagged `working-baseline-2`): make baseline compile cleanly (QtWebEngine removed; `screenDisplayMode` reference fixed), and make `build_only.sh` exit on failure with a detached error window.

`build_only.sh` succeeds and `launch_openpilot.sh` boots the full manager process set.

## Where the Old Code Lives

| Location | What it is |
|---|---|
| `/data/openpilot/` | This repo. Working baseline + startup port. **Active.** |
| `/data/openpilot-broken-2026-05-03/` | Full snapshot (with `.git`) of the prior modified-but-broken tree. Reference for porting features. |
| `/data/clearpilot-baseline/` | The original baseline source we copied in. Kept for safety; do not modify. |
| `/data/openpilot-features-broken/` | Pre-existing snapshot from an earlier reset attempt — **unverified**, leave alone. |

| Tag | Commit | What |
|---|---|---|
| `pre-reset-2026-05-03` | `f7e602c` | Last commit of the broken-but-feature-complete tree. |
| `working-baseline-2` | `b287fd0` | Current head after the reset + startup port + compile fixes. |

Both tags are pushed to `origin/clearpilot`.

## Pending Feature Port Roadmap

Everything below is present in `/data/openpilot-broken-2026-05-03/` and **not** in this tree. Port them in small, testable batches — one feature area per commit, build + launch test in between.

**Driving behavior (HDA2 specifics):**
- Lateral disabled (car's radar cruise handles steering; openpilot longitudinal only)
- Brief disengage when turn signal is active during lane changes
- Driver-monitoring timeout adjustments
- Custom driving models in `selfdrive/clearpilot/models/`: `duck-amigo.thneed`, `farmville.onnx`, `wd-40.thneed`

**Onroad UI:**
- Onroad layout changes (number positions, sidebar hidden during drive)
- New ready/splash screen and home/offroad menu (the "ClearPilot menu" — sidebar settings panel replacing stock home; General / Network / Dashcam / Debug panels)
- Status window with live system stats (temp, fan, storage, RAM, WiFi, VPN, GPS, telemetry, dashcam)
- Crash handler in UI with stack-trace dump for SIGSEGV/SIGABRT
- Display modes via the LFA steering-wheel button (`ScreenDisplayMode`: auto-normal, nightrider, manual normal, screen off, manual nightrider) — including auto day/night switching driven by GPS sunrise/sunset

**Speed / cruise logic:**
- `selfdrive/clearpilot/speed_logic.py` — speed-limit display, cruise-vs-limit warning signs (different thresholds above/below 50 mph), ding sound on warning transitions
- New params: `ClearpilotSpeedDisplay`, `ClearpilotSpeedLimitDisplay`, `ClearpilotCruiseWarning`, `ClearpilotPlayDing`, etc.

**Dashcam:**
- `selfdrive/clearpilot/dashcamd` (native C++) — VisionIPC frames → OMX H.264 hardware encoder, 3-min MP4 segments + SRT GPS subtitles in `/data/media/0/videos/`
- Trip lifecycle (waits for time + GPS + drive gear; closes on park/ignition off)
- `system/loggerd/deleter.py` trip-aware storage rotation
- Disables `encoderd` / `stream_encoderd`; reuses upstream `omx_encoder.cc`

**GPS:**
- `system/clearpilot/gpsd.py` — replacement for broken `qcomgpsd` diag interface; polls Quectel modem via `mmcli` AT commands at 1Hz, publishes `gpsLocation`
- NOAA solar-position calc for sunrise/sunset (drives display auto day/night)

**Telemetry:**
- `selfdrive/clearpilot/telemetry.py` (client) + `telemetryd.py` (collector) — diff-based CSV logger over ZMQ
- Toggleable via `TelemetryEnabled` memory param from Debug panel
- Auto-disabled if `/data` free < 5 GB; auto-disabled on every manager start
- Hyundai CAN-FD data logged from `selfdrive/car/hyundai/carstate.py update_canfd()`

**Bench mode (UI testing without a car):**
- `--bench` flag → `BENCH_MODE=1` → enables `bench_onroad.py`, blocks real car processes
- `bench_cmd.py` for setting fake vehicle state via params
- UI introspection RPC at `ipc:///tmp/clearpilot_ui_rpc` (widget-tree dump)

**Power/thermal:**
- Standstill power saving: `modeld` and `dmonitoringmodeld` throttled to 1fps when stopped
- Fan controller uses offroad clamps at standstill
- Park CPU savings + virtual battery shutdown fix

**Memory params (`/dev/shm/params`):**
Lots of new keys for runtime UI state — `TelemetryEnabled`, `VpnEnabled`, `ModelStandby`, `ScreenDisplayMode`, `DashcamState`, `DashcamFrames`, `DashcamShutdown`, `LogDirInitialized`, plus the speed/cruise display set.

**Session logging:**
- `/data/log2/current/` per-process stderr capture; aggregate `session.log` of major events
- Time-resolved log dir rename via GPS/NTP; 30-day rotation
- See `selfdrive/manager/process.py` and `manager.py` changes

**Already in this tree (just listing for reference, do NOT re-port):**
- `system/clearpilot/vpn-monitor.sh` + `vpn.ovpn` — OpenVPN auto-connect to `vpn.hanson.xyz`
- `system/clearpilot/nice-monitor.sh`
- `system/clearpilot/provision.sh` (apt installs, Claude Code installer, git remote fix, fast-forward)
- `system/clearpilot/on_start.sh` (SSH keys, ssh.service, git.hanson.xyz Host config, WiFi radio on)
- `system/clearpilot/dev/id_ed25519.{cpt,pub.cpt}` (DongleId-keyed)
- `system/clearpilot/startup_logo/bg.jpg` + scripts
- `selfdrive/ui/qt/spinner` + `spinner.cc/.h` (custom logo)
- `build_only.sh`, `build_preflight.sh`

## Working Rules

### CRITICAL: Change Control

This is self-driving software. All changes must be deliberate and well-understood.

- **NEVER make changes outside of what is explicitly requested**
- **Always explain proposed changes first** — describe the change, the logic, and the architecture; let Brian review and approve before writing any code
- **Brian is an expert on this software** — do not override his judgment or assume responsibility for changes he doesn't understand
- **Every line must be understood** — work slowly and deliberately
- **Test everything thoroughly** — Brian must always be in the loop
- **Do not refactor, clean up, or "improve" code beyond the specific request**

### Logging

**NEVER use `cloudlog`.** It's comma.ai's cloud telemetry pipeline, not ours — writes go to a publisher that's effectively a black hole for us (and the only thing it could do if ever reachable is bother the upstream FrogPilot developer). Our changes must always use **file logging** instead.

Use `print(..., file=sys.stderr, flush=True)`. `manager.py` redirects each managed process's stderr to `/data/log2/current/{process}.log` (once that feature is re-ported), so these lines land in the per-process log we already grep. Prefix custom log lines with `CLP ` so they're easy to filter out from upstream noise.

Example:
```python
import sys
print(f"CLP frogpilotPlan valid=False: carState_freq_ok={sm.freq_ok['carState']}", file=sys.stderr, flush=True)
```

Do not use `cloudlog.warning`, `cloudlog.info`, `cloudlog.error`, `cloudlog.event`, or `cloudlog.exception` in any CLEARPILOT-added code. Existing upstream/FrogPilot `cloudlog` calls can stay untouched.

### File Ownership

We operate as `root` on this device, but openpilot runs as the `comma` user (uid=1000, gid=1000). After any code changes that touch multiple files or before testing:

```bash
chown -R comma:comma /data/openpilot
```

### Git

- Remote: `git@git.hanson.xyz:brianhansonxyz/clearpilot.git`
- Branch: `clearpilot`
- Large model files are tracked in git (intentional — this is a backup)
- The `clearpilot` branch was force-pushed on 2026-05-03 as part of the reset; the prior history is reachable via the `pre-reset-2026-05-03` tag.

### Samba Share

- Share name: `openpilot` (e.g. `\\comma-3889765b\openpilot`)
- Path: `/data/openpilot`
- Username: `comma`
- Password: `i-like-to-drive-cars`
- Runs as `comma:comma` via force user/group — files created over SMB are owned correctly
- Enabled at boot (`smbd` + `nmbd`)

### Testing Changes

Use `build_only.sh` to compile, then start the manager separately. Never compile individual targets with scons directly — always use the full build script. Always start the manager after a successful build — don't wait for the user to ask.

```bash
# 1. Fix ownership
chown -R comma:comma /data/openpilot

# 2. Build (kills running manager, removes prebuilt, compiles, exits)
#    build_only.sh tees output to /tmp/build.log and propagates the build's
#    exit code via PIPESTATUS. On failure: error text window stays on screen
#    fully detached; the script exits non-zero and stderr has the compile error.
su - comma -c "bash /data/openpilot/build_only.sh"

# 3. If build succeeded ($? == 0), start openpilot
su - comma -c "bash /data/openpilot/launch_openpilot.sh"

# 4. Inspect logs
ls /data/log2/current/
cat /data/log2/current/session.log
```

### Adding New Params

The params system uses a C++ whitelist. Adding a new param name without registering it will crash with `UnknownKeyName`. To add one:

1. Register the key in `common/params.cc` (alphabetically, with `PERSISTENT` or `CLEAR_ON_*` flag)
2. Set the default in `selfdrive/manager/manager.py` `manager_init()`
3. Remove `prebuilt`, `common/params.o`, and `common/libcommon.a` to force rebuild

### Memory Params (paramsMemory)

Once re-ported, ClearPilot will use memory params (`/dev/shm/params/d/`) for UI toggles that should reset on boot. Conventions:

- **Registration**: register in `common/params.cc` as `PERSISTENT` (the registration flag does NOT control which path the param lives at)
- **C++ access**: `Params{"/dev/shm/params"}` — the Params class appends `/d/` internally, so `Params("/dev/shm/params/d")` would resolve to `/dev/shm/params/d/d/`
- **Python access**: `Params("/dev/shm/params")`
- **UI toggles**: use `ToggleControl` with manual `toggleFlipped` lambda, not `ParamControl` (which only handles persistent params)
- **IMPORTANT — method names differ between C++ and Python**: C++ uses camelCase (`putBool`, `getBool`, `getInt`), Python uses snake_case (`put_bool`, `get_bool`, `get_int`). This is a common source of silent failures.

### Changing a Service's Publish Rate

SubMaster's `freq_ok` check requires observed rate to fall within `[0.8 × min_freq, 1.2 × max_freq]` of the value declared in `cereal/services.py`. Publishing *faster* than declared trips `commIssue` just as surely as too slow. If you change how often a process publishes, update the rate in `cereal/services.py` to match.

## Device: comma 3x

- Qualcomm Snapdragon SoC (aarch64), serial `comma-3889765b`
- Storage: WDC SDINDDH4-128G, 128 GB UFS 2.1
- Ubuntu 20.04.6 LTS on AGNOS 9.7
- Kernel 4.9.103+ (custom comma.ai PREEMPT build, vendor-patched Qualcomm)
- Python 3.11.4 via pyenv at `/usr/local/pyenv/versions/3.11.4/` (system python 3.8 — do not use)
- Display: Weston (Wayland) on tty1
- Hardware encoding: OMX (`OMX.qcom.video.encoder.avc` / `.hevc`); V4L2 VIDC exists but is not usable from ffmpeg subprocess

### Filesystem mount quirks

| Mount | Device | Type | Notes |
|---|---|---|---|
| `/` | /dev/sda7 | ext4 | rw |
| `/data` | /dev/sda12 | ext4 | **persistent** — openpilot lives here |
| `/home` | overlay | overlayfs | **volatile** (upper on tmpfs) — changes lost on reboot |
| `/tmp` | tmpfs | tmpfs | volatile |
| `/persist` | /dev/sda2 | ext4 | persistent config/certs, noexec |
| `/dsp` | /dev/sde26 | ext4 | **read-only** Qualcomm DSP firmware |
| `/firmware` | /dev/sde4 | vfat | **read-only** firmware blobs |

### GPS

The device has **no u-blox chip** (`/dev/ttyHS0` does not exist). GPS is the **Quectel EC25 LTE modem**'s built-in GPS, accessed via AT commands through `mmcli`. The original `qcomgpsd` is broken on this device because the diag interface hangs after setup. Once re-ported, `system/clearpilot/gpsd.py` replaces it.

## Boot Sequence

```
Power On
  → systemd: comma.service (runs as comma user)
    → /usr/comma/comma.sh (waits for Weston, handles factory reset)
      → /data/continue.sh
        → /data/openpilot/launch_openpilot.sh
          → kill stale instances (launch_openpilot, launch_chffrplus, manager.py, ./ui, selfdrive/ui/text)
          → bash system/clearpilot/on_start.sh  (SSH, WiFi, run provision.sh)
          → background system/clearpilot/vpn-monitor.sh
          → background system/clearpilot/nice-monitor.sh
          → exec ./launch_chffrplus.sh
            → source launch_env.sh
            → run agnos_init
            → set PYTHONPATH
            → if no `prebuilt`: run build.py (spinner + scons)
            → exec selfdrive/manager/manager.py
              → manager_init() sets default params
              → ensure_running() loop starts managed processes
```