clearpilot/CLAUDE.md

# ClearPilot — CLAUDE.md

## Project Overview

ClearPilot is a custom fork of **FrogPilot** (itself a fork of comma.ai's openpilot), based on a 2024 release. It is purpose-built for Brian Hanson's **Hyundai Tucson** (HDA2 equipped). The vehicle's HDA2 system has specific quirks around how it synchronizes driving state with openpilot that require careful handling.

### Key Customizations in This Fork

- **UI changes** to the onroad driving interface
- **Lane change behavior**: brief disengage when turn signal is active during lane changes
- **Lateral control disabled**: the car's own radar cruise control handles lateral; openpilot handles longitudinal only
- **Driver monitoring timeouts**: modified safety timeouts for the driver monitoring system
- **Custom driving models**: `duck-amigo.thneed`, `farmville.onnx`, `wd-40.thneed` in `selfdrive/clearpilot/models/`
- **ClearPilot service**: Node.js service at `selfdrive/clearpilot/` with behavior scripts for lane change and longitudinal control
- **Native dashcamd**: C++ process capturing raw camera frames via VisionIPC with OMX H.264 hardware encoding
- **Standstill power saving**: model inference throttled to 1fps and fan quieted when car is stopped
- **ClearPilot menu**: sidebar settings panel replacing stock home screen (Home, Dashcam, Debug panels)
- **Status window**: live system stats (temp, fan, storage, RAM, WiFi, VPN, GPS, telemetry, dashcam status)
- **Debug button (LFA)**: steering wheel button repurposed for screen toggle and future UI actions
- **Telemetry system**: diff-based CSV logger via ZMQ IPC, toggleable from Debug panel
- **Bench mode**: `--bench` flag for onroad UI testing without car connected
- **GPS**: custom AT-command based GPS daemon (`system/clearpilot/gpsd.py`) replacing broken qcomgpsd diag interface
- **OpenVPN tunnel**: auto-connecting VPN to expose device on remote network for SSH access

See `GOALS.md` for feature roadmap.

## Working Rules

### CRITICAL: Change Control

This is self-driving software. All changes must be deliberate and well-understood.

- **NEVER make changes outside of what is explicitly requested**
- **Always explain proposed changes first** — describe the change, the logic, and the architecture; let Brian review and approve before writing any code
- **Brian is an expert on this software** — do not override his judgment or assume responsibility for changes he doesn't understand
- **Every line must be understood** — work slowly and deliberately
- **Test everything thoroughly** — Brian must always be in the loop
- **Do not refactor, clean up, or "improve" code beyond the specific request**

### File Ownership

We operate as `root` on this device, but openpilot runs as the `comma` user (uid=1000, gid=1000). After any code changes that touch multiple files or before testing:

```bash
chown -R comma:comma /data/openpilot
```

### Git

- Remote: `git@git.hanson.xyz:brianhansonxyz/clearpilot.git`
- Branch: `clearpilot`
- Large model files are tracked in git (intentional — this is a backup)

### Samba Share

- Share name: `openpilot` (e.g. `\\comma-3889765b\openpilot`)
- Path: `/data/openpilot`
- Username: `comma`
- Password: `i-like-to-drive-cars`
- Runs as `comma:comma` via force user/group — files created over SMB are owned correctly
- Enabled at boot (`smbd` + `nmbd`)

### Testing Changes

Always use `build_only.sh` to compile, then start the manager separately. Never compile individual targets with scons directly — always use the full build script. Always use full paths with `su - comma` — the login shell drops into `/home/comma` (volatile tmpfs), not `/data/openpilot`. **Always start the manager after a successful build** — don't wait for the user to ask.

```bash
# 1. Fix ownership (we edit as root, openpilot runs as comma)
chown -R comma:comma /data/openpilot

# 2. Build (kills running manager, removes prebuilt, compiles, exits)
#    Shows progress spinner on screen. On failure, shows error on screen
#    and prints to stderr. Does NOT start the manager.
su - comma -c "bash /data/openpilot/build_only.sh"

# 3. If build succeeded ($? == 0), start openpilot
su - comma -c "bash /data/openpilot/launch_openpilot.sh"

# 4. Review the aggregate session log for errors
cat /data/log2/current/session.log

# 5. Check per-process stdout/stderr logs if needed
ls /data/log2/current/
cat /data/log2/current/gpsd.log
```

### Changing a Service's Publish Rate

SubMaster's `freq_ok` check requires observed rate to fall within
`[0.8 × min_freq, 1.2 × max_freq]` of the value declared in
`cereal/services.py`. Publishing *faster* than declared fails the check
just as surely as publishing too slowly — it trips `all_freq_ok()` →
`EventName.commIssue` → "TAKE CONTROL IMMEDIATELY / Communication Issue
Between Processes".

If you change how often a process publishes, you **must** update the rate
in `cereal/services.py` to match. Example (real bug we hit):

- Bumped thermald from 2Hz → 4Hz (DT_TRML 0.5→0.25) for faster fan control
- Bumped gpsd from 1Hz → 2Hz
- manager publishes `managerState` gated by `sm.update()` returning on
  `deviceState`, so it picks up thermald's rate automatically

`services.py` still declared `deviceState` and `managerState` as 2Hz
(ceiling 2.4Hz) and `gpsLocation` as 1Hz (ceiling 1.2Hz), so controlsd
fired `commIssue` on every cycle the moment any of these arrived at the
new faster rate.

Fix: update the second tuple element (Hz) in `cereal/services.py` for the
affected service. Cereal will use the updated rate for all consumers.

### Adding New Params

The params system uses a C++ whitelist. Adding a new param name in `manager.py` alone will crash with `UnknownKeyName`. You must:

1. Register the key in `common/params.cc` (alphabetically, with `PERSISTENT` or `CLEAR_ON_*` flag)
2. Add the default value in `selfdrive/manager/manager.py` in `manager_init()`
3. Remove `prebuilt`, `common/params.o`, and `common/libcommon.a` to force rebuild

### Memory Params (paramsMemory)

ClearPilot uses memory params (`/dev/shm/params/d/`) for UI toggles that should reset on boot. Key rules:

- **Registration**: Register the key in `common/params.cc` as `PERSISTENT` (same as FrogPilot pattern — the registration flag does NOT control which path the param lives at)
- **C++ access**: Use `Params{"/dev/shm/params"}` — stored as a class member `paramsMemory` in onroad.h. Do NOT use `Params("/dev/shm/params/d")` — the Params class appends `/d/` internally, so that would resolve to `/dev/shm/params/d/d/`
- **Python access**: Use `Params("/dev/shm/params")`
- **Defaults**: Set in `manager_init()` via `Params("/dev/shm/params").put(key, value)`
- **UI toggles**: Use `ToggleControl` with manual `toggleFlipped` lambda that writes via `Params("/dev/shm/params")`. Do NOT use `ParamControl` for memory params — it reads/writes persistent params only
- **Current memory params**: `TelemetryEnabled` (default "0"), `VpnEnabled` (default "1"), `ModelStandby` (default "0"), `ScreenDisplayMode`, `DashcamState` (default "stopped"), `DashcamFrames` (default "0"), `DashcamShutdown` (default "0")
- **IMPORTANT — method names differ between C++ and Python**: C++ uses camelCase (`putBool`, `getBool`, `getInt`), Python uses snake_case (`put_bool`, `get_bool`, `get_int`). This is a common source of silent failures — the wrong casing compiles/runs but doesn't work.

### Building Native (C++) Processes

- SCons is the build system. Static libraries (`common`, `messaging`, `cereal`, `visionipc`) must be imported as SCons objects, not `-l` flags
- The `--as-needed` linker flag can cause link order issues with static libs — disable it in your SConscript if needed
- OMX encoder object (`omx_encoder.o`) is compiled by the UI build — reference the pre-built `.o` file rather than recompiling (avoids "two environments" scons error)
- `prebuilt` is recreated after every successful build — always remove it before rebuilding

## Bench Mode (Onroad UI Testing)

Bench mode allows testing the onroad UI without a car connected. It runs a fake vehicle simulator (`bench_onroad.py`) as a managed process and disables real car processes (pandad, thermald, controlsd, etc.).

### Usage

**IMPORTANT**: Do NOT use `echo` to write bench params — `echo` appends a newline which causes param parsing to fail silently (e.g. gear stays in park). Always use the `bench_cmd.py` tool.

```bash
# 1. Build first
chown -R comma:comma /data/openpilot
su - comma -c "bash /data/openpilot/build_only.sh"

# 2. Start in bench mode
su - comma -c "bash /data/openpilot/launch_openpilot.sh --bench"

# 3. Wait for UI to be ready (polls RPC every 1s, up to 20s)
su - comma -c "PYTHONPATH=/data/openpilot python3 -m selfdrive.clearpilot.bench_cmd wait_ready"

# 4. Control vehicle state
su - comma -c "PYTHONPATH=/data/openpilot python3 -m selfdrive.clearpilot.bench_cmd gear D"
su - comma -c "PYTHONPATH=/data/openpilot python3 -m selfdrive.clearpilot.bench_cmd speed 20"
su - comma -c "PYTHONPATH=/data/openpilot python3 -m selfdrive.clearpilot.bench_cmd speedlimit 45"
su - comma -c "PYTHONPATH=/data/openpilot python3 -m selfdrive.clearpilot.bench_cmd cruise 55"
su - comma -c "PYTHONPATH=/data/openpilot python3 -m selfdrive.clearpilot.bench_cmd engaged 1"

# 5. Inspect UI widget tree (RPC call, instant response)
su - comma -c "PYTHONPATH=/data/openpilot python3 -m selfdrive.clearpilot.bench_cmd dump"
```

### Debugging Crashes

The UI has a SIGSEGV/SIGABRT crash handler (`selfdrive/ui/main.cc`) that prints a stack trace to stderr, captured in the per-process log:

```bash
# Check for crash traces (use /data/log2/current which is always the active session)
grep -A 30 "CRASH" /data/log2/current/ui.log

# Resolve addresses to source lines
addr2line -e /data/openpilot/selfdrive/ui/ui -f 0xADDRESS

# bench_cmd dump detects crash loops automatically:
# if UI process uptime < 5s, it reports "likely crash looping"

# Check per-process logs
ls /data/log2/current/
cat /data/log2/current/session.log
cat /data/log2/current/gpsd.log
```

### UI Introspection RPC

The UI process runs a ZMQ REP server at `ipc:///tmp/clearpilot_ui_rpc`. Send `"dump"` to get a recursive widget tree showing class name, visibility, geometry, and stacked layout current indices. This is the primary debugging tool for understanding what the UI is displaying.

- `bench_cmd dump` — queries the RPC and prints the widget tree
- `bench_cmd wait_ready` — polls the RPC every second until `ReadyWindow` is visible (up to 10s)
- `ui_dump.py` — standalone dump tool (same as `bench_cmd dump`)

### Architecture

- `launch_openpilot.sh --bench` sets `BENCH_MODE=1` env var
- `manager.py` reads `BENCH_MODE`, blocks real car processes, enables `bench_onroad` process
- `bench_onroad.py` publishes fake `pandaStates` (ignition=true), `carState`, `controlsState` — thermald reads the fake pandaStates to determine ignition and publishes `deviceState.started=true` on its own
- thermald and camerad run normally in bench mode (thermald manages CPU cores needed for camerad)
- Blocked processes in bench mode: pandad, controlsd, radard, plannerd, calibrationd, torqued, paramsd, locationd, sensord, ubloxd, pigeond, dmonitoringmodeld, dmonitoringd, modeld, soundd, loggerd, micd, dashcamd

### Key Files

| File | Role |
|------|------|
| `selfdrive/clearpilot/bench_onroad.py` | Fake vehicle state publisher |
| `selfdrive/clearpilot/bench_cmd.py` | Command tool for setting bench params and querying UI |
| `selfdrive/clearpilot/ui_dump.py` | Standalone UI widget tree dump |
| `selfdrive/manager/process_config.py` | Registers bench_onroad as managed process (enabled=BENCH_MODE) |
| `selfdrive/manager/manager.py` | Blocks conflicting processes in bench mode |
| `launch_openpilot.sh` | Accepts `--bench` flag, exports BENCH_MODE env var |
| `selfdrive/ui/qt/window.cc` | UI RPC server (`ipc:///tmp/clearpilot_ui_rpc`), widget tree dump |

### Resolved Issues

- **SIGSEGV in onroad view (fixed)**: `update_model()` in `ui.cc` dereferenced empty model position data when modeld wasn't running. Fixed by guarding against empty `plan_position.getX()`. The root cause was found using the crash handler + `addr2line`.
- **`showDriverView` overriding transitions (fixed)**: was forcing `slayout` to onroad/home every frame at 20Hz, overriding park/drive logic. Fixed to only act when not in started state.
- **Sidebar appearing during onroad transition (fixed)**: `MainWindow::closeSettings()` was re-enabling the sidebar. Fixed by not calling `closeSettings` during `offroadTransition`.

## Performance Profiling

Use `py-spy` to find CPU hotspots in any Python process. It's installed at `/home/comma/.local/bin/py-spy`. (If missing: `su - comma -c "/usr/local/pyenv/versions/3.11.4/bin/pip install py-spy"`.)

```bash
# Find the target pid
ps -eo pid,cmd | grep -E "selfdrive.controls.controlsd" | grep -v grep

# Record 10s of stacks at 200Hz, raw (folded) format
/home/comma/.local/bin/py-spy record -o /tmp/ctrl.txt --pid <PID> --duration 10 --rate 200 --format raw

# Aggregate: which line of step() is consuming the most samples
awk -F';' '{
  for(i=1;i<=NF;i++) if ($i ~ /step \(selfdrive\/controls\/controlsd.py/) step_line=i;
  if (step_line && step_line < NF) {
    n=split($NF, parts, " "); count=parts[n];
    caller = $(step_line+1);
    sum[caller] += count;
  }
  step_line=0;
} END { for (c in sum) printf "%6d  %s\n", sum[c], c }' /tmp/ctrl.txt | sort -rn | head -15

# Aggregate by a source file — shows hottest lines in that file
awk -F';' '{
  for(i=1;i<=NF;i++) if ($i ~ /carstate\.py:/) {
    match($i, /:[0-9]+/); ln = substr($i, RSTART+1, RLENGTH-1);
    n=split($NF, parts, " "); count=parts[n];
    sum[ln] += count;
  }
} END { for (l in sum) printf "%5d  line %s\n", sum[l], l }' /tmp/ctrl.txt | sort -rn | head -15

# Quick stack dump (single sample, no recording)
/home/comma/.local/bin/py-spy dump --pid <PID>
```

**Known performance trap — hot `Params` writes**: `Params.put()` does `mkstemp` + `fsync` + `flock` + `rename` + `fsync_dir`. At 100Hz even on tmpfs the `flock` contention is ruinous. Cache the last-written value and skip writes when unchanged. Found this pattern in `carstate.py` and `controlsd.py` — controlsd went from 69% → 28% CPU after gating writes.

## Session Logging

Per-process stderr and an aggregate event log are captured in `/data/log2/current/`.

### Log Directory

- `/data/log2/current/` is always the active session directory
- `init_log_dir()` is called once from `manager_init()` — creates a fresh `/data/log2/current/` real directory
- If a previous `current/` real directory exists (unresolved session), it's renamed using its mtime timestamp
- If a previous `current` symlink exists, it's removed
- Once system time is valid (GPS/NTP), the real directory is renamed to `/data/log2/YYYY-MM-DD-HH-MM-SS/` and `current` becomes a symlink to it
- `LogDirInitialized` param: `"0"` until time resolves, then `"1"`
- Open file handles survive the rename (same inode, same filesystem)
- Session directories older than 30 days are deleted on manager startup

### Per-Process Logs

- Every `PythonProcess` and `NativeProcess` has stderr redirected to `{name}.log` in the session directory
- `DaemonProcess` (athenad) redirects both stdout+stderr (existing behavior)
- Stderr is redirected via `os.dup2` inside the forked child process

### Aggregate Session Log (`session.log`)

A single `session.log` in each session directory records major events:

- Manager start/stop/crash
- Process starts, deaths (with exit codes), watchdog restarts
- Onroad/offroad transitions

### Key Files

| File | Role |
|------|------|
| `selfdrive/manager/process.py` | Log directory creation, stderr redirection, session_log logger |
| `selfdrive/manager/manager.py` | Log rotation cleanup, session event logging |
| `build_only.sh` | Build-only script (no manager start) |

## Dashcam (dashcamd)

### Architecture

`dashcamd` is a native C++ process that captures raw camera frames directly from `camerad` via VisionIPC and encodes to MP4 using the Qualcomm OMX H.264 hardware encoder. This replaces the earlier FrogPilot screen recorder approach (`QWidget::grab()` -> OMX).

- **Codec**: H.264 AVC (hardware accelerated via `OMX.qcom.video.encoder.avc`)
- **Resolution**: 1928x1208 (full camera resolution, no downscaling)
- **Bitrate**: 2500 kbps
- **Container**: MP4 (remuxed via libavformat) + SRT subtitle sidecar
- **Segment length**: 3 minutes per file
- **Save path**: `/data/media/0/videos/YYYYMMDD-HHMMSS/YYYYMMDD-HHMMSS.mp4` (trip directories)
- **GPS subtitles**: companion `.srt` file per segment with 1Hz entries (speed MPH, lat/lon, UTC timestamp)
- **Trip lifecycle**: waits in WAITING state until valid system time + GPS fix + car in drive; records until car parked 10 min or ignition off; then returns to WAITING
- **Graceful shutdown**: thermald sets `DashcamShutdown` param, dashcamd closes segment and acks within 15s
- **Storage**: ~56 MB per 3-minute segment at 2500 kbps (verified: actual bitrate ~2570 kbps)
- **Crash handler**: SIGSEGV/SIGABRT handler writes backtrace to `/tmp/dashcamd_crash.log`
- **Storage device**: WDC SDINDDH4-128G UFS 2.1 — automotive grade, ~384 TB write endurance, no concern for continuous writes

### OmxEncoder

The OMX encoder (`selfdrive/frogpilot/screenrecorder/omx_encoder.cc`) was ported from upstream FrogPilot with the following key properties:

- Each encoder instance calls `OMX_Init()` in constructor and `OMX_Deinit()` in destructor — manages its own OMX lifecycle
- Constructor takes 5 args: `(path, width, height, fps, bitrate)` — no h265/downscale params
- `encoder_close()` calls `av_write_trailer` + ffmpeg faststart remux (`-movflags +faststart`)
- Destructor has null guards and error handling on all OMX state transitions
- ClearPilot addition: `encode_frame_nv12()` for direct NV12 input (dashcamd), alongside original `encode_frame_rgba()` (screen recorder)

### Key Files

| File | Role |
|------|------|
| `selfdrive/clearpilot/dashcamd.cc` | Main dashcam process — VisionIPC -> OMX encoder |
| `selfdrive/clearpilot/SConscript` | Build config for dashcamd |
| `selfdrive/frogpilot/screenrecorder/omx_encoder.cc` | OMX encoder (upstream FrogPilot port + `encode_frame_nv12`) |
| `selfdrive/frogpilot/screenrecorder/omx_encoder.h` | Encoder header |
| `selfdrive/manager/process_config.py` | dashcamd registered as NativeProcess, camerad always_run, encoderd disabled |
| `system/loggerd/deleter.py` | Trip-aware storage rotation (oldest trip first, then segments within active trip) |

### Params

- `DashcamShutdown` (memory param) — set by thermald before power-off, dashcamd acks by clearing it
- `DashcamState` (memory param) — "stopped", "waiting", or "recording" — published every 5s
- `DashcamFrames` (memory param) — per-trip encoded frame count, resets each trip — published every 5s

## Standstill Power Saving

When `carState.standstill` is true:

- **modeld**: skips GPU inference on 19/20 frames (1fps vs 20fps), reports 0 frame drops to avoid triggering `modeldLagging` in controlsd. Runs full 20fps during calibration (`liveCalibration.calStatus != calibrated`)
- **dmonitoringmodeld**: same 1fps throttle, added `carState` subscription
- **Fan controller**: uses offroad clamps (0-30%) instead of onroad (30-100%) at standstill; thermal protection still active via feedforward if temp > 60°C

### Key Files

| File | Role |
|------|------|
| `selfdrive/modeld/modeld.py` | Standstill frame skip logic |
| `selfdrive/modeld/dmonitoringmodeld.py` | Standstill frame skip logic |
| `selfdrive/thermald/fan_controller.py` | Standstill-aware fan clamps |
| `selfdrive/thermald/thermald.py` | Passes standstill to fan controller via carState |

## Display Modes (LFA/LKAS Debug Button)

The Hyundai Tucson's LFA steering wheel button cycles through 5 display modes via `ScreenDisplayMode` param (`/dev/shm/params`, CLEAR_ON_MANAGER_START, default 0).

### Display States

| State | Name | Display | Camera | Path Drawing |
|-------|------|---------|--------|-------------|
| 0 | auto-normal | on | yes | filled (normal) |
| 1 | auto-nightrider | on | black | 2px outline only |
| 2 | normal (manual) | on | yes | filled (normal) |
| 3 | screen off | GPIO off | n/a | n/a |
| 4 | nightrider (manual) | on | black | 2px outline only |

### Auto Day/Night Switching

- `gpsd.py` computes `is_daylight(lat, lon, utc_dt)` using NOAA solar position algorithm
- First calculation immediately on GPS fix + valid clock, then every 30 seconds
- State 0 + sunset → auto-switch to 1; State 1 + sunrise → auto-switch to 0
- States 2/3/4 are never touched by auto logic
- `IsDaylight` param written to `/dev/shm/params` for reference

### Button Behavior

**Onroad (car in drive gear):** 0→4, 1→2, 2→3, 3→4, 4→2 (never back to auto via button)

**Not in drive (parked/off):** any except 3 → 3 (screen off), state 3 → 0 (auto-normal)

**Shift to drive from screen off:** auto-resets to mode 0 (auto-normal) via `controlsd`

**Shift to park from nightrider:** auto-switches to mode 3 (screen off) via `home.cc`

**Tap screen while screen off:** resets to mode 0 (auto-normal) via `window.cc` touch handler

### Nightrider Mode

- Camera feed suppressed (OpenGL clears to black instead of rendering camera texture)
- All HUD elements (speed, alerts, telemetry indicator) still visible
- Path/lane polygons drawn as 2px outlines only (no gradient fill)
- Lane lines, road edges, blind spot paths, adjacent paths all use outline rendering

### Signal Chain

```
Steering wheel LFA button press
  -> CAN-FD: cruise_btns_msg_canfd["LFA_BTN"]
  -> Edge detection → ButtonEvent(type=FrogPilotButtonType.lkas)
  -> controlsd.clearpilot_state_control()
     -> Reads ScreenDisplayMode, applies transition table based on driving_gear
     -> Writes new ScreenDisplayMode to /dev/shm/params
  -> UI reads ScreenDisplayMode in paintEvent() (for camera/nightrider)
     and drawHud() (for display power on/off)
```

### Key Files

| File | Role |
|------|------|
| `selfdrive/controls/controlsd.py` | Button state machine, writes ScreenDisplayMode |
| `selfdrive/ui/qt/onroad.cc` | Nightrider rendering, display power control |
| `selfdrive/ui/qt/home.cc` | Display power for park/splash state |
| `system/clearpilot/gpsd.py` | Sunset/sunrise calc, auto day/night transitions |
| `selfdrive/clearpilot/bench_cmd.py` | `debugbutton` command simulates button press |

## Screen Timeout / Display Power

Display power is managed by `Device::updateWakefulness()` in `selfdrive/ui/ui.cc`.

- **Ignition off (offroad)**: screen blanks after `ScreenTimeout` seconds (default 120) of no touch. Tapping wakes it.
- **Ignition on (onroad)**: screen stays on unconditionally — ignition=true short-circuits the timeout check.
- **ScreenDisplayMode 3 override**: `updateWakefulness` checks `ScreenDisplayMode` first — if mode 3, calls `setAwake(false)` unconditionally, preventing ignition-on from overriding screen-off.
- **Debug button (LFA)**: cycles through display modes including screen off (state 3).
- **Park transition**: always shows splash screen; if coming from nightrider mode, auto-switches to screen off (mode 3) via `home.cc`.
- **Touch wake**: tapping screen while in mode 3 resets to mode 0 via `window.cc` event filter.

## Offroad UI (ClearPilot Menu)

The offroad home screen (`selfdrive/ui/qt/home.cc`) is a sidebar settings panel replacing the stock home screen. The original system settings are no longer accessible — the ClearPilot menu is the only settings interface.

### Panels

- **General**: Status window, Reset Calibration, Shutdown Timer, Reboot/Soft Reboot/Power Off
- **Network**: WiFi management, tethering, roaming, hidden networks (APN settings removed)
- **Dashcam**: placeholder for future dashcam footage viewer
- **Debug**: Telemetry logging toggle

### Navigation

- Tapping the splash screen (ReadyWindow) opens the ClearPilot menu
- "← Back" button returns to splash or onroad view
- Sidebar with stock metrics (TEMP, VEHICLE, CONNECT) is hidden

## Device: comma 3x

### Hardware

- Qualcomm Snapdragon SoC (aarch64)
- Serial: comma-3889765b
- Storage: WDC SDINDDH4-128G, 128 GB UFS 2.1
- Connects to the car via comma panda (CAN bus interface)

### Operating System

- **Ubuntu 20.04.6 LTS (Focal Fossa)** on aarch64
- **Kernel**: 4.9.103+ (custom comma.ai PREEMPT build, Feb 2024) — very old, vendor-patched Qualcomm kernel
- **Python**: 3.11.4 via pyenv at `/usr/local/pyenv/versions/3.11.4/` (system python is 3.8, do not use)
- **AGNOS version**: 9.7 (comma's custom OS layer on top of Ubuntu)
- **Display server**: Weston (Wayland compositor) on tty1
- **SELinux**: mounted (enforcement status varies)

### Users

- `comma` (uid=1000) — the user openpilot runs as; member of root, sudo, disk, gpu, gpio groups
- `root` — what we SSH in as; files must be chowned back to comma before running openpilot

### Filesystem / Mount Quirks

| Mount       | Device      | Type    | Notes |
|-------------|-------------|---------|-------|
| `/`         | /dev/sda7   | ext4    | Root filesystem, read-write |
| `/data`     | /dev/sda12  | ext4    | **Persistent**. Openpilot lives here. Survives reboots. |
| `/home`     | overlay     | overlayfs | **VOLATILE** — upperdir on tmpfs, changes lost on reboot |
| `/tmp`      | tmpfs       | tmpfs   | Volatile, 150 MB |
| `/var`      | tmpfs       | tmpfs   | Volatile, 128 MB (fstab) / 1.5 GB (actual) |
| `/systemrw` | /dev/sda10  | ext4    | Writable system overlay, noexec |
| `/persist`  | /dev/sda2   | ext4    | Persistent config/certs, noexec |
| `/cache`    | /dev/sda11  | ext4    | Android-style cache partition |
| `/dsp`      | /dev/sde26  | ext4    | **Read-only** Qualcomm DSP firmware |
| `/firmware`  | /dev/sde4   | vfat    | **Read-only** firmware blobs |

### Hardware Encoding

- **OMX**: `OMX.qcom.video.encoder.avc` (H.264) and `OMX.qcom.video.encoder.hevc` — used by dashcamd and screen recorder
- **V4L2**: Qualcomm VIDC at `/dev/v4l/by-path/platform-aa00000.qcom_vidc-video-index1` — used by encoderd (now disabled). Not accessible from ffmpeg due to permission/driver issues
- **ffmpeg**: v4.2.2, has `h264_v4l2m2m` and `h264_omx` listed but neither works from ffmpeg subprocess (OMX port issues, V4L2 device not found). Use OMX directly via the C++ encoder

### Fan Control

Software-controlled via `thermald` -> `fan_controller.py` -> panda USB -> PWM. Target temp 70°C, PI+feedforward controller. See Standstill Power Saving section for standstill-aware clamps.

## Boot Sequence

```
Power On
  -> systemd: comma.service (runs as comma user)
    -> /usr/comma/comma.sh (waits for Weston, handles factory reset)
      -> /data/continue.sh (exec bridge to openpilot)
        -> /data/openpilot/launch_openpilot.sh
          -> Kills other instances of itself and manager.py
          -> Runs on_start.sh (logo, reverse SSH)
          -> exec launch_chffrplus.sh
            -> Sources launch_env.sh (thread counts, AGNOS_VERSION)
            -> Runs agnos_init (marks boot slot, GPU perms, checks OS update)
            -> Sets PYTHONPATH, symlinks /data/pythonpath
            -> Runs build.py if no `prebuilt` marker
            -> Launches selfdrive/manager/manager.py
              -> manager_init() sets default params
              -> ensure_running() loop starts all managed processes
```

## Openpilot Architecture

### Process Manager

`selfdrive/manager/manager.py` orchestrates all processes defined in `selfdrive/manager/process_config.py`.

### Always-Running Processes (offroad + onroad)

- `thermald` — thermal management and fan control
- `pandad` — panda CAN bus interface
- `ui` — Qt-based onroad/offroad UI
- `deleter` — storage cleanup (9 GB threshold)
- `statsd`, `timed`, `logmessaged`, `tombstoned` — telemetry/logging
- `manage_athenad` — comma cloud connectivity
- `fleet_manager`, `frogpilot_process` — FrogPilot additions

### Onroad-Only Processes (when driving)

- `controlsd` — main vehicle control loop
- `plannerd` — path planning
- `radard` — radar processing
- `modeld` — driving model inference (throttled to 1fps at standstill)
- `dmonitoringmodeld` — driver monitoring model (throttled to 1fps at standstill)
- `locationd`, `calibrationd`, `paramsd`, `torqued` — localization and calibration
- `sensord` — IMU/sensor data
- `soundd` — alert sounds
- `camerad` — camera capture
- `loggerd` — CAN/sensor log recording (video encoding disabled)

### ClearPilot Processes

- `dashcamd` — raw camera dashcam recording (always runs; manages its own trip lifecycle)

### GPS

- Device has **no u-blox chip** (`/dev/ttyHS0` does not exist) — `ubloxd`/`pigeond` never start
- GPS hardware is a **Quectel EC25 LTE modem** (USB, `lsusb: 2c7c:0125`) with built-in GPS
- GPS is accessed via AT commands through `mmcli`: `mmcli -m any --command='AT+QGPSLOC=2'`
- **`qcomgpsd`** (original openpilot process) uses the modem's diag interface which is broken on this device — the diag recv loop blocks forever after setup. Commented out.
- **`system/clearpilot/gpsd.py`** is the replacement — polls GPS via AT commands at 1Hz, publishes `gpsLocation` cereal messages
- GPS data flows: `gpsd` → `gpsLocation` → `locationd` → `liveLocationKalman` → `timed` (sets system clock)
- `locationd` checks `UbloxAvailable` param (false on this device) to subscribe to `gpsLocation` instead of `gpsLocationExternal`
- `mmcli` returns `response: '...'` wrapper — `at_cmd()` strips it before parsing (fixed)
- GPS antenna power must be enabled via GPIO: `gpio_set(GPIO.GNSS_PWR_EN, True)`
- System `/usr/sbin/gpsd` daemon may respawn and interfere — should be disabled or killed

### Telemetry

- **Client**: `selfdrive/clearpilot/telemetry.py` — `tlog(group, data)` sends JSON over ZMQ PUSH
- **Collector**: `selfdrive/clearpilot/telemetryd.py` — diffs against previous state, writes changed values to CSV
- **Toggle**: `TelemetryEnabled` memory param, controlled from Debug panel in ClearPilot menu
- **Auto-disable**: disabled on every manager start; disabled if `/data` free < 5GB
- **Hyundai CAN-FD data**: logged from `selfdrive/car/hyundai/carstate.py` `update_canfd()` — groups: `car`, `cruise`, `speed_limit`, `buttons`
- **GPS data**: logged directly by telemetryd via cereal `gpsLocation` subscription at 1Hz — group: `gps` (latitude, longitude, speed, altitude, bearing, accuracy)
- **CSV location**: `/data/log2/current/telemetry.csv` (or session directory)
- **Onroad overlay**: when telemetry enabled, status bar shows temp, fan %, model FPS, and STANDSTILL indicator
- **Speed limit**: processed by `selfdrive/clearpilot/speed_logic.py` (SpeedState class), converts m/s to display units, writes to memory params. Cruise warning signs appear when cruise set speed exceeds speed limit by threshold (10 mph if limit >= 50, 7 mph if < 50) or is 5+ mph under. Ding sound plays when warning sign appears or speed limit changes while visible (30s cooldown)

### Key Dependencies

- **Python 3.11** (via pyenv) with: numpy, casadi, onnx/onnxruntime, pycapnp, pyzmq, sentry-sdk, sympy, Cython
- **capnp (Cap'n Proto)** — IPC message serialization between all processes
- **ZeroMQ** — IPC transport layer
- **Qt 5** — UI framework (with WebEngine available but not used for rotation reasons)
- **OpenMAX (OMX)** — hardware video encoding
- **libavformat** — MP4 container muxing
- **libyuv** — color space conversion
- **SCons** — build system for native C++ components