brianhansonxyz/clearpilot
1
0
Fork 0
Code Issues Pull Requests Actions 4 Packages Projects Releases Wiki Activity

dashcamd v2: native C++ process with direct camera capture

Browse Source 
- New dashcamd: connects to camerad via VisionIPC, feeds raw NV12
  frames directly to OMX H.264 encoder. Full 1928x1208 resolution,
  4Mbps, 3-minute MP4 segments. Works regardless of UI state.
- Added encode_frame_nv12() to OmxEncoder — skips RGBA->NV12 conversion
- Suspends recording after 10 minutes of standstill
- Disabled old screen recorder timer in onroad.cc
- Suppress debug button alert (clpDebug event still fires for screen toggle)
- launch_openpilot.sh self-cleans other instances before starting
- Register DashcamDebug param in params.cc and manager.py
- Add dashcamd to build system (SConscript) and process_config
- Updated CLAUDE.md with all session changes
- Added GOALS.md feature roadmap

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Brian Hanson
2026-04-11 08:54:41 +00:00
parent ed9c14616a
commit 4afd25fb5b
15 changed files with 560 additions and 119 deletions
Download Patch File Download Diff File Expand all files Collapse all files

258
CLAUDE.md
View File

@@ -12,13 +12,12 @@ ClearPilot is a custom fork of **FrogPilot** (itself a fork of comma.ai's openpi
- **Driver monitoring timeouts**: modified safety timeouts for the driver monitoring system
- **Custom driving models**: `duck-amigo.thneed`, `farmville.onnx`, `wd-40.thneed` in `selfdrive/clearpilot/models/`
- **ClearPilot service**: Node.js service at `selfdrive/clearpilot/` with behavior scripts for lane change and longitudinal control
- **Native dashcamd**: C++ process capturing raw camera frames via VisionIPC with OMX H.264 hardware encoding
- **Standstill power saving**: model inference throttled to 1fps and fan quieted when car is stopped
- **Clean offroad UI**: grid launcher replacing stock home screen
- **Debug button (LFA)**: steering wheel button repurposed for screen toggle and future UI actions
### Short-Term Goals
- ~~Fix the dashcam/screen recorder feature~~ (done — see Dashcam section below)
- Fix GPS tracking feature
- Add a safe-speed-exceeded chime
- Implement interactions for the "debug function button"
See `GOALS.md` for feature roadmap.
## Working Rules
@@ -58,67 +57,142 @@ chown -R comma:comma /data/openpilot
### Testing Changes
To restart the full openpilot stack after making changes:
The launch script now self-cleans — it kills other instances of itself, `launch_chffrplus.sh`, and `manager.py` before starting. No need to manually kill first.
```bash
# Fix ownership first (we edit as root, openpilot runs as comma)
# Fix ownership (we edit as root, openpilot runs as comma)
chown -R comma:comma /data/openpilot
# Kill existing stack — use pgrep/kill to avoid pkill matching our own shell
kill $(pgrep -f 'python.*manager') 2>/dev/null; sleep 2
# Remove prebuilt to force recompilation (it is recreated after each successful build)
rm -f /data/openpilot/prebuilt
# Must use a login shell as comma — sudo -u won't set up the right Python/env
# Must use a login shell as comma — sudo -u won't set up the right Python/env (3.11 via pyenv)
su - comma -c "bash /data/openpilot/launch_openpilot.sh"
```
Note: `pkill -f manager` or `pkill -f launch_openpilot` will match the invoking shell's own command line and kill the launch process itself. Use `pgrep`+`kill` or run the kill as a separate step before launching.
### Adding New Params
**Adding new params**: The params system uses a C++ whitelist. Adding a new param name in `manager.py` alone will crash with `UnknownKeyName`. You must also register the key in `common/params.cc` in the key list (alphabetically, with `PERSISTENT` or `CLEAR_ON_*` flag), then rebuild. The rebuild will recompile `params.cc` -> `libcommon.a` and re-link all binaries that use it.
The params system uses a C++ whitelist. Adding a new param name in `manager.py` alone will crash with `UnknownKeyName`. You must:
The `prebuilt` marker file skips compilation. It is **recreated automatically** after each successful build, so you must remove it every time you want to recompile:
```bash
rm -f /data/openpilot/prebuilt
```
1. Register the key in `common/params.cc` (alphabetically, with `PERSISTENT` or `CLEAR_ON_*` flag)
2. Add the default value in `selfdrive/manager/manager.py` in `manager_init()`
3. Remove `prebuilt`, `common/params.o`, and `common/libcommon.a` to force rebuild
## Dashcam / Screen Recorder
### Building Native (C++) Processes
- SCons is the build system. Static libraries (`common`, `messaging`, `cereal`, `visionipc`) must be imported as SCons objects, not `-l` flags
- The `--as-needed` linker flag can cause link order issues with static libs — disable it in your SConscript if needed
- OMX encoder object (`omx_encoder.o`) is compiled by the UI build — reference the pre-built `.o` file rather than recompiling (avoids "two environments" scons error)
- `prebuilt` is recreated after every successful build — always remove it before rebuilding
## Dashcam (dashcamd)
### Architecture
The dashcam is the FrogPilot `ScreenRecorder` — it captures the onroad UI screen (with overlays) and encodes to MP4 using the Qualcomm OMX H.264 hardware encoder.
`dashcamd` is a native C++ process that captures raw camera frames directly from `camerad` via VisionIPC and encodes to MP4 using the Qualcomm OMX H.264 hardware encoder. This replaces the earlier FrogPilot screen recorder approach (`QWidget::grab()` -> OMX).
- **Codec**: H.264 AVC (hardware accelerated via `OMX.qcom.video.encoder.avc`)
- **Resolution**: 1440x720 (downscaled from 2160x1080)
- **Bitrate**: 2 Mbps
- **Container**: MP4
- **Resolution**: 1928x1208 (full camera resolution, no downscaling)
- **Bitrate**: 4 Mbps
- **Container**: MP4 (remuxed via libavformat)
- **Segment length**: 3 minutes per file
- **Save path**: `/data/media/0/videos/YYYYMMDD-HHMMSS.mp4`
- **Standstill timeout**: suspends recording after 10 minutes of standstill, resumes when car moves
- **Storage**: ~90 MB per 3-minute segment, ~43 hours of footage in 78 GB free space
- **Storage device**: WDC SDINDDH4-128G UFS 2.1 — automotive grade, ~384 TB write endurance, no concern for continuous writes
### Changes Made (2026-04-11)
### Key Differences from Old Screen Recorder
1. **Disabled comma training data video** (`selfdrive/manager/process_config.py`): commented out `encoderd` and `stream_encoderd`. CAN/sensor logs (`rlog`/`qlog`) are still recorded by `loggerd`.
2. **Re-enabled screen recorder** (`selfdrive/ui/qt/onroad.cc`): uncommented the `QTimer` that feeds frames to the encoder at UI_FREQ rate.
3. **Auto-start recording** (`selfdrive/frogpilot/screenrecorder/screenrecorder.cc`): modified `update_screen()` to automatically start recording when the car is on (`scene.started`) and stop when off. No button press needed.
4. **Hidden UI elements**: the record button is constructed but never made visible or added to layout. Recording is invisible to the driver.
5. **Debug flag** (`ScreenRecorderDebug` param): when set to `"1"`, recording starts even without a car connected. Used for bench testing. Read via `scene.screen_recorder_debug` in `ui.h`/`ui.cc`.
6. **Deleter updated** (`system/loggerd/deleter.py`): free space threshold raised from 5 GB to 9 GB. Oldest videos in `/data/media/0/videos/` are deleted first before falling back to log segments.
| | Old (screen recorder) | New (dashcamd) |
|---|---|---|
| Source | `QWidget::grab()` screen capture | Raw NV12 from VisionIPC |
| Resolution | 1440x720 | 1928x1208 |
| Works with screen off | No (needs visible widget) | Yes (independent of UI) |
| Process type | Part of UI process | Standalone native process |
| Encoder input | RGBA -> NV12 conversion | NV12 direct (added `encode_frame_nv12`) |
### Key Files
| File | Role |
|------|------|
| `selfdrive/frogpilot/screenrecorder/screenrecorder.cc` | Screen capture and auto-start/stop logic |
| `selfdrive/frogpilot/screenrecorder/omx_encoder.cc` | OMX H.264 hardware encoder wrapper |
| `selfdrive/ui/qt/onroad.cc` | Timer that drives frame capture |
| `selfdrive/ui/ui.h` | `screen_recorder_debug` scene flag |
| `selfdrive/ui/ui.cc` | Reads `ScreenRecorderDebug` param |
| `selfdrive/manager/manager.py` | Default params |
| `selfdrive/manager/process_config.py` | encoderd disabled here |
| `system/loggerd/deleter.py` | Storage rotation (9 GB threshold, videos + logs) |
| `selfdrive/clearpilot/dashcamd.cc` | Main dashcam process — VisionIPC -> OMX encoder |
| `selfdrive/clearpilot/SConscript` | Build config for dashcamd |
| `selfdrive/frogpilot/screenrecorder/omx_encoder.cc` | OMX encoder (added `encode_frame_nv12` method) |
| `selfdrive/frogpilot/screenrecorder/omx_encoder.h` | Encoder header |
| `selfdrive/manager/process_config.py` | dashcamd registered as NativeProcess, encoderd disabled |
| `system/loggerd/deleter.py` | Storage rotation (9 GB threshold, oldest videos deleted first) |
### Params
- `DashcamDebug` — when `"1"`, dashcamd runs even without car connected (for bench testing)
- `IsDriverViewEnabled` — must be `"1"` to start camerad on bench (no car ignition)
## Standstill Power Saving
When `carState.standstill` is true:
- **modeld**: skips GPU inference on 19/20 frames (1fps vs 20fps), reports 0 frame drops to avoid triggering `modeldLagging` in controlsd
- **dmonitoringmodeld**: same 1fps throttle, added `carState` subscription
- **Fan controller**: uses offroad clamps (0-30%) instead of onroad (30-100%) at standstill; thermal protection still active via feedforward if temp > 60°C
### Key Files
| File | Role |
|------|------|
| `selfdrive/modeld/modeld.py` | Standstill frame skip logic |
| `selfdrive/modeld/dmonitoringmodeld.py` | Standstill frame skip logic |
| `selfdrive/thermald/fan_controller.py` | Standstill-aware fan clamps |
| `selfdrive/thermald/thermald.py` | Passes standstill to fan controller via carState |
## Debug Function Button (LFA/LKAS Steering Wheel Button)
The Hyundai Tucson's LFA (Lane Following Assist) steering wheel button is repurposed as a general-purpose UI control button. It has no driving function in ClearPilot since lateral control is disabled.
### Signal Chain
```
Steering wheel LFA button press
-> CAN-FD message: cruise_btns_msg_canfd["LFA_BTN"]
[selfdrive/car/hyundai/carstate.py:332-339]
-> Edge detection: lkas_enabled vs lkas_previously_enabled
-> create_button_events() -> ButtonEvent(type=FrogPilotButtonType.lkas)
[selfdrive/car/hyundai/interface.py:168]
-> controlsd.update_clearpilot_events(CS)
[selfdrive/controls/controlsd.py:1235-1239]
-> events.add(EventName.clpDebug)
-> controlsd.clearpilot_state_control(CC, CS)
[selfdrive/controls/controlsd.py:1241-1258]
-> Toggles ScreenDisaplayMode param (0=on, 1=off) in /dev/shm/params
-> UI reads ScreenDisaplayMode in drawHud()
[selfdrive/ui/qt/onroad.cc:390-403]
-> mode=1 and no alert: Hardware::set_display_power(false)
-> mode=0 or alert visible: Hardware::set_display_power(true)
```
### Current Behavior
- Each press toggles the display on/off instantly (debug alert suppressed)
- `ScreenDisaplayMode` is in-memory params (`/dev/shm/params`), resets on reboot
- `max_display_mode = 1` — currently only two states (on/off); can be extended for future modes
### Key Files
| File | Role |
|------|------|
| `selfdrive/car/hyundai/carstate.py` | Reads LFA_BTN from CAN-FD |
| `selfdrive/car/hyundai/interface.py` | Creates ButtonEvent with FrogPilotButtonType.lkas |
| `selfdrive/controls/controlsd.py` | Fires clpDebug event, toggles ScreenDisaplayMode |
| `selfdrive/controls/lib/events.py` | clpDebug event definition (alert suppressed) |
| `selfdrive/ui/qt/onroad.cc` | Reads ScreenDisaplayMode, controls display power |
## Offroad UI
The offroad home screen (`selfdrive/ui/qt/home.cc`) was replaced with a clean grid launcher. Stock FrogPilot widgets (date, version, update/alert notifications) were removed.
- **Settings button**: opens the original comma/FrogPilot settings (backdoor to all original settings)
- **Dashcam button**: placeholder for future dashcam footage viewer
- Tapping the splash screen (ReadyWindow) goes directly to the grid launcher (no sidebar)
- Sidebar with metrics (TEMP, VEHICLE, CONNECT) is hidden but still accessible via settings path
## Device: comma 3x
@@ -126,12 +200,14 @@ The dashcam is the FrogPilot `ScreenRecorder` — it captures the onroad UI scre
- Qualcomm Snapdragon SoC (aarch64)
- Serial: comma-3889765b
- Storage: WDC SDINDDH4-128G, 128 GB UFS 2.1
- Connects to the car via comma panda (CAN bus interface)
### Operating System
- **Ubuntu 20.04.6 LTS (Focal Fossa)** on aarch64
- **Kernel**: 4.9.103+ (custom comma.ai PREEMPT build, Feb 2024) — very old, vendor-patched Qualcomm kernel
- **Python**: 3.11.4 via pyenv at `/usr/local/pyenv/versions/3.11.4/` (system python is 3.8, do not use)
- **AGNOS version**: 9.7 (comma's custom OS layer on top of Ubuntu)
- **Display server**: Weston (Wayland compositor) on tty1
- **SELinux**: mounted (enforcement status varies)
@@ -155,23 +231,16 @@ The dashcam is the FrogPilot `ScreenRecorder` — it captures the onroad UI scre
| `/cache` | /dev/sda11 | ext4 | Android-style cache partition |
| `/dsp` | /dev/sde26 | ext4 | **Read-only** Qualcomm DSP firmware |
| `/firmware` | /dev/sde4 | vfat | **Read-only** firmware blobs |
| `/data/media`| NVMe | auto | 69 GB dashcam video storage |
### Unusual Packages / Services
### Hardware Encoding
- `vnstat` — network traffic monitor
- `cdsprpcd`, `qseecomd` — Qualcomm DSP and secure execution daemons
- `tftp_server` — TFTP server running (Qualcomm firmware access)
- armhf multiarch libraries present (32-bit binary support)
- Reverse SSH tunnel running via screen (`/data/brian/reverse_ssh.sh`)
- `phantom_touch_logger.py`, `power_drop_monitor.py` — comma hardware monitors
- **OMX**: `OMX.qcom.video.encoder.avc` (H.264) and `OMX.qcom.video.encoder.hevc` — used by dashcamd and screen recorder
- **V4L2**: Qualcomm VIDC at `/dev/v4l/by-path/platform-aa00000.qcom_vidc-video-index1` — used by encoderd (now disabled). Not accessible from ffmpeg due to permission/driver issues
- **ffmpeg**: v4.2.2, has `h264_v4l2m2m` and `h264_omx` listed but neither works from ffmpeg subprocess (OMX port issues, V4L2 device not found). Use OMX directly via the C++ encoder
### Large Files on Device
### Fan Control
- `/data/media` — ~69 GB (dashcam video segments)
- `/data/scons_cache` — ~1.9 GB (build cache)
- `/data/safe_staging` — ~1.5 GB (OTA update staging)
- Model files in repo: ~238 MB total (see models section below)
Software-controlled via `thermald` -> `fan_controller.py` -> panda USB -> PWM. Target temp 70°C, PI+feedforward controller. See Standstill Power Saving section for standstill-aware clamps.
## Boot Sequence
@@ -181,13 +250,16 @@ Power On
-> /usr/comma/comma.sh (waits for Weston, handles factory reset)
-> /data/continue.sh (exec bridge to openpilot)
-> /data/openpilot/launch_openpilot.sh
-> Sources launch_env.sh (thread counts, AGNOS_VERSION)
-> Runs agnos_init (marks boot slot, GPU perms, checks OS update)
-> Sets PYTHONPATH, symlinks /data/pythonpath
-> Runs build.py if no `prebuilt` marker
-> Launches selfdrive/manager/manager.py
-> manager_init() sets default params
-> ensure_running() loop starts all managed processes
-> Kills other instances of itself and manager.py
-> Runs on_start.sh (logo, reverse SSH)
-> exec launch_chffrplus.sh
-> Sources launch_env.sh (thread counts, AGNOS_VERSION)
-> Runs agnos_init (marks boot slot, GPU perms, checks OS update)
-> Sets PYTHONPATH, symlinks /data/pythonpath
-> Runs build.py if no `prebuilt` marker
-> Launches selfdrive/manager/manager.py
-> manager_init() sets default params
-> ensure_running() loop starts all managed processes
```
## Openpilot Architecture
@@ -198,73 +270,43 @@ Power On
### Always-Running Processes (offroad + onroad)
- `thermald` — thermal management, high CPU (~5.6%)
- `thermald` — thermal management and fan control
- `pandad` — panda CAN bus interface
- `ui` — Qt-based onroad/offroad UI
- `deleter` — storage cleanup
- `deleter` — storage cleanup (9 GB threshold)
- `statsd`, `timed`, `logmessaged`, `tombstoned` — telemetry/logging
- `manage_athenad` — comma cloud connectivity
- `fleet_manager`, `frogpilot_process` — FrogPilot additions
### Onroad-Only Processes (when driving)
- `controlsd` — main vehicle control loop
- `plannerd` — path planning
- `radard` — radar processing
- `modeld` — driving model inference
- `dmonitoringmodeld` — driver monitoring model
- `locationd` — positioning/localization
- `calibrationd` — camera calibration
- `paramsd`, `torqued` — parameter estimation
- `modeld` — driving model inference (throttled to 1fps at standstill)
- `dmonitoringmodeld` — driver monitoring model (throttled to 1fps at standstill)
- `locationd`, `calibrationd`, `paramsd`, `torqued` — localization and calibration
- `sensord` — IMU/sensor data
- `soundd` — alert sounds
- `camerad` — camera capture
- `loggerd`, `encoderd` — video logging/encoding
- `boardd` — board communication
- `loggerd` — CAN/sensor log recording (video encoding disabled)
### ClearPilot Processes
- `dashcamd` — raw camera dashcam recording (runs onroad or with DashcamDebug flag)
### GPS
- `ubloxd` + `pigeond` for u-blox GPS hardware
- `qcomgpsd`, `ugpsd`, `navd` currently **commented out** in process_config
### FrogPilot Additions
- `selfdrive/frogpilot/frogpilot_process.py` — FrogPilot main process
- `selfdrive/frogpilot/controls/` — custom planner, control libraries
- `selfdrive/frogpilot/fleetmanager/` — fleet management web UI
- `selfdrive/frogpilot/screenrecorder/` — C++ OMX-based screen recorder (dashcam)
- `selfdrive/frogpilot/ui/qt/` — custom UI widgets and settings panels
- `selfdrive/frogpilot/assets/` — custom assets
### ClearPilot Additions
- `selfdrive/clearpilot/clearpilot.js` — Node.js service
- `selfdrive/clearpilot/behavior/` — lane change, longitudinal control mode scripts
- `selfdrive/clearpilot/models/` — custom driving models (duck-amigo, farmville, wd-40)
- `selfdrive/clearpilot/manager/api.js` — manager API
- `selfdrive/clearpilot/theme/`, `selfdrive/clearpilot/resource/` — theming and assets
### Car Interface (Hyundai)
- `selfdrive/car/hyundai/interface.py` — main car interface
- `selfdrive/car/hyundai/carcontroller.py` — actuator commands
- `selfdrive/car/hyundai/carstate.py` — vehicle state parsing
- `selfdrive/car/hyundai/radar_interface.py` — radar data
- `selfdrive/car/hyundai/hyundaicanfd.py` — CAN-FD message definitions (HDA2 uses CAN-FD)
- `selfdrive/car/hyundai/values.py`, `fingerprints.py` — car-specific constants
### UI Code
- `selfdrive/ui/` — Qt/C++ based
- `selfdrive/ui/qt/onroad.cc` — main driving screen (contains uiDebug publish, screen recorder integration)
- `selfdrive/ui/qt/home.cc` — home/offroad screen
- `selfdrive/ui/qt/sidebar.cc` — sidebar
- `selfdrive/ui/ui.cc`, `ui.h` — UI state management
### Key Dependencies
- **Python 3.11** with: numpy, casadi, onnx/onnxruntime, pycapnp, pyzmq, sentry-sdk, sympy, Cython
- **Python 3.11** (via pyenv) with: numpy, casadi, onnx/onnxruntime, pycapnp, pyzmq, sentry-sdk, sympy, Cython
- **capnp (Cap'n Proto)** — IPC message serialization between all processes
- **ZeroMQ** — IPC transport layer
- **Qt 5** — UI framework
- **OpenMAX (OMX)** — hardware video encoding (screen recorder)
- **Qt 5** — UI framework (with WebEngine available but not used for rotation reasons)
- **OpenMAX (OMX)** — hardware video encoding
- **libavformat** — MP4 container muxing
- **libyuv** — color space conversion
- **SCons** — build system for native C++ components