docs: protocol audit 2026-05-25, update architecture + Obsidian vault
Audit: - docs/AUDIT-2026-05-25.md: full protocol audit covering 8 findings (4 critical, 2 high, 5 medium, 4 low) with code references and fix effort estimates - vault/Audit/Tasks.md: Obsidian Tasks plugin file tracking all audit items with priorities, due dates, and per-step checklists Architecture docs updated for Wire format v2 and Wave 5/6 features: - ARCHITECTURE.md: adds wzp-video to dependency graph and project structure; wire format updated to v2 (16B header, 5B MiniHeader); relay concurrency section corrected (DashMap+RwLock is current, not a future optimization); test count 571→702; Android note - PROGRESS.md: Wave 5 and Wave 6 sections appended; test count 372→702; current status and open blockers as of 2026-05-25 - ROAD-TO-VIDEO.md: implementation status table inserted (✅/🟡/🔴/🔲 per phase); 6-step critical path to first video call - WZP-SPEC.md: MediaHeader updated to v2 (16B byte-aligned); MiniHeader updated to 5B with seq_delta; codec IDs 9-12 added (H.264/H.265/AV1); version negotiation section added Obsidian vault (vault/): - 114 files across Architecture/, PRDs/, Reports/, Android/, Reference/, Audit/ with YAML frontmatter - 00 - Home.md index note with wiki links - .obsidian/app.json config Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Siavash Sameni
@@ -59,6 +59,7 @@ graph TD
|
||||
FEC["wzp-fec<br/>RaptorQ FEC"]
|
||||
CRYPTO["wzp-crypto<br/>ChaCha20 + Identity"]
|
||||
TRANSPORT["wzp-transport<br/>QUIC / Quinn"]
|
||||
VIDEO["wzp-video<br/>H.264 + H.265 + AV1"]
|
||||
|
||||
RELAY["wzp-relay<br/>Relay Daemon"]
|
||||
CLIENT["wzp-client<br/>CLI + Call Engine"]
|
||||
@@ -68,16 +69,19 @@ graph TD
|
||||
PROTO --> FEC
|
||||
PROTO --> CRYPTO
|
||||
PROTO --> TRANSPORT
|
||||
PROTO --> VIDEO
|
||||
|
||||
CODEC --> CLIENT
|
||||
FEC --> CLIENT
|
||||
CRYPTO --> CLIENT
|
||||
TRANSPORT --> CLIENT
|
||||
VIDEO --> CLIENT
|
||||
|
||||
CODEC --> RELAY
|
||||
FEC --> RELAY
|
||||
CRYPTO --> RELAY
|
||||
TRANSPORT --> RELAY
|
||||
VIDEO --> RELAY
|
||||
|
||||
CLIENT --> WEB
|
||||
TRANSPORT --> WEB
|
||||
@@ -90,9 +94,10 @@ graph TD
|
||||
style CLIENT fill:#00b894,color:#fff
|
||||
style WEB fill:#0984e3,color:#fff
|
||||
style FC fill:#fd79a8,color:#fff
|
||||
style VIDEO fill:#a29bfe,color:#fff
|
||||
```
|
||||
|
||||
**Star pattern**: Each leaf crate (`wzp-codec`, `wzp-fec`, `wzp-crypto`, `wzp-transport`) depends only on `wzp-proto`. No leaf depends on another leaf. Integration crates (`wzp-relay`, `wzp-client`, `wzp-web`) depend on all leaves.
|
||||
**Star pattern**: Each leaf crate (`wzp-codec`, `wzp-fec`, `wzp-crypto`, `wzp-transport`, `wzp-video`) depends only on `wzp-proto`. No leaf depends on another leaf. Integration crates (`wzp-relay`, `wzp-client`, `wzp-web`) depend on all leaves.
|
||||
|
||||
## Audio Encode Pipeline
|
||||
|
||||
@@ -106,7 +111,7 @@ sequenceDiagram
|
||||
participant DT as DredTuner<br/>(wzp-proto)
|
||||
participant FEC as RaptorQ FEC
|
||||
participant INT as Interleaver<br/>(depth=3)
|
||||
participant HDR as MediaHeader<br/>(12B or Mini 4B)
|
||||
participant HDR as MediaHeader<br/>(16B or Mini 5B)
|
||||
participant Enc as ChaCha20-Poly1305
|
||||
participant QUIC as QUIC Datagram
|
||||
participant QPS as QuinnPathSnapshot
|
||||
@@ -144,7 +149,7 @@ sequenceDiagram
|
||||
- RNNoise processes **2 x 480** samples (ML-based noise suppression via nnnoiseless)
|
||||
- Silence detection uses VAD + 100ms hangover before switching to ComfortNoise
|
||||
- FEC symbols are padded to **256 bytes** with a 2-byte LE length prefix
|
||||
- MiniHeaders (4 bytes) replace full headers (12 bytes) for 49 of every 50 frames
|
||||
- MiniHeaders (5 bytes) replace full headers (16 bytes) for 49 of every 50 audio frames; video always uses full headers
|
||||
- DRED tuner polls quinn path stats every 25 frames (~500ms) and adjusts DRED lookback duration continuously
|
||||
- Opus tiers bypass RaptorQ entirely -- DRED handles loss recovery at the codec layer
|
||||
- Opus6k DRED window: 1040ms (maximum libopus allows)
|
||||
@@ -324,35 +329,29 @@ sequenceDiagram
|
||||
|
||||
## Wire Formats
|
||||
|
||||
### MediaHeader (12 bytes)
|
||||
### `MediaHeader` v2 (16 bytes, byte-aligned)
|
||||
|
||||
```
|
||||
Byte 0: [V:1][T:1][CodecID:4][Q:1][FecRatioHi:1]
|
||||
Byte 1: [FecRatioLo:6][unused:2]
|
||||
Bytes 2-3: sequence (u16 BE)
|
||||
Bytes 4-7: timestamp_ms (u32 BE)
|
||||
Byte 8: fec_block_id (u8)
|
||||
Byte 9: fec_symbol_idx (u8)
|
||||
Byte 10: reserved
|
||||
Byte 11: csrc_count
|
||||
Byte 0: version (u8) 0x02
|
||||
Byte 1: flags (u8) [T:1][Q:1][KeyFrame:1][FrameEnd:1][reserved:4]
|
||||
T = FEC repair, Q = QualityReport trailer
|
||||
KeyFrame = packet belongs to an I-frame (video)
|
||||
FrameEnd = last packet of an access unit (video)
|
||||
Byte 2: media_type (u8) 0=audio, 1=video, 2=data, 3=control
|
||||
Byte 3: codec_id (u8) widened from 4-bit (room for 256 codec IDs)
|
||||
Byte 4: stream_id (u8) simulcast layer; 0=base
|
||||
Byte 5: fec_ratio (u8) 0..200 → 0.0..2.0
|
||||
Bytes 6-9: sequence (u32 BE) wrapping packet sequence number
|
||||
Bytes 10-13: timestamp_ms (u32 BE) milliseconds since session start
|
||||
Bytes 14-15: fec_block_id (u16 BE)
|
||||
audio: low 8 bits = block_id, high 8 bits = symbol_idx
|
||||
video: full u16 block_id (large blocks for I-frames)
|
||||
```
|
||||
|
||||
| Field | Bits | Description |
|
||||
|-------|------|-------------|
|
||||
| V (version) | 1 | Protocol version (0 = v1) |
|
||||
| T (is_repair) | 1 | 1 = FEC repair packet, 0 = source media |
|
||||
| CodecID | 4 | Codec identifier (0-8, see table below) |
|
||||
| Q | 1 | 1 = QualityReport trailer appended |
|
||||
| FecRatio | 7 | FEC ratio encoded as 0-127 mapping to 0.0-2.0 |
|
||||
| sequence | 16 | Wrapping packet sequence number |
|
||||
| timestamp_ms | 32 | Milliseconds since session start |
|
||||
| fec_block_id | 8 | FEC source block ID (wrapping) |
|
||||
| fec_symbol_idx | 8 | Symbol index within FEC block |
|
||||
| reserved | 8 | Reserved flags |
|
||||
| csrc_count | 8 | Contributing source count (future mixing) |
|
||||
|
||||
#### CodecID Values
|
||||
|
||||
**Audio codecs (media_type = 0)**
|
||||
|
||||
| Value | Codec | Bitrate | Sample Rate | Frame Duration |
|
||||
|-------|-------|---------|-------------|---------------|
|
||||
| 0 | Opus 24k | 24 kbps | 48 kHz | 20ms |
|
||||
@@ -365,15 +364,25 @@ Byte 11: csrc_count
|
||||
| 7 | Opus 48k | 48 kbps | 48 kHz | 20ms |
|
||||
| 8 | Opus 64k | 64 kbps | 48 kHz | 20ms |
|
||||
|
||||
### MiniHeader (4 bytes, compressed)
|
||||
**Video codecs (media_type = 1)**
|
||||
|
||||
| Value | Codec | Notes |
|
||||
|-------|-------|-------|
|
||||
| 9 | H.264 Baseline | Universal HW encode coverage |
|
||||
| 10 | H.264 Main | Slight quality win over baseline |
|
||||
| 11 | H.265 Main | Apple A10+, Snapdragon ~2017, NVENC GTX 9xx+; ~30% better than H.264 |
|
||||
| 12 | AV1 Main | Apple M3/A17+, Snapdragon 8 Gen 3+, RTX 40+; best efficiency, narrow HW |
|
||||
|
||||
### `MiniHeader` v2 (5 bytes)
|
||||
|
||||
```
|
||||
[FRAME_TYPE_MINI: 0x01]
|
||||
Bytes 0-1: timestamp_delta_ms (u16 BE)
|
||||
Bytes 2-3: payload_len (u16 BE)
|
||||
[FRAME_TYPE_MINI = 0x01]
|
||||
Byte 0: seq_delta (u8) delta from last full header's seq
|
||||
Bytes 1-2: timestamp_delta_ms (u16 BE)
|
||||
Bytes 3-4: payload_len (u16 BE)
|
||||
```
|
||||
|
||||
Used for 49 of every 50 frames (~1s cycle). Saves 8 bytes per packet (67% header reduction). Full header is sent every 50th frame to resynchronize state.
|
||||
Used for audio only (49 of every 50 frames). Saves 11 bytes per audio packet vs the full 16B header. Full header is sent every 50th frame to resynchronize state. Video always uses full 16B headers.
|
||||
|
||||
### TrunkFrame (batched datagrams)
|
||||
|
||||
@@ -482,9 +491,12 @@ sequenceDiagram
|
||||
|
||||
### Shared State & Locking
|
||||
|
||||
The `RoomManager` stores `DashMap<String, Arc<RwLock<Room>>>`. The DashMap guard is held only long enough to clone the `Arc`; all per-room operations then acquire the room-level `RwLock`. Concurrent fan-out calls share a read lock; join/leave acquire write lock.
|
||||
|
||||
| Lock | Protected Data | Hold Duration | Contention |
|
||||
|------|---------------|---------------|------------|
|
||||
| `RoomManager` (Mutex) | Rooms, participants, quality tiers | ~1ms/packet | O(N) per room |
|
||||
| `DashMap<room_id, Arc<RwLock<Room>>>` | Room registry | Instant (clone Arc only) | Near-zero |
|
||||
| `Room` (RwLock) | Participants, quality tiers | ~1ms/packet (read); ~1ms (write on join/leave) | Low (concurrent reads) |
|
||||
| `PresenceRegistry` (Mutex) | Fingerprint registrations | ~1ms | Low (join/leave only) |
|
||||
| `SessionManager` (Mutex) | Active session tracking | ~1ms | Low |
|
||||
| `FederationManager.peer_links` (Mutex) | Peer connections | ~10ms during forward | Per-federation-packet |
|
||||
@@ -492,15 +504,9 @@ sequenceDiagram
|
||||
### Scaling Characteristics
|
||||
|
||||
- **Many small rooms**: Scales well across all cores (rooms are independent)
|
||||
- **Large single room (100+ participants)**: Serialized by RoomManager lock
|
||||
- **Large single room (100+ participants)**: Fan-out reads share RwLock (non-blocking); only join/leave serializes
|
||||
- **Federation**: Per-peer tasks scale; `peer_links` lock held during send loop
|
||||
|
||||
### Primary Bottleneck
|
||||
|
||||
The RoomManager Mutex is acquired per-packet by every participant to get the fan-out peer list. Lock is released before I/O (sends happen outside lock), but packet processing is serialized through the lock within a room.
|
||||
|
||||
Future optimization: per-room locks or lock-free participant lists via `DashMap`.
|
||||
|
||||
## Client Architecture
|
||||
|
||||
### Desktop Engine (Tauri)
|
||||
@@ -553,6 +559,8 @@ Key design decisions:
|
||||
|
||||
### Android Engine (Kotlin + JNI)
|
||||
|
||||
> **Note (2026-05-12):** The Kotlin+JNI Android app (`android/app/`) described below is superseded by the **Tauri 2.x mobile build** (`desktop/src-tauri/` + `crates/wzp-native/`). The Tauri approach uses the same Rust call engine as desktop, with Oboe audio via `wzp-native` cdylib. The Kotlin codebase is maintained for reference but the Tauri build is the live production app.
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Compose UI"
|
||||
@@ -902,6 +910,20 @@ warzonePhone/
|
||||
│ │ └── rekey.rs # Forward secrecy rekeying
|
||||
│ ├── wzp-transport/ # QUIC transport layer
|
||||
│ │ └── src/lib.rs # QuinnTransport, send/recv media/signal/trunk
|
||||
│ ├── wzp-video/ # Video codecs + framer
|
||||
│ │ └── src/
|
||||
│ │ ├── factory.rs # VideoEncoder factory (platform dispatch)
|
||||
│ │ ├── framer.rs # NAL fragmentation (H.264/H.265)
|
||||
│ │ ├── depacketizer.rs # NAL reassembly, access unit emit
|
||||
│ │ ├── controller.rs # VideoQualityController
|
||||
│ │ ├── simulcast.rs # Simulcast layer management
|
||||
│ │ ├── encoder_mode.rs # Encoder mode selection
|
||||
│ │ ├── av1_obu.rs # AV1 OBU framing + depacketizer
|
||||
│ │ ├── dav1d.rs # dav1d AV1 software decoder
|
||||
│ │ ├── svt_av1.rs # SVT-AV1 software encoder (non-Android)
|
||||
│ │ ├── videotoolbox.rs # VideoToolbox H.265 + AV1 (macOS)
|
||||
│ │ ├── mediacodec.rs # MediaCodec H.264/H.265/AV1 (Android, NDK 0.9 migration pending)
|
||||
│ │ └── nack.rs # NACK sender/receiver framework
|
||||
│ ├── wzp-relay/ # Relay daemon
|
||||
│ │ └── src/
|
||||
│ │ ├── main.rs # CLI, connection loop, auth + handshake
|
||||
@@ -917,6 +939,10 @@ warzonePhone/
|
||||
│ │ ├── presence.rs # PresenceRegistry
|
||||
│ │ ├── route.rs # RouteResolver
|
||||
│ │ ├── trunk.rs # TrunkBatcher
|
||||
│ │ ├── audio_scorer.rs # Per-stream audio quality scoring
|
||||
│ │ ├── response_policy.rs # Relay response policy (rate-limit, drop)
|
||||
│ │ ├── verdict.rs # Verdict enum (Allow/RateLimit/Drop/Malicious)
|
||||
│ │ ├── video_scorer.rs # VideoScorer (legitimacy scoring, keyframe regularity)
|
||||
│ │ └── ws.rs # WebSocket handler for browser clients
|
||||
│ ├── wzp-client/ # Call engine + CLI
|
||||
│ │ └── src/
|
||||
@@ -956,7 +982,7 @@ warzonePhone/
|
||||
|
||||
## Test Coverage
|
||||
|
||||
571 tests across all crates, 0 failures:
|
||||
702 tests across all crates (excluding wzp-android), 0 failures:
|
||||
|
||||
| Crate | Tests | Key Coverage |
|
||||
|-------|-------|-------------|
|
||||
@@ -965,7 +991,8 @@ warzonePhone/
|
||||
| wzp-fec | 21 | RaptorQ encode/decode, loss recovery, interleaving |
|
||||
| wzp-crypto | 64 | Encrypt/decrypt, handshake, anti-replay, featherChat identity |
|
||||
| wzp-transport | 11 | QUIC connection setup, path monitoring |
|
||||
| wzp-relay | 122 | Room ACL, session mgmt, metrics, probes, mesh, trunking |
|
||||
| wzp-relay | 137 | Room ACL, session mgmt, metrics, probes, mesh, trunking, scoring, verdict |
|
||||
| wzp-video | 88 | NAL framing, AV1 OBU, simulcast, quality controller, NACK |
|
||||
| wzp-client | 170 | Encoder/decoder, quality adapter, silence, drift, sweep |
|
||||
| wzp-web | 2 | Metrics |
|
||||
| wzp-native | 0 | Native platform bindings (no unit tests) |
|
||||
|
||||
231
docs/AUDIT-2026-05-25.md
Normal file
231
docs/AUDIT-2026-05-25.md
Normal file
@@ -0,0 +1,231 @@
|
||||
# WarzonePhone Protocol Audit — 2026-05-25
|
||||
|
||||
**Auditor:** Claude Sonnet 4.6 (assisted)
|
||||
**Branch:** `experimental-ui` @ `f3e3ee5`
|
||||
**Scope:** All workspace crates (`wzp-proto`, `wzp-codec`, `wzp-fec`, `wzp-crypto`, `wzp-transport`, `wzp-relay`, `wzp-client`, `wzp-android`, `wzp-native`, `wzp-video`)
|
||||
**Test baseline:** 702 passing (excludes `wzp-android`)
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The audio call path is functionally correct and cryptographically sound on clean network paths. **There is a session-breaking bug in the crypto nonce derivation (C1) that will cause a permanent decryption failure on any out-of-order UDP delivery.** This is the single highest-priority fix — it will manifest as periodic session crashes under normal internet conditions. Video has a solid architectural foundation but three hard blockers remain before shipping: the AEAD coverage gap (C2), dead video scorer (C3), and Android MediaCodec compile failure (C4).
|
||||
|
||||
The project is in good shape overall. The crypto design (X25519, HKDF, ChaCha20-Poly1305, Ed25519 identity, SAS verification) is sound. The SFU-never-decrypts architecture is rare and valuable. The codec adaptation (Opus DRED + Codec2 RaptorQ split) is genuinely innovative. The eight issues below are fixable in ~12 engineer-hours.
|
||||
|
||||
---
|
||||
|
||||
## Critical
|
||||
|
||||
### C1 — Nonce derives from `recv_seq` counter, not `MediaHeader.seq`
|
||||
|
||||
**File:** `crates/wzp-crypto/src/session.rs:132`
|
||||
**Severity:** Critical — session-breaking on any packet reorder
|
||||
|
||||
```rust
|
||||
// decrypt()
|
||||
let nonce_bytes = nonce::build_nonce(&self.session_id, self.recv_seq, Direction::Send);
|
||||
// ...
|
||||
self.recv_seq = self.recv_seq.wrapping_add(1); // line 148
|
||||
```
|
||||
|
||||
`recv_seq` increments once per successful `decrypt()` call. The sender's `send_seq` also increments once per `encrypt()` call (line 120). In perfect in-order delivery they stay synchronized. With any reorder or mid-stream packet loss they permanently diverge. Once diverged, every subsequent packet uses the wrong nonce → AEAD tag mismatch → every packet fails for the rest of the session.
|
||||
|
||||
This isn't a low-probability edge case. UDP over any internet path reorders packets routinely. The `multiple_packets_roundtrip` test (line 254) only exercises in-order delivery. HANDOFF-2026-05-12.md acknowledges this as a known latent item: *"AEAD nonce derivation: switch to `MediaHeader::seq`"*.
|
||||
|
||||
The anti-replay check at lines 152–161 already parses `MediaHeader` and has `header.seq` available. The fix is one line in `decrypt()`:
|
||||
|
||||
```rust
|
||||
// Use sender's wire-level seq as nonce input, not a local counter.
|
||||
// This survives reordering because both sides derive the same nonce from
|
||||
// the same field. recv_seq was wrong: it diverged from send_seq on any
|
||||
// reorder, breaking all subsequent decryptions for the session.
|
||||
let header = parse_header(header_bytes)
|
||||
.ok_or_else(|| CryptoError::Internal("header parse failed".into()))?;
|
||||
let nonce_bytes = nonce::build_nonce(&self.session_id, header.seq, Direction::Send);
|
||||
```
|
||||
|
||||
Remove `recv_seq` field from `ChaChaSession` (it's now redundant — anti-replay uses `header.seq` directly). On the encrypt side, verify that `self.send_seq` equals the `seq` written into the `MediaHeader` at the call site.
|
||||
|
||||
**Estimated effort:** ~1 hour including test coverage for out-of-order delivery.
|
||||
|
||||
> **Note on rekey seq reset:** The agent initially flagged `send_seq/recv_seq = 0` in `complete_rekey()` as a separate critical issue. This is a false positive — `install_key()` rotates `session_id` (hash of new key), so pre-/post-rekey nonces live in distinct namespaces. The reset is intentional and cryptographically safe.
|
||||
|
||||
---
|
||||
|
||||
### C2 — AEAD not wired to every QUIC datagram send path
|
||||
|
||||
**File:** `crates/wzp-client/src/analyzer.rs:363` (only confirmed decrypt call site)
|
||||
**Severity:** Critical — potential plaintext media leakage
|
||||
|
||||
The HANDOFF document explicitly flags this: *"Encryption is implemented in `wzp-crypto` but not yet on every QUIC datagram path."* The `analyzer.rs` path decrypts inbound packets. What needs verification: every outbound `send_datagram()` / `write_datagram()` call across `wzp-client` and `wzp-transport` must pass through `ChaChaSession::encrypt()`.
|
||||
|
||||
**Required action:** Grep every `send_datagram` call site. Confirm each path encrypts before transmit. Add a CI-level test or `#[forbid(dead_code)]`-style assertion that makes a plaintext send path impossible to merge. Until this is verified, the E2E security claim cannot be made.
|
||||
|
||||
**Estimated effort:** ~1 hour audit + test.
|
||||
|
||||
---
|
||||
|
||||
### C3 — `VideoScorer::observe()` never called — scorer is dead code
|
||||
|
||||
**File:** `crates/wzp-relay/src/room.rs:1263–1266`
|
||||
**Severity:** Critical — relay abuse control for video is completely absent
|
||||
|
||||
```rust
|
||||
// T6.2-follow-up: feed video packets to VideoScorer here.
|
||||
// video_scorer.observe(&pkt.header, pkt.payload.len(), now, bwe_kbps);
|
||||
```
|
||||
|
||||
`video_scorer.rs` was delivered in T6.2 with legitimacy scoring, keyframe regularity checks, I/P ratio analysis, and a verdict enum. The observe call was never wired into the packet forwarding loop. The scorer compiles but accumulates no data. Any participant can flood the room with malformed video or synthetic keyframe bursts and the relay will forward everything without challenge.
|
||||
|
||||
**Fix:** Wire `video_scorer.observe(...)` at the TODO marker and integrate `legitimacy_score()` into the forwarding decision (drop or rate-limit streams with `Verdict::Malicious`). Add an integration test: synthetic high-frequency keyframe bursts should trigger a `Malicious` verdict within 2 seconds.
|
||||
|
||||
**Estimated effort:** ~2 hours.
|
||||
|
||||
---
|
||||
|
||||
### C4 — `wzp-video` Android target fails to compile (31 errors)
|
||||
|
||||
**File:** `crates/wzp-video/src/mediacodec.rs`
|
||||
**Severity:** Critical — Android video is completely blocked
|
||||
|
||||
Five error categories from the NDK 0.9 API migration, all documented in HANDOFF-2026-05-12.md. `dav1d`/`svt-av1` were cfg-gated off Android in `f3e3ee5`; these 31 errors are the remaining MediaCodec API mismatch.
|
||||
|
||||
| Error | Count | Root cause | Fix |
|
||||
|---|---|---|---|
|
||||
| `E0277` `NonNull<AMediaCodec>` not `Send` | ~3 | Raw pointer held across `tokio::spawn` boundary | `struct SendMediaCodec(NonNull<…>); unsafe impl Send for SendMediaCodec {}` — or use `ndk::media::MediaCodec` owned type (already `Send`) |
|
||||
| `E0308` `&[MaybeUninit<u8>]` vs `&[u8]` | many | NDK 0.9 returns uninit slices | `MaybeUninit::write_slice` or transmute pattern |
|
||||
| `E0425` missing `BITRATE_MODE_CBR` | 1+ | Constant renamed in NDK 0.9 | Check `ndk` crate docs for current name |
|
||||
| `E0433` `ndk_sys` not a dep | several | Direct `ndk_sys` import; only `ndk = "0.9"` declared | Add `ndk-sys` as explicit dep or use safe `ndk` wrappers |
|
||||
| `E0599` `InputBuffer::index()` / `OutputBuffer::index()` private | 2 | API changed in NDK 0.9 | Use buffer through safe queue/dequeue API |
|
||||
|
||||
Nothing live is blocked today — `wzp-video` is not yet consumed by Tauri Android. But video on Android cannot progress until this compiles.
|
||||
|
||||
**Reproduce:**
|
||||
```bash
|
||||
ssh -i ~/CascadeProjects/wzp manwe@manwehs \
|
||||
'cd ~/wzp-builder/data/source && \
|
||||
docker run --rm \
|
||||
-v ~/wzp-builder/data/source:/build/source \
|
||||
-v ~/wzp-builder/data/cache/cargo-registry:/home/builder/.cargo/registry \
|
||||
-v ~/wzp-builder/data/cache/cargo-git:/home/builder/.cargo/git \
|
||||
-v ~/wzp-builder/data/cache/target:/build/source/target \
|
||||
wzp-android-builder:latest \
|
||||
bash -c "cd /build/source && cargo build --target aarch64-linux-android -p wzp-video 2>&1 | tail -60"'
|
||||
```
|
||||
|
||||
**Estimated effort:** ~2 hours (one commit per error category).
|
||||
|
||||
---
|
||||
|
||||
## High
|
||||
|
||||
### H1 — AV1 call engine wiring missing
|
||||
|
||||
**Source:** HANDOFF-2026-05-12.md (T6.1.2 open item)
|
||||
**File:** `crates/wzp-video/src/factory.rs`
|
||||
|
||||
`factory.rs` and step tables landed in commit `086d0a4`. No caller yet invokes `create_video_encoder(Av1Main, ...)`. The entire AV1 path is reachable only from tests. Video on macOS/Linux desktop requires wiring `create_video_encoder` into the call engine's media negotiation path.
|
||||
|
||||
**Estimated effort:** ~1–2 hours.
|
||||
|
||||
---
|
||||
|
||||
### H2 — `fec_block_id: u8` wraps every ~25 seconds
|
||||
|
||||
**File:** `crates/wzp-fec/src/encoder.rs` (`block_id.wrapping_add(1)` on u8)
|
||||
**Reference:** PROTOCOL-AUDIT.md W2 (deferred P2)
|
||||
|
||||
At 5 frames/block (Codec2), u8 ID wraps at block 256 ≈ 25 seconds. A slow reconstructor or late-joining peer will collide block IDs with in-flight blocks. The window distance check in `block_manager.rs` partially mitigates this but can't prevent all collisions. Widen to `u16` in the next wire-format revision.
|
||||
|
||||
---
|
||||
|
||||
## Medium
|
||||
|
||||
### M1 — `SignalMessage` has no version byte
|
||||
|
||||
**File:** `crates/wzp-proto/src/session.rs` (SignalMessage enum)
|
||||
**Reference:** PROTOCOL-AUDIT.md W12
|
||||
|
||||
`bincode + serde(default)` handles field additions but not variant removal or semantic changes. Any variant deprecation is silent at the wire level. This becomes a correctness risk when federation routes `SignalMessage`s across relay versions. Add `version: u8` as a leading field to all variants before federation ships.
|
||||
|
||||
---
|
||||
|
||||
### M2 — BWE not consumed by `AdaptiveQualityController`
|
||||
|
||||
**Reference:** PROTOCOL-AUDIT.md W6, deferred to Phase V2
|
||||
|
||||
Quinn exposes `cwnd` and `bytes_in_flight`, but `AdaptiveQualityController` does not consume them. Loss + RTT adaptation works for audio. For video, without bandwidth estimation the encoder cannot detect available uplink capacity and will either oscillate or permanently under-utilize bandwidth. Mandatory before video production.
|
||||
|
||||
---
|
||||
|
||||
### M3 — PLI suppression window hardcoded at 200ms
|
||||
|
||||
**File:** `crates/wzp-relay/src/room.rs:1060`
|
||||
|
||||
Not adaptive to link speed. On slow links 200ms may allow multiple keyframe requests. Accept for Phase 1; make configurable in Phase 2.
|
||||
|
||||
---
|
||||
|
||||
### M4 — Repair packet index wrapping in FEC encoder
|
||||
|
||||
**File:** `crates/wzp-fec/src/encoder.rs:140`
|
||||
|
||||
```rust
|
||||
let idx = (num_source as u8).wrapping_add(i as u8);
|
||||
```
|
||||
|
||||
If `num_source + repair_count > 255`, indices wrap silently. In practice bounded by `frames_per_block` (5–10), so max sum is ~20. Low risk today; widen to u16 when `fec_block_id` is widened (H2).
|
||||
|
||||
---
|
||||
|
||||
### M5 — `timestamp_ms` monotonicity after rekey not enforced
|
||||
|
||||
**Reference:** PROTOCOL-AUDIT.md W3
|
||||
|
||||
Spec: `timestamp_ms` must not reset on rekey. The code correctly does not reset it, but there is no assertion to prevent regression. Add a debug assert in `complete_rekey()` that `new_session.next_timestamp >= old_session.last_timestamp`.
|
||||
|
||||
---
|
||||
|
||||
## Low / Accepted Debt
|
||||
|
||||
| ID | Description | File | Accepted in |
|
||||
|---|---|---|---|
|
||||
| L1 | 9 pre-existing clippy lints in `wzp-codec` | `aec.rs`, `denoise.rs`, `opus_enc.rs`, `codec2_{enc,dec}.rs`, `resample.rs` | PROTOCOL-AUDIT.md |
|
||||
| L2 | 3 clippy errors in `deps/featherchat` submodule | `ratchet.rs`, `types.rs` | PROTOCOL-AUDIT.md |
|
||||
| L3 | Audio anti-replay window 64 packets | `wzp-crypto/src/session.rs:89` | Accepted — jitter buffer + PLC masks loss |
|
||||
| L4 | Debug tap logs at INFO with no rate limiting | `wzp-relay/src/room.rs:46–59` | Safe in dev; add 1:100 sampling for prod |
|
||||
|
||||
---
|
||||
|
||||
## What Was Not Found
|
||||
|
||||
These are explicitly confirmed sound after code-level verification:
|
||||
|
||||
- **Anti-replay bitmap** — correct u32 wrapping, per-stream isolation, window sizing by `MediaType`
|
||||
- **HKDF + X25519 + Ed25519 key agreement** — standard construction, no gaps
|
||||
- **SAS code derivation** — SHA-256(shared_secret)[:4] as 4-digit voice verification code
|
||||
- **Rekey forward secrecy** — `session_id` rotation on rekey isolates nonce namespaces; seq counter reset is intentional and safe
|
||||
- **MiniHeader v2 `seq_delta`** — fully implemented at `wzp-proto/src/packet.rs:469–526` with tests; PROTOCOL-AUDIT resolution table is accurate
|
||||
- **SFU E2E preservation** — relay ciphertext passthrough, no plaintext access
|
||||
- **RaptorQ for Codec2** — correct tool for the bitrate regime
|
||||
- **DRED continuous tuning** — better than discrete tiers; 15% loss floor is empirically grounded
|
||||
- **Jitter buffer** — BTreeMap with wrapping-aware comparisons, EWMA adaptive playout delay, solid
|
||||
- **Quinn QUIC datagram transport** — correct primitives for unreliable media
|
||||
|
||||
---
|
||||
|
||||
## Fix Priority Table
|
||||
|
||||
| # | Issue | Category | Effort | Blocks |
|
||||
|---|---|---|---|---|
|
||||
| 1 | C1: nonce → `MediaHeader.seq` | Crypto | 1h | All sessions on lossy paths |
|
||||
| 2 | C2: verify AEAD on all datagram send paths | Crypto | 1h | E2E security claim |
|
||||
| 3 | C3: wire `VideoScorer::observe()` into room | Relay | 2h | Relay abuse control for video |
|
||||
| 4 | C4: NDK 0.9 `mediacodec.rs` migration (5 categories) | Android | 2h | Android video |
|
||||
| 5 | H1: wire AV1 factory into call engine | Video | 2h | Desktop video |
|
||||
| 6 | H2: widen `fec_block_id` to `u16` | FEC/Wire | 30min | Next protocol release |
|
||||
| 7 | M1: `SignalMessage` version byte | Proto | 1h | Federation correctness |
|
||||
| 8 | M2: BWE into `AdaptiveQualityController` | Transport | 2–3 days | Video production quality |
|
||||
|
||||
**Total for C1–H1 (items 1–5):** ~8 hours focused engineering.
|
||||
166
docs/HANDOFF-2026-05-12.md
Normal file
166
docs/HANDOFF-2026-05-12.md
Normal file
@@ -0,0 +1,166 @@
|
||||
# Handoff — 2026-05-12 EOD
|
||||
|
||||
## TL;DR
|
||||
|
||||
Wave 5 (Phase 5) and Wave 6 (Phase 6) implementation is complete and approved on the board. Stopping for the night with one open issue: `wzp-video` does not target-compile for `aarch64-linux-android` and needs a focused `ndk = "0.9"` API migration session (~1–2 h). Nothing live is blocked — Tauri Android does not yet consume `wzp-video`.
|
||||
|
||||
**Branch state:** local `experimental-ui` HEAD `f3e3ee5`, pushed to `github` only. **Not yet on `fj`** (deploy key was read-only). Build server (`manwe@manwehs`) is up to date via github fetch.
|
||||
|
||||
---
|
||||
|
||||
## What landed today
|
||||
|
||||
| Wave | Tasks approved | New crates / files | Test delta |
|
||||
|---|---|---|---|
|
||||
| 5 | T5.1, T5.1.1, T5.2, T5.3, T5.4, T5.5, T5.6, T5.7, T5.7.1, T5.8 | `crates/wzp-relay/src/audio_scorer.rs`, `response_policy.rs`, `verdict.rs`; `wzp-video/src/controller.rs`, `simulcast.rs`, `encoder_mode.rs`; H.265 path in VT + MediaCodec | wzp-relay 99→127, wzp-video 43→71 |
|
||||
| 6 | T6.1 (+ rework), T6.1.2, T6.2 | `wzp-video/src/av1_obu.rs`, `dav1d.rs`, `svt_av1.rs`, `factory.rs`; VT AV1 decoder; MediaCodec AV1; `wzp-relay/src/video_scorer.rs` | wzp-video 76→88, wzp-relay 127→137 |
|
||||
|
||||
Total: ~30 task units approved across the two waves. Workspace tests at 702 passing (excluding `wzp-android`).
|
||||
|
||||
---
|
||||
|
||||
## Open / next-up
|
||||
|
||||
### Top of queue
|
||||
|
||||
- **T4.3.1.1 (deferred → in-progress, blocked)** — Android target-compile of `wzp-video`. We started this tonight and hit 31 errors in `crates/wzp-video/src/mediacodec.rs` against the actual `ndk = "0.9"` API. Error categories captured below; resume with one fix-per-category commit, then attempt device instrumentation.
|
||||
- **T6.3 — federated reputation gossip.** Design exploration committed (`1e729e4`, `docs/PRD/PRD-relay-federation-gossip.md`). **Decision made: Approach 3 (Ban-List Distribution).** My answers to the 6 blocker questions are in the chat thread, awaiting conversion to a real Files/Steps/Verify/Done-when task spec for the agent. The user opted not to run the agent immediately; the task spec is a write-then-park.
|
||||
- **T5.1.1 follow-ups** — none. T5.1.1 closed clean.
|
||||
|
||||
### Latent follow-ups from earlier waves
|
||||
|
||||
These pre-date wave 6 and are still open:
|
||||
|
||||
- **AEAD wired into prod send/recv path** (referenced in T1.5 / T1.6 reports). Encryption is implemented in `wzp-crypto` but not yet on every QUIC datagram path.
|
||||
- **AEAD nonce derivation: switch to `MediaHeader::seq`** (cited in T1.5.x reports). Current scheme works but isn't tied to wire-level seq.
|
||||
- **`wzp-codec` clippy debt sprint** — 9 errors documented as known debt in `docs/PROTOCOL-AUDIT.md`.
|
||||
- **T6.1.2 — wire AV1 into actual call engine.** The factory + step tables landed (commit `086d0a4`); no caller invokes `create_video_encoder(Av1Main, …)` yet. Real video sender wiring (the originally-blocked task) is unstarted.
|
||||
- **T6.2-follow-up — wire `VideoScorer::observe()` into the packet path.** TODO marker at `crates/wzp-relay/src/room.rs:1263`.
|
||||
|
||||
### Permanently deferred
|
||||
|
||||
- **T6.1.1 — Android MediaCodec AV1 device validation.** Deferred indefinitely: the user does not own an AV1-encode-capable Android or iPhone, and AV1 hardware will not be widespread for years. Revisit when devices land.
|
||||
|
||||
---
|
||||
|
||||
## The T4.3.1.1 Android build situation
|
||||
|
||||
What we did tonight:
|
||||
|
||||
1. Pushed `experimental-ui` to `github` (deploy key on `fj` is read-only).
|
||||
2. Added `github` as a remote on `manwe@manwehs:~/wzp-builder/data/source/` and checked out `experimental-ui`.
|
||||
3. Ran `cargo build --target aarch64-linux-android -p wzp-video` inside the `wzp-android-builder:latest` docker image.
|
||||
4. First failure: `shiguredo_dav1d` and `shiguredo_svt_av1` build scripts panic with `unsupported target: os=android, arch=aarch64`. Fixed in commit `f3e3ee5` (`fix(wzp-video): cfg-gate dav1d + svt-av1 off Android target`) — those crates now live under `[target.'cfg(not(target_os = "android"))'.dependencies]`, since Android uses MediaCodec for AV1 anyway.
|
||||
5. Re-ran the build → 31 errors in `mediacodec.rs`. **Stopped here.**
|
||||
|
||||
### Error categories to fix tomorrow
|
||||
|
||||
Run the same docker invocation and tackle these one fix-commit per category:
|
||||
|
||||
| Error | Count | Root cause | Likely fix |
|
||||
|---|---|---|---|
|
||||
| `E0277` `NonNull<AMediaCodec>` not `Send` | ~3 | Raw pointer field on a struct held across `tokio::spawn`-able boundaries | Wrap in `struct SendMediaCodec(NonNull<…>); unsafe impl Send for SendMediaCodec {}` or use the `ndk` crate's owned `MediaCodec` type which already implements `Send` |
|
||||
| `E0308` `&[MaybeUninit<u8>]` vs `&[u8]` | many | `ndk 0.9` returns uninitialized buffer slices; agent wrote into them as if initialized | Use `MaybeUninit::write_slice` or transmute pattern; pattern matches what `InputBuffer::write` expects |
|
||||
| `E0425` missing `BITRATE_MODE_CBR` | 1+ | Constant moved/renamed in `ndk 0.9` | Search `ndk` crate docs for current constant name (likely under `MediaCodec::set_parameters` enum) |
|
||||
| `E0433` `ndk_sys` not linked | several | Agent imported `ndk_sys` directly; it's not a dep, only `ndk = "0.9"` is | Replace direct `ndk_sys` calls with safe wrappers from the `ndk` crate, or add `ndk_sys` as an explicit dep |
|
||||
| `E0599` `InputBuffer::index()` / `OutputBuffer::index()` private | 2 | Both are private fields in `ndk 0.9`; were public methods in older versions | Either use the buffer through its safe API (queue/dequeue by handle) or expose index via a different accessor — read the `ndk` source for current API |
|
||||
|
||||
### Reproduce the build
|
||||
|
||||
```bash
|
||||
ssh -i ~/CascadeProjects/wzp manwe@manwehs \
|
||||
'cd ~/wzp-builder/data/source && \
|
||||
docker run --rm \
|
||||
-v ~/wzp-builder/data/source:/build/source \
|
||||
-v ~/wzp-builder/data/cache/cargo-registry:/home/builder/.cargo/registry \
|
||||
-v ~/wzp-builder/data/cache/cargo-git:/home/builder/.cargo/git \
|
||||
-v ~/wzp-builder/data/cache/target:/build/source/target \
|
||||
wzp-android-builder:latest \
|
||||
bash -c "cd /build/source && cargo build --target aarch64-linux-android -p wzp-video 2>&1 | tail -100"'
|
||||
```
|
||||
|
||||
After local fixes:
|
||||
|
||||
```bash
|
||||
git push github experimental-ui && \
|
||||
ssh -i ~/CascadeProjects/wzp manwe@manwehs \
|
||||
'cd ~/wzp-builder/data/source && git fetch github && git reset --hard github/experimental-ui'
|
||||
# then re-run the docker build
|
||||
```
|
||||
|
||||
### Device instrumentation half (post-compile)
|
||||
|
||||
User has a physical Android device. Once `cargo build --target aarch64-linux-android -p wzp-video` is clean:
|
||||
|
||||
- Build a minimal test harness binary (probably under `wzp-video/examples/` or a new `wzp-android-test/` crate) that does encode → decode of a synthetic frame via MediaCodec.
|
||||
- Use `adb push` and `adb shell run` to exercise it.
|
||||
- Compare output bytes against the dav1d/SVT-AV1 SW roundtrip from `crates/wzp-video/src/svt_av1.rs:101 svt_av1_dav1d_roundtrip_10_frames`.
|
||||
|
||||
Out of scope for tomorrow if the API migration eats the whole session.
|
||||
|
||||
---
|
||||
|
||||
## T6.3 — Approach 3 decision
|
||||
|
||||
User picked Approach 3 (Ban-List Distribution) from `docs/PRD/PRD-relay-federation-gossip.md`. My answers to the 6 open questions:
|
||||
|
||||
1. **Trust model:** Single admin key (user). Strongest Sybil resistance, lowest complexity.
|
||||
2. **Key infra:** Reuse `wzp-crypto` Ed25519. Admin pubkey in relay config; relays verify list signatures.
|
||||
3. **Fingerprint scope:** Ed25519 pubkey, not IP. Resistant to NAT rebind evasion.
|
||||
4. **Privacy:** Publish `SHA-256(pubkey)` hashes, not raw pubkeys. Relays compute `H(observed)` and match. 256-bit space makes brute-force infeasible; loses some audit trail.
|
||||
5. **TTL:** 30-day per-entry auto-expiry. Forces ops to actively re-publish persistent bans; prevents forever-by-mistake.
|
||||
6. **Rate limiting:** N/A under Approach 3 (no gossip channel; relays poll a signed list at configurable interval, that interval is the rate limit).
|
||||
|
||||
Next step: turn these into a Files/Steps/Verify/Done-when task spec in `docs/PRD/TASKS.md` and move T6.3 from `Blocked` → `Open` ready for the agent to claim. User did not want this kicked off tonight.
|
||||
|
||||
---
|
||||
|
||||
## Build / sync state
|
||||
|
||||
| Location | Branch | HEAD |
|
||||
|---|---|---|
|
||||
| Local (Mac) | `experimental-ui` | `f3e3ee5 fix(wzp-video): cfg-gate dav1d + svt-av1 off Android target` |
|
||||
| `github` remote | `experimental-ui` | `f3e3ee5` (pushed) |
|
||||
| `fj` remote | `experimental-ui` | **not pushed** (deploy key read-only on `fj`) |
|
||||
| `origin` (git.manko.yoga) | `experimental-ui` | **not pushed** |
|
||||
| Build server `~/wzp-builder/data/source` | `experimental-ui` | `f3e3ee5` |
|
||||
|
||||
If you want everything on `fj` / `origin` too, get the deploy key write-privileged or push from a different identity.
|
||||
|
||||
`fj/main` and `github/main` have one commit (`9ae9441 fix(audio): check capture ring available...`) that doesn't exist on `experimental-ui` — a small audio fix from May 11. Cherry-pick or merge before merging `experimental-ui` back into `main`.
|
||||
|
||||
### Gitleaks allowlist
|
||||
|
||||
Added `.gitleaks.toml` in commit `f28f39d` to allowlist 4 pre-existing historical findings. Two are real tokens (paste.tbs.amn.gg and paste.dk.manko.yoga `Authorization` headers in `scripts/build*.sh`). **Rotate those tokens if those endpoints still authenticate** — the allowlist only silences the pre-push hook; the secrets are still in git history.
|
||||
|
||||
---
|
||||
|
||||
## Agent process notes for tomorrow
|
||||
|
||||
The Kimi Code CLI agent on this project has a **stable, well-documented fabrication tic** — one verifiable detail per report is wrong (SHA, "updated X in same commit", fmt/clippy passes, etc.). Pattern survived an explicit CR on T6.1.
|
||||
|
||||
**Updated policy** (in `memory/feedback_kimi_report_fabrication.md`):
|
||||
|
||||
1. **Always verify the SHA** in the report header against `git log`.
|
||||
2. **Always run** `cargo fmt --check` and `cargo clippy -- -D warnings` yourself — don't trust the report's claims.
|
||||
3. **Don't CR fabrications anymore** — the T6.1 CR didn't change the behavior. Reviewer-fix the detail, note on the board, move on. Reserve CRs for substance issues.
|
||||
|
||||
The substance of the code has been consistently good. Don't let the fabrication tic bias review of the code itself.
|
||||
|
||||
### Rebase tic
|
||||
|
||||
Agent has twice rewritten already-pushed commits to address CR feedback (T5.7.1 `d3b2da6` → `517d0eb`; T6.1 `0de9522` → `9334aa5`). Forward fix commits are the rule; rebasing wasn't asked for and breaks reviewer references. Mention this only if it happens a third time.
|
||||
|
||||
---
|
||||
|
||||
## Tomorrow's suggested checklist
|
||||
|
||||
1. **(20 min)** Read this doc, the `feedback_kimi_report_fabrication.md` memory, and the T6.1 / T6.2 / T6.1.2 board rows on `docs/PRD/TASKS.md` to reload context.
|
||||
2. **(1–2 h)** Resume T4.3.1.1: ndk-0.9 API migration in `crates/wzp-video/src/mediacodec.rs`. One commit per error category.
|
||||
3. **(30 min)** If migration lands clean, attempt the minimal device test on the user's Android phone.
|
||||
4. **(20 min, optional)** Convert the T6.3 design answers into a task spec block in `TASKS.md`, leave it `Open` for the agent. Don't kick off the agent unless asked.
|
||||
5. **(parking lot)** AEAD prod wiring + nonce switch + wzp-codec clippy sprint — none urgent.
|
||||
|
||||
---
|
||||
|
||||
*Generated 2026-05-12, end of Wave 6 push.*
|
||||
104
docs/PROGRESS.md
104
docs/PROGRESS.md
@@ -389,3 +389,107 @@ Run with `wzp-bench --all`. Representative results (Apple M-series, single core)
|
||||
- `RegisterPresenceAck` populates `relay_region` from config, `available_relays` from federation peers
|
||||
- Desktop `place_call`/`answer_call` call `acquire_port_mapping()` and fill mapped addr fields
|
||||
- Legacy `build-android-docker.sh` renamed to `build-android-docker-LEGACY.sh` to prevent accidental use
|
||||
|
||||
## Wave 5: Video Infrastructure (2026-05-12)
|
||||
|
||||
**Tasks completed:** T5.1, T5.1.1, T5.2, T5.3, T5.4, T5.5, T5.6, T5.7, T5.7.1, T5.8
|
||||
|
||||
### Relay: Audio + Video Scoring
|
||||
|
||||
New files in `crates/wzp-relay/src/`:
|
||||
|
||||
- `audio_scorer.rs` — per-stream audio quality scorer tracking packet loss, codec consistency, bitrate stability
|
||||
- `response_policy.rs` — relay response policy engine mapping scores to action thresholds
|
||||
- `verdict.rs` — `Verdict` enum: `Allow`, `RateLimit`, `Drop`, `Malicious`
|
||||
- `video_scorer.rs` — `VideoScorer` with legitimacy scoring: keyframe regularity, I/P ratio, bandwidth responsiveness. **Note: wired but `observe()` not yet called from room forwarding path — T6.2 follow-up open.**
|
||||
|
||||
### Video: H.265 + Quality Controller
|
||||
|
||||
New files in `crates/wzp-video/src/`:
|
||||
|
||||
- `controller.rs` — `VideoQualityController`: maps (bwe_bps, loss_pct, rtt_ms, priority_mode) to (target_bitrate, target_fps, target_resolution, simulcast_layer)
|
||||
- `simulcast.rs` — simulcast layer management (base + enhancement layers)
|
||||
- `encoder_mode.rs` — encoder mode selection (CBR/VBR, keyframe intervals, quality presets)
|
||||
|
||||
H.265 encode/decode path added to:
|
||||
- `videotoolbox.rs` — VideoToolbox H.265 encoder + decoder (macOS/iOS)
|
||||
- `mediacodec.rs` — MediaCodec H.265 encoder + decoder (Android; NDK 0.9 compile errors pending in T4.3.1.1)
|
||||
|
||||
**Test delta:** wzp-relay 99→127, wzp-video 43→71
|
||||
|
||||
---
|
||||
|
||||
## Wave 6: AV1 + Federation Gossip Design (2026-05-12)
|
||||
|
||||
**Tasks completed:** T6.1, T6.1.2, T6.2
|
||||
|
||||
### Video: AV1 Codec Support
|
||||
|
||||
New files in `crates/wzp-video/src/`:
|
||||
|
||||
- `av1_obu.rs` — AV1 OBU (Open Bitstream Unit) framing and depacketizer
|
||||
- `dav1d.rs` — dav1d AV1 software decoder (non-Android; gated via cfg)
|
||||
- `svt_av1.rs` — SVT-AV1 software encoder (non-Android; gated via cfg)
|
||||
|
||||
Updated files:
|
||||
- `videotoolbox.rs` — VideoToolbox AV1 decoder + encoder (macOS M3+, iOS A17+)
|
||||
- `mediacodec.rs` — MediaCodec AV1 (Android; compile errors pending)
|
||||
- `factory.rs` — `create_video_encoder(codec, platform)` dispatcher added; H.264, H.265, AV1 wired
|
||||
|
||||
**T6.1.2 follow-up open:** `create_video_encoder(Av1Main, ...)` has no caller in the call engine yet — wiring step is unstarted.
|
||||
|
||||
### Relay: Federation Reputation Gossip (Design Phase)
|
||||
|
||||
- T6.3 design exploration committed at `1e729e4`
|
||||
- `docs/PRD/PRD-relay-federation-gossip.md` — Ban-List Distribution approach selected (Approach 3)
|
||||
- Implementation not started; task spec pending conversion
|
||||
|
||||
### Test Counts
|
||||
|
||||
**Test delta Wave 6:** wzp-video 76→88, wzp-relay 127→137
|
||||
|
||||
**Total workspace tests: 702** (excluding `wzp-android`)
|
||||
|
||||
| Crate | Tests |
|
||||
|---|---|
|
||||
| wzp-proto | 112 |
|
||||
| wzp-codec | 69 |
|
||||
| wzp-fec | 21 |
|
||||
| wzp-crypto | 64 |
|
||||
| wzp-transport | 11 |
|
||||
| wzp-relay | 137 |
|
||||
| wzp-client | 200 |
|
||||
| wzp-video | 88 |
|
||||
| wzp-web | 2 |
|
||||
| wzp-native | 0 |
|
||||
|
||||
---
|
||||
|
||||
## Current Status (2026-05-25)
|
||||
|
||||
### What Works (Audio)
|
||||
|
||||
All audio path items from previous status section remain working. Additionally:
|
||||
|
||||
- MediaHeader v2 (16 bytes) deployed across all paths
|
||||
- MiniHeader v2 (5 bytes with seq_delta) deployed
|
||||
- Anti-replay windows per stream with media-type-aware sizing (audio 64, video 1024)
|
||||
- Relay DashMap + RwLock concurrency model (T3.1 resolved the Mutex bottleneck)
|
||||
|
||||
### What Works (Video — partial)
|
||||
|
||||
- H.264 framer/depacketizer with FU-A fragmentation handling
|
||||
- H.264, H.265, AV1 VideoToolbox encode/decode (macOS)
|
||||
- AV1 dav1d + SVT-AV1 software path (non-Android)
|
||||
- Video quality controller, simulcast, encoder mode selection (controller only; no active call wiring yet)
|
||||
- Video scorer (scoring logic complete; not yet wired into relay forwarding)
|
||||
- NACK framework (`nack.rs`; not yet wired into room forwarding)
|
||||
|
||||
### Open Blockers
|
||||
|
||||
- **Android video:** `mediacodec.rs` has 31 NDK 0.9 compile errors (T4.3.1.1 in progress)
|
||||
- **AV1 call wiring:** `create_video_encoder(Av1Main, ...)` has no caller (T6.1.2 follow-up)
|
||||
- **VideoScorer wiring:** `VideoScorer::observe()` commented out at `room.rs:1263` (T6.2 follow-up)
|
||||
- **NACK wiring:** NACK path not wired into room forwarding (Phase V2/V4)
|
||||
- **BWE:** `AdaptiveQualityController` does not consume `cwnd`/`bytes_in_flight` (Phase V2)
|
||||
- **Crypto nonce bug:** `decrypt()` uses `recv_seq` instead of `MediaHeader.seq` (see AUDIT-2026-05-25.md C1)
|
||||
|
||||
@@ -12,6 +12,36 @@ The transport, crypto, session, federation, and SFU layers are codec-agnostic. T
|
||||
4. Keyframe semantics (PLI, NACK, keyframe cache at SFU)
|
||||
5. Capture / encode pipeline (VideoToolbox / MediaCodec / NVENC)
|
||||
|
||||
## Implementation Status (as of 2026-05-25)
|
||||
|
||||
| Phase | Description | Status |
|
||||
|---|---|---|
|
||||
| V1 — Wire format | 16B MediaHeader v2, 5B MiniHeader v2, MediaType, u32 seq, 8-bit CodecID | ✅ Complete (T1.x) |
|
||||
| V2 — Transport additions | BWE, NACK loop, TransportFeedback, dynamic FEC boost on I-frames | 🔲 Not started |
|
||||
| V3 — `wzp-video` crate | H.264 baseline framer/depacketizer, VideoToolbox/MediaCodec/dav1d encoders | ✅ Substantially complete (T4.x, T5.x, T6.x) |
|
||||
| V3 — H.264 Baseline | Single-layer H.264 | ✅ Complete |
|
||||
| V3 — H.265 | VideoToolbox + MediaCodec H.265 | ✅ Complete (T5.x) |
|
||||
| V3 — AV1 | dav1d + SVT-AV1 (non-Android), VideoToolbox AV1 (macOS M3+) | ✅ Complete; Android MediaCodec AV1 compile errors pending (T4.3.1.1) |
|
||||
| V3 — Android MediaCodec | NDK 0.9 API migration for `mediacodec.rs` | 🔴 Blocked (31 compile errors) |
|
||||
| V3 — Call engine wiring | `create_video_encoder()` integrated into active call negotiation | 🔴 Not started (T6.1.2 follow-up) |
|
||||
| V4 — Keyframe & loss policy | NACK path, PLI, keyframe cache at SFU | 🟡 Framework present (`nack.rs`); not wired |
|
||||
| V5 — Video adaptive controller | `VideoQualityController` + `PriorityMode` | 🟡 Controller built (`controller.rs`); not wired into call |
|
||||
| V5 — Simulcast | Simulcast layer management | 🟡 `simulcast.rs` present; not wired |
|
||||
| V6 — SFU changes | Keyframe cache, per-receiver layer selection, PLI suppression | 🟡 PLI suppression wired; keyframe cache + layer selection not started |
|
||||
| V6 — Video scorer | `VideoScorer` legitimacy detection | 🟡 Built (`video_scorer.rs`); `observe()` not wired into room forwarding |
|
||||
| V7 — Capture pipeline | Camera capture (AVCaptureSession, Camera2, NVENC) | 🔲 Not started |
|
||||
|
||||
**Legend:** ✅ Complete · 🟡 Partial/Framework only · 🔴 Blocked · 🔲 Not started
|
||||
|
||||
### Critical path to first video call
|
||||
|
||||
1. Fix Android MediaCodec compile errors (T4.3.1.1) — ~2h
|
||||
2. Wire `create_video_encoder()` into call engine codec negotiation (T6.1.2) — ~2h
|
||||
3. Fix crypto nonce bug (`decrypt()` must use `MediaHeader.seq`) — see `AUDIT-2026-05-25.md` C1 — ~1h
|
||||
4. Wire `VideoScorer::observe()` into relay room forwarding (T6.2 follow-up) — ~2h
|
||||
5. Implement Phase V2 BWE (mandatory for usable video) — ~3–4 days
|
||||
6. Implement capture pipeline for at least one platform (V7) — ~1 week
|
||||
|
||||
## Phase V1 — Wire format & negotiation (no new code paths yet)
|
||||
|
||||
Bump protocol version. Land all wire changes together so compat breaks exactly once.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
> Distilled from `docs/ARCHITECTURE.md` and the `wzp-proto` crate. Authoritative wire details live in `crates/wzp-proto/src/packet.rs`.
|
||||
>
|
||||
> **Status:** v1 (audio-only) is the deployed protocol. v2 (audio + video, 16 B header, MediaType, u32 seq, etc.) is specified in `ROAD-TO-VIDEO.md` Phase V1 and supersedes this document when implemented.
|
||||
> **Status:** v2 is the deployed protocol (audio + video, 16 B header, MediaType, u32 seq). v1 clients are rejected with `Hangup::ProtocolVersionMismatch`.
|
||||
|
||||
## Layer summary
|
||||
|
||||
@@ -16,42 +16,47 @@
|
||||
| Loss recovery | **RaptorQ FEC + Opus DRED + classical PLC** | NACK / PLI + reference-picture selection |
|
||||
| Adaptive | 3-tier hysteresis (Good / Degraded / Catastrophic) + continuous DRED tuner | Per-frame bitrate ladder |
|
||||
| Topology | SFU rooms + inter-relay federation + P2P via ICE | Mesh ≤ ~3, SFU above, Apple relays |
|
||||
| Header | 12 B `MediaHeader` / 4 B `MiniHeader` (49 of 50), 4 B `QualityReport` trailer | RTP 12 B + extensions |
|
||||
| Header | 16 B `MediaHeader` v2 / 5 B `MiniHeader` (49 of 50), 4 B `QualityReport` trailer | RTP 12 B + extensions |
|
||||
|
||||
## Distinctive choices
|
||||
|
||||
- **QUIC datagrams instead of raw UDP + SRTP.** Brings TLS 1.3, PLPMTUD, path migration, and ACK-based RTT/loss estimation for free.
|
||||
- **Continuous DRED tuning.** Maps live `(loss%, RTT, jitter)` to a continuous Opus DRED lookback window. Most stacks treat DRED as discrete tiers.
|
||||
- **MiniHeader (4 B for 49/50 packets).** Saves ~8 B/packet ≈ 400 B/s/stream at 50 pps.
|
||||
- **MiniHeader (5 B for 49/50 packets).** Saves ~11 B/packet ≈ 550 B/s/stream at 50 pps vs. the full 16 B header.
|
||||
- **E2E-preserving SFU.** The relay forwards encrypted datagrams; it never decrypts media. Room membership uses SNI = `hash(room_name)`.
|
||||
- **Codec coordination via `QualityReport` trailer.** Receivers attach 4-byte loss/RTT/jitter/cap to media packets; the SFU broadcasts `QualityDirective` so all senders in a room converge on the same tier.
|
||||
|
||||
## Wire format (current — v1)
|
||||
## Wire format (current — v2)
|
||||
|
||||
### `MediaHeader` (12 bytes)
|
||||
### `MediaHeader` v2 (16 bytes, byte-aligned)
|
||||
|
||||
```
|
||||
Byte 0: [V:1][T:1][CodecID:4][Q:1][FecRatioHi:1]
|
||||
Byte 1: [FecRatioLo:6][unused:2]
|
||||
Bytes 2-3: sequence (u16 BE)
|
||||
Bytes 4-7: timestamp_ms (u32 BE)
|
||||
Byte 8: fec_block_id (u8)
|
||||
Byte 9: fec_symbol_idx (u8)
|
||||
Byte 10: reserved
|
||||
Byte 11: csrc_count
|
||||
Byte 0: version (u8) 0x02
|
||||
Byte 1: flags (u8) [T:1][Q:1][KeyFrame:1][FrameEnd:1][reserved:4]
|
||||
Byte 2: media_type (u8) 0=audio, 1=video, 2=data, 3=control
|
||||
Byte 3: codec_id (u8) 0-255 (see codec table)
|
||||
Byte 4: stream_id (u8) simulcast layer; 0=base
|
||||
Byte 5: fec_ratio (u8) 0..200 → 0.0..2.0
|
||||
Bytes 6-9: sequence (u32 BE)
|
||||
Bytes 10-13: timestamp_ms (u32 BE)
|
||||
Bytes 14-15: fec_block_id (u16 BE)
|
||||
```
|
||||
|
||||
| Field | Bits | Meaning |
|
||||
|---|---|---|
|
||||
| V | 1 | Protocol version |
|
||||
| T | 1 | 1 = FEC repair packet |
|
||||
| CodecID | 4 | See codec table |
|
||||
| Q | 1 | QualityReport trailer present |
|
||||
| FecRatio | 7 | 0–127 → 0.0–2.0 |
|
||||
| sequence | 16 | Wrapping packet seq |
|
||||
| version | 8 | Must be `0x02`; v1 clients receive `Hangup::ProtocolVersionMismatch` |
|
||||
| T (bit 7 of flags) | 1 | 1 = FEC repair packet |
|
||||
| Q (bit 6 of flags) | 1 | QualityReport trailer present |
|
||||
| KeyFrame (bit 5 of flags) | 1 | Packet belongs to a video I-frame |
|
||||
| FrameEnd (bit 4 of flags) | 1 | Last packet of an access unit |
|
||||
| reserved (bits 3-0 of flags) | 4 | Must be zero |
|
||||
| media_type | 8 | 0=audio, 1=video, 2=data, 3=control |
|
||||
| codec_id | 8 | See codec table (widened from v1's 4-bit field) |
|
||||
| stream_id | 8 | Simulcast layer; 0=base layer |
|
||||
| fec_ratio | 8 | 0..200 → 0.0..2.0 |
|
||||
| sequence | 32 | Monotonically increasing packet seq (not reset by rekey) |
|
||||
| timestamp_ms | 32 | ms since session start. Monotonic across the full session; **not reset by rekey** |
|
||||
| fec_block_id | 8 | FEC source block ID |
|
||||
| fec_symbol_idx | 8 | Symbol index in block |
|
||||
| fec_block_id | 16 | FEC source block ID |
|
||||
|
||||
### Codec table
|
||||
|
||||
@@ -66,13 +71,18 @@ Byte 11: csrc_count
|
||||
| 6 | Opus 32k | 32 kbps | 48 kHz | 20 ms |
|
||||
| 7 | Opus 48k | 48 kbps | 48 kHz | 20 ms |
|
||||
| 8 | Opus 64k | 64 kbps | 48 kHz | 20 ms |
|
||||
| 9 | H.264 Baseline | — | — | — |
|
||||
| 10 | H.264 Main | — | — | — |
|
||||
| 11 | H.265 Main | — | — | — |
|
||||
| 12 | AV1 Main | — | — | — |
|
||||
|
||||
### `MiniHeader` (4 bytes, compressed — 49 of every 50 packets)
|
||||
### `MiniHeader` v2 (5 bytes, compressed — 49 of every 50 packets)
|
||||
|
||||
```
|
||||
[FRAME_TYPE_MINI = 0x01]
|
||||
Bytes 0-1: timestamp_delta_ms (u16 BE)
|
||||
Bytes 2-3: payload_len (u16 BE)
|
||||
Byte 0: seq_delta (u8)
|
||||
Bytes 1-2: timestamp_delta_ms (u16 BE)
|
||||
Bytes 3-4: payload_len (u16 BE)
|
||||
```
|
||||
|
||||
Full header sent every 50th packet to resync.
|
||||
@@ -95,6 +105,12 @@ Byte 2: jitter_ms (0-255 ms)
|
||||
Byte 3: bitrate_cap_kbps (0-255 kbps)
|
||||
```
|
||||
|
||||
### Version negotiation
|
||||
|
||||
- `version=0x02` in `MediaHeader` is a hard switch — there is no fallback negotiation.
|
||||
- Both endpoints must speak v2. A v1 peer receives `Hangup::ProtocolVersionMismatch` immediately.
|
||||
- Relays inspect only `version` and `media_type`; they never downgrade or translate between versions.
|
||||
|
||||
## Session lifecycle
|
||||
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user