133 lines
5.0 KiB
Markdown
133 lines
5.0 KiB
Markdown
# PRD: Video v1 — H.264 Single-Layer
|
||
|
||
> **Status:** proposed
|
||
> **Resolves:** Road-to-video Phases V3 + V4 (encoder/decoder, framer, NACK, keyframe cache).
|
||
> **Depends on:** PRD #1 (wire format v2), PRD #3 (TransportFeedback + BWE).
|
||
|
||
## Problem
|
||
|
||
WZP has no video path. Add a working unidirectional video call (macOS↔macOS first, then Android↔macOS) using H.264 baseline, with loss recovery appropriate for lossy mobile links.
|
||
|
||
## Goals
|
||
|
||
- New `wzp-video` crate parallel to `wzp-codec`.
|
||
- H.264 baseline encode/decode using platform hardware encoders.
|
||
- NAL fragmentation and access-unit reassembly conformant to our 16 B `MediaHeader` v2.
|
||
- NACK loop for P-frame loss (RTT-gated).
|
||
- Dynamic FEC ratio boost on I-frame packets.
|
||
- SFU keyframe cache for fast join-to-first-frame.
|
||
- PLI suppression at SFU to bound upstream keyframe-request traffic.
|
||
|
||
## Non-goals
|
||
|
||
- Multi-codec negotiation (PRD #6).
|
||
- Simulcast or per-receiver layer selection (PRD #8).
|
||
- VideoQualityController logic beyond a fixed bitrate target (PRD #7).
|
||
- Native camera capture pipelines (separate platform work).
|
||
|
||
## Design
|
||
|
||
### `wzp-video` crate
|
||
|
||
```
|
||
wzp-video/
|
||
src/
|
||
encoder.rs # trait VideoEncoder
|
||
# VideoToolboxEncoder (macOS)
|
||
# MediaCodecEncoder (Android, JNI)
|
||
# OpenH264Encoder (software fallback)
|
||
decoder.rs # trait VideoDecoder; mirror per-platform
|
||
framer.rs # H.264 NAL fragmentation to MTU-sized chunks
|
||
depacketizer.rs # Reassemble NALs, emit access units
|
||
keyframe.rs # Keyframe request handling, sender + receiver
|
||
config.rs # SPS/PPS shipment over signal stream
|
||
```
|
||
|
||
### Framing
|
||
|
||
One access unit (frame) → N packets, each ≤ `MTU - 16 (header) - 16 (AEAD tag)`.
|
||
|
||
- `sequence` global per (session, stream_id), advances per packet.
|
||
- `timestamp_ms` is presentation time, equal across all packets of a single access unit.
|
||
- `KeyFrame` bit set on every packet of an I-frame.
|
||
- `FrameEnd` bit set on the last packet of the access unit.
|
||
- `fec_block_id` per access unit (u16 in v2, large blocks).
|
||
|
||
Parameter sets (SPS/PPS) ride on the **signal stream**, not media datagrams. Sent at session start and on codec change. Reliable, ordered, one-time.
|
||
|
||
### NACK loop
|
||
|
||
```
|
||
SignalMessage::Nack {
|
||
version: u8,
|
||
stream_id: u8,
|
||
seqs: Vec<u32>, // missing P-frame packets
|
||
}
|
||
```
|
||
|
||
Receiver behavior:
|
||
- If access unit incomplete after `frame_interval` ms:
|
||
- If `RTT < 2 × frame_interval`: emit `Nack`.
|
||
- Else: emit `PictureLossIndication`.
|
||
- Backoff: max 1 Nack per (stream, seq) per 2 × RTT.
|
||
|
||
Sender behavior:
|
||
- On `Nack`: re-transmit if packet is still in send buffer (last 500 ms).
|
||
- On `PictureLossIndication`: emit a fresh I-frame within 200 ms.
|
||
|
||
### Dynamic FEC on I-frames
|
||
|
||
Encoder marks packets belonging to I-frames. FEC layer applies a higher ratio (default 0.5) to I-frame blocks, vs. nominal (0.1) for P-frames. Configurable.
|
||
|
||
### SFU keyframe cache
|
||
|
||
`RoomManager` maintains per `(room, sender, stream_id)`:
|
||
|
||
```rust
|
||
struct KeyframeCache {
|
||
packets: Vec<Bytes>, // most recent complete I-frame
|
||
timestamp_ms: u32,
|
||
sequence_first: u32,
|
||
}
|
||
```
|
||
|
||
On new participant join, cache is replayed before live forwarding starts. Eliminates 2 s black-screen-on-join.
|
||
|
||
Cache TTL: replaced whenever a new complete I-frame arrives.
|
||
|
||
### PLI suppression
|
||
|
||
If ≥ 2 receivers PLI within 200 ms for the same `(sender, stream_id)`, the SFU emits one `KeyframeRequest` upstream, not N. Tracked per-(sender, stream).
|
||
|
||
## Implementation outline
|
||
|
||
1. `wzp-video` crate scaffold (T4.1).
|
||
2. Framer/depacketizer with property tests (T4.1).
|
||
3. VideoToolbox encoder/decoder (macOS) (T4.2).
|
||
4. MediaCodec encoder/decoder (Android, JNI) (T4.3).
|
||
5. NACK signal + sender/receiver state machines (T4.4).
|
||
6. I-frame FEC ratio hint plumbed from encoder to FEC layer (T4.5).
|
||
7. SFU keyframe cache (T4.6).
|
||
8. PLI suppression (T4.7).
|
||
9. End-to-end test: macOS sender → relay → macOS receiver, 5 min call, < 1 % loss network.
|
||
|
||
## Acceptance criteria
|
||
|
||
- Unidirectional H.264 720p30 call macOS↔macOS, CPU < 5 % on M1.
|
||
- Android↔macOS works with MediaCodec (surface-texture path).
|
||
- Black-screen-on-join < 200 ms when keyframe cache is warm.
|
||
- Under 5 % synthetic packet loss at 50 ms RTT: NACK recovery keeps video smooth, < 1 keyframe / 2 s.
|
||
- Under 5 % synthetic packet loss at 300 ms RTT: PLI fallback fires, keyframe rate ~ 1 / s.
|
||
- Upstream PLI traffic at SFU < 2 / s under simulated mass packet loss with 8 receivers.
|
||
|
||
## Risks
|
||
|
||
- **MediaCodec surface-texture edge cases.** Per-device matrix; software fallback path mandatory.
|
||
- **VideoToolbox H.264 baseline restrictions** (some profiles are main-only in HW). Mitigation: profile detection at session start.
|
||
- **NACK storm under heavy loss.** Mitigation: rate cap (max 50 Nacks/s/receiver) and exponential backoff.
|
||
- **Keyframe cache memory footprint** (one I-frame per active stream per room). Mitigation: cap cache at 200 KB; if exceeded, drop and rely on PLI.
|
||
|
||
## Effort
|
||
|
||
~3 weeks (Wave 4 tasks T4.1–T4.7).
|