T1.5: Migrate emit/parse sites to v2 wire format
This commit is contained in:
Siavash Sameni
111
docs/PRD/PRD-video-multicodec.md
Normal file
111
docs/PRD/PRD-video-multicodec.md
Normal file
@@ -0,0 +1,111 @@
|
||||
# PRD: Multi-Codec Video Negotiation (H.264 + H.265 + AV1)
|
||||
|
||||
> **Status:** proposed
|
||||
> **Resolves:** Road-to-video Phase V3 codec rollout; reserves `CodecID` slots 9–13.
|
||||
> **Depends on:** PRD #5 (video v1 working with H.264).
|
||||
|
||||
## Problem
|
||||
|
||||
H.264 baseline ships first because it has universal hardware encode coverage. H.265 offers ~30 % efficiency at equal quality and is now broadly supported in HW (Apple A10+, Snapdragon since ~2017, NVENC since GTX 9xx). AV1 is the long-term target but hardware encode is limited (Apple M3/A17+, Snapdragon 8 Gen 3+, RTX 40+).
|
||||
|
||||
We need codec negotiation so each session uses the best mutually-supported codec without manual configuration, and so we can roll AV1 in gated on real telemetry.
|
||||
|
||||
## Goals
|
||||
|
||||
- `CodecID` assignments for H.264 baseline (9), H.264 main (10), H.265 main (11), AV1 (12), VP9 reserved (13).
|
||||
- Capability declaration in `CallOffer.supported_codecs`.
|
||||
- Picker logic: highest mutually-supported codec from a deterministic preference cascade.
|
||||
- Hardware-encode detection at session start; refuse codecs requiring SW encode on battery-powered devices.
|
||||
- Existing framer/depacketizer reused — only the codec wrapper changes.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- New codecs beyond this list.
|
||||
- Per-receiver codec selection (one codec per stream for v1; could be revisited with simulcast).
|
||||
|
||||
## Design
|
||||
|
||||
### Codec capability declaration
|
||||
|
||||
```rust
|
||||
pub struct CodecCapability {
|
||||
pub codec_id: u8,
|
||||
pub max_resolution: (u16, u16),
|
||||
pub max_fps: u8,
|
||||
pub hardware: bool, // true if HW encode available
|
||||
}
|
||||
|
||||
pub struct CallOffer {
|
||||
...
|
||||
pub supported_codecs: Vec<CodecCapability>,
|
||||
}
|
||||
```
|
||||
|
||||
### Preference cascade
|
||||
|
||||
```
|
||||
preference: [AV1, H.265 main, H.264 main, H.264 baseline]
|
||||
|
||||
pick = first codec in `preference` where:
|
||||
caller.supported.contains(codec)
|
||||
AND callee.supported.contains(codec)
|
||||
AND (codec.hardware on both sides OR codec.allow_software)
|
||||
```
|
||||
|
||||
`allow_software` defaults to `false` for AV1 (battery cost too high), `true` for H.264 (cheap SW fallback).
|
||||
|
||||
### Per-codec details
|
||||
|
||||
| ID | Codec | Encoder priority |
|
||||
|---|---|---|
|
||||
| 9 | H.264 baseline | VideoToolbox / MediaCodec / NVENC / QSV / AMF / VAAPI; OpenH264 SW |
|
||||
| 10 | H.264 main | Same HW; same SW |
|
||||
| 11 | H.265 main | VideoToolbox A10+ / MediaCodec / NVENC GTX 9xx+ / QSV Skylake+; x265 SW (slow, disabled by default) |
|
||||
| 12 | AV1 | VideoToolbox M3+/A17+ / MediaCodec SD8G3+ / NVENC RTX 40+; SVT-AV1 SW (gated) |
|
||||
| 13 | VP9 | Reserved; may not implement |
|
||||
|
||||
### Framer reuse
|
||||
|
||||
The 16 B `MediaHeader` carries `codec_id`. The framer doesn't care which codec — it fragments NALs (for H.264/H.265) or OBUs (for AV1) into MTU-sized chunks, sets `KeyFrame`/`FrameEnd` bits, and passes payload through. Per-codec parameter sets (SPS/PPS for H.264/H.265, sequence header OBU for AV1) ship on the signal stream.
|
||||
|
||||
### Mid-call codec switch
|
||||
|
||||
Optional in v1. If implemented:
|
||||
- Sender sends `SignalMessage::CodecSwitch { stream_id, new_codec_id, parameter_sets }`.
|
||||
- Receiver swaps decoder and emits PLI to force a clean keyframe.
|
||||
|
||||
## Implementation outline
|
||||
|
||||
1. `CodecCapability` declaration + serde (additive change).
|
||||
2. HW probe at session start (per platform).
|
||||
3. Picker logic in `CallOffer`/`CallAnswer` flow.
|
||||
4. H.265 encoder/decoder wrappers (VideoToolbox + MediaCodec).
|
||||
5. AV1 encoder/decoder wrappers, gated on HW (SVT-AV1 fallback behind flag).
|
||||
6. Prometheus: `wzp_session_codec_id_total{codec}` for telemetry on actual codec usage.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Two macOS clients (M1 + M3) pick H.265 by default; M3 + iPhone 15 Pro pick AV1.
|
||||
- M1 + Android device without H.265 HW picks H.264.
|
||||
- Codec selection is deterministic given both sides' capabilities.
|
||||
- AV1 refused on devices without HW unless `allow_software` flag explicitly set.
|
||||
|
||||
## Rollout gates
|
||||
|
||||
- H.264 baseline + main: ship with PRD #5.
|
||||
- H.265: enable by default once HW probe accuracy verified on 5+ macOS + 5+ Android devices.
|
||||
- AV1: 20 % of session-start probes must report HW encode capability before enabling by default. Until then, available only via debug flag.
|
||||
|
||||
## Risks
|
||||
|
||||
- **AV1 SW encode torches battery.** Mitigation: HW gate is mandatory; SW fallback off by default.
|
||||
- **H.265 patent surface.** Mitigation: rely on platform-provided HW encoders (license covered upstream); avoid shipping x265 binary.
|
||||
- **HW probe lies on some Android devices.** Mitigation: in-session fallback if encoder errors at start; degrade one codec tier.
|
||||
|
||||
## Effort
|
||||
|
||||
- H.265 wrappers: 3 d (T5.4)
|
||||
- AV1 wrappers + HW gate: 5 d (T6.1)
|
||||
- Picker + capability declaration: 1 d
|
||||
|
||||
Total: ~9 engineer-days, in Waves 5–6.
|
||||
Reference in New Issue
Block a user