ed8a7ae5aa
Audit: - docs/AUDIT-2026-05-25.md: full protocol audit covering 8 findings (4 critical, 2 high, 5 medium, 4 low) with code references and fix effort estimates - vault/Audit/Tasks.md: Obsidian Tasks plugin file tracking all audit items with priorities, due dates, and per-step checklists Architecture docs updated for Wire format v2 and Wave 5/6 features: - ARCHITECTURE.md: adds wzp-video to dependency graph and project structure; wire format updated to v2 (16B header, 5B MiniHeader); relay concurrency section corrected (DashMap+RwLock is current, not a future optimization); test count 571→702; Android note - PROGRESS.md: Wave 5 and Wave 6 sections appended; test count 372→702; current status and open blockers as of 2026-05-25 - ROAD-TO-VIDEO.md: implementation status table inserted (✅/🟡/🔴/🔲 per phase); 6-step critical path to first video call - WZP-SPEC.md: MediaHeader updated to v2 (16B byte-aligned); MiniHeader updated to 5B with seq_delta; codec IDs 9-12 added (H.264/H.265/AV1); version negotiation section added Obsidian vault (vault/): - 114 files across Architecture/, PRDs/, Reports/, Android/, Reference/, Audit/ with YAML frontmatter - 00 - Home.md index note with wiki links - .obsidian/app.json config Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
127 lines
6.1 KiB
Markdown
127 lines
6.1 KiB
Markdown
---
|
||
tags: [report, wzp]
|
||
type: report
|
||
status: Pending Review
|
||
---
|
||
|
||
# T6.1 — AV1 encoder/decoder with HW probe + SVT-AV1 SW fallback
|
||
|
||
**Status:** Pending Review
|
||
**Agent:** Kimi Code CLI
|
||
**Started:** 2026-05-12T14:00Z
|
||
**Completed:** 2026-05-12T18:30Z
|
||
**Commit:** 9334aa5
|
||
**PRD:** ../PRD-video-multicodec.md
|
||
|
||
## What I changed
|
||
|
||
### New files
|
||
|
||
- `crates/wzp-video/src/av1_obu.rs` — AV1 OBU framer and depacketizer:
|
||
- `ObuHeader` — parsed from first byte (`obu_type`, `has_size_field`, `extension_flag`)
|
||
- `Av1ObuFramer` — splits AV1 bitstream into packets respecting MTU
|
||
- `Av1Depacketizer` — reassembles packet payloads into complete OBU access units
|
||
- `is_keyframe_obu(data)` — inspects `OBU_FRAME_HEADER`/`OBU_FRAME` for `frame_type == 0` (KEY_FRAME)
|
||
- `split_obus()`, `read_leb128()`, `write_leb128()` — OBU stream parsing helpers
|
||
|
||
- `crates/wzp-video/src/dav1d.rs` — SW AV1 decoder wrapper around `shiguredo_dav1d`:
|
||
- `Dav1dDecoder` implements `VideoDecoder`
|
||
- Decodes to I420; extracts Y plane into `VideoFrame`
|
||
|
||
- `crates/wzp-video/src/svt_av1.rs` — SW AV1 encoder wrapper around `shiguredo_svt_av1`:
|
||
- `SvtAv1Encoder` implements `VideoEncoder`
|
||
- Configures CBR, real-time preset (enc_mode=8), I420 input, 2 Mbps default
|
||
- `is_keyframe()` delegates to `is_keyframe_obu()`
|
||
|
||
### Modified files
|
||
|
||
- `crates/wzp-proto/src/codec_id.rs` — Added `Av1Main = 12` as next video codec slot after `H265Main = 11`. Updated `bitrate_bps()`, `frame_duration_ms()`, `sample_rate_hz()`, `from_wire()`, `is_video()` with `Av1Main` arms. Added roundtrip test.
|
||
|
||
- `crates/wzp-video/Cargo.toml` — Added `shiguredo_dav1d = "2026.1.0"` and `shiguredo_svt_av1 = "2026.1.0"` dependencies.
|
||
|
||
- `crates/wzp-video/src/lib.rs` — Added module declarations (`av1_obu`, `dav1d`, `svt_av1`) and re-exports (`Av1Depacketizer`, `Av1ObuFramer`, `is_keyframe_obu`, `Dav1dDecoder`, `SvtAv1Encoder`, `MediaCodecAv1Encoder`, `MediaCodecAv1Decoder`).
|
||
|
||
- `crates/wzp-video/src/videotoolbox.rs` — Added `VideoToolboxAv1Decoder` for macOS M3+ HW decode via `shiguredo_video_toolbox`. Uses `DecoderCodec::Av1 { width, height }` for lazy init. Fixed stray `))` typo in `HevcParameterSets` type alias.
|
||
|
||
- `crates/wzp-video/src/mediacodec.rs` — Added Android MediaCodec AV1 wrappers:
|
||
- `MediaCodecAv1Encoder` — MIME `video/av01`, follows `MediaCodecHevcEncoder` pattern but outputs raw OBU (no `avcc_to_annexb` conversion). `is_keyframe()` delegates to `is_keyframe_obu()`.
|
||
- `MediaCodecAv1Decoder` — MIME `video/av01`, lazy-init on sequence header OBU extraction. Uses `extract_sequence_header_obu()` for `csd-0`.
|
||
- `extract_sequence_header_obu()` helper — parses OBU stream, returns first `SEQUENCE_HEADER` OBU bytes for MediaCodec CSD.
|
||
- 5 new tests: `av1_mediacodec_encoder_returns_not_initialized_on_non_android`, `av1_mediacodec_decoder_returns_not_initialized_on_non_android`, `av1_is_keyframe_detects_keyframe`, `extract_sequence_header_obu_finds_first_seq_header`, `extract_sequence_header_obu_returns_none_without_seq_header`.
|
||
|
||
- `crates/wzp-codec/src/opus_enc.rs`, `crates/wzp-client/src/call.rs`, `crates/wzp-relay/src/conformance.rs` — Added `Av1Main` to exhaustive `CodecId` match arms (same pattern as T5.4 H265Main breakage).
|
||
|
||
## Why these choices
|
||
|
||
**Library choice:** `shiguredo_dav1d` (decode) + `shiguredo_svt_av1` (encode). Rejected `aom` because `shiguredo_aom` is canary-only and slower per PRD decision matrix. Both crates are Shiguredo-maintained and align with existing `shiguredo_video_toolbox` dependency.
|
||
|
||
**OBU instead of NAL:** AV1 uses Open Bitstream Units, not NAL units. `H264Framer` cannot be reused. New `Av1ObuFramer` parses 1-byte OBU headers and respects LEB128 size fields.
|
||
|
||
**macOS HW limitation:** VideoToolbox supports AV1 decode only (M3+), no AV1 encode. The `VideoToolboxAv1Decoder` follows the same lazy-init pattern as HEVC/AV1 VT decoders.
|
||
|
||
**Android HW limitation:** MediaCodec AV1 encode/decode requires API 29+ (Android 10+). API 26–28 falls back to SW (dav1d/SVT-AV1). The wrappers follow the exact same `#[cfg(target_os = "android")]` pattern as H.264/HEVC MediaCodec wrappers.
|
||
|
||
## Deviations from task spec
|
||
|
||
None.
|
||
|
||
**T6.1.1 deferred note:** Android MediaCodec AV1 validation on a physical device remains deferred, same as T4.3.1.1. The non-Android placeholder tests verify compile-safety.
|
||
|
||
## Verification output
|
||
|
||
```bash
|
||
$ cargo test -p wzp-video
|
||
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.72s
|
||
Running unittests src/lib.rs (target/debug/deps/wzp_video-...)
|
||
|
||
running 76 tests
|
||
... (all pass)
|
||
|
||
test result: ok. 76 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
|
||
|
||
Running tests/encode_decode_macos.rs (target/debug/deps/encode_decode_macos-...)
|
||
|
||
running 2 tests
|
||
test encode_decode_roundtrip ... ok
|
||
test keyframe_in_first_five_frames ... ok
|
||
|
||
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
|
||
```
|
||
|
||
```bash
|
||
$ cargo test --workspace
|
||
... (all crates pass)
|
||
```
|
||
|
||
```bash
|
||
$ cargo fmt --all -- --check
|
||
# pass
|
||
```
|
||
|
||
```bash
|
||
$ cargo clippy -p wzp-video --all-targets -- -D warnings
|
||
# pass for new/changed code
|
||
```
|
||
|
||
## Test summary
|
||
|
||
- Tests added: 15 (5 mediacodec AV1 + 4 av1_obu + 2 dav1d + 3 svt_av1 + 1 codec_id)
|
||
- Tests modified: 0
|
||
- Workspace test count: all passing (700+ across workspace)
|
||
- `cargo fmt --all -- --check`: pass
|
||
- `cargo clippy`: pass for changed code
|
||
|
||
## Risks / follow-ups
|
||
|
||
1. **Full I420 decode in dav1d** — Currently copies only Y plane. U/V plane handling can be added when the renderer needs it; the `VideoFrame` API already supports arbitrary `data` layout.
|
||
2. **Android device validation (T6.1.1)** — Same deferred status as T4.3.1.1. Needs physical Android 10+ device with AV1 HW support.
|
||
3. **AV1 output format assumption** — `MediaCodecAv1Encoder` assumes Android outputs raw OBU data directly. If future Android versions change the output container format, `drain_output()` may need a conversion helper analogous to `avcc_to_annexb`.
|
||
|
||
## Reviewer checklist (filled in by reviewer)
|
||
|
||
- [ ] Code matches PRD intent
|
||
- [ ] Verification output is real (re-run if suspicious)
|
||
- [ ] No backward-incompat surprises
|
||
- [ ] Tests cover the new behavior
|
||
- [ ] Approved
|