docs: protocol audit 2026-05-25, update architecture + Obsidian vault
Audit: - docs/AUDIT-2026-05-25.md: full protocol audit covering 8 findings (4 critical, 2 high, 5 medium, 4 low) with code references and fix effort estimates - vault/Audit/Tasks.md: Obsidian Tasks plugin file tracking all audit items with priorities, due dates, and per-step checklists Architecture docs updated for Wire format v2 and Wave 5/6 features: - ARCHITECTURE.md: adds wzp-video to dependency graph and project structure; wire format updated to v2 (16B header, 5B MiniHeader); relay concurrency section corrected (DashMap+RwLock is current, not a future optimization); test count 571→702; Android note - PROGRESS.md: Wave 5 and Wave 6 sections appended; test count 372→702; current status and open blockers as of 2026-05-25 - ROAD-TO-VIDEO.md: implementation status table inserted (✅/🟡/🔴/🔲 per phase); 6-step critical path to first video call - WZP-SPEC.md: MediaHeader updated to v2 (16B byte-aligned); MiniHeader updated to 5B with seq_delta; codec IDs 9-12 added (H.264/H.265/AV1); version negotiation section added Obsidian vault (vault/): - 114 files across Architecture/, PRDs/, Reports/, Android/, Reference/, Audit/ with YAML frontmatter - 00 - Home.md index note with wiki links - .obsidian/app.json config Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Siavash Sameni
226
vault/PRDs/PRD-coordinated-codec.md
Normal file
226
vault/PRDs/PRD-coordinated-codec.md
Normal file
@@ -0,0 +1,226 @@
|
||||
---
|
||||
tags: [prd, wzp]
|
||||
type: prd
|
||||
---
|
||||
|
||||
# PRD: Coordinated Codec Switching (Relay-Judged Quality)
|
||||
|
||||
## Problem
|
||||
|
||||
The current adaptive quality system (`QualityAdapter` in call.rs) exists but isn't wired into either engine. Clients encode at a fixed quality chosen at call start. When network conditions change mid-call, audio degrades instead of gracefully stepping down. When conditions improve, clients stay on low quality unnecessarily.
|
||||
|
||||
Additionally, in SFU mode with multiple participants, uncoordinated codec switching creates asymmetry: if client A upgrades to 64k while B stays on 24k, bandwidth is wasted. Participants should switch together.
|
||||
|
||||
## Solution
|
||||
|
||||
The **relay acts as the quality judge** since it sees both sides of every connection. It monitors packet loss, jitter, and RTT per participant, then signals quality recommendations. Clients react to these signals with coordinated codec switches.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────┐ ┌─────────┐ ┌─────────┐
|
||||
│ Client A │◄──────►│ Relay │◄──────►│ Client B │
|
||||
│ │ │ (judge) │ │ │
|
||||
│ Encoder │ │ │ │ Encoder │
|
||||
│ Decoder │ │ Monitor │ │ Decoder │
|
||||
└─────────┘ │ per-peer│ └─────────┘
|
||||
│ quality │
|
||||
└────┬────┘
|
||||
│
|
||||
Quality Signals:
|
||||
- StableSignal (conditions good)
|
||||
- DegradeSignal (conditions bad)
|
||||
- UpgradeProposal (try higher quality?)
|
||||
- UpgradeConfirm (all agreed, switch at T)
|
||||
```
|
||||
|
||||
## Quality Classification (Relay-Side)
|
||||
|
||||
The relay monitors each participant's connection quality:
|
||||
|
||||
| Condition | Classification | Action |
|
||||
|-----------|---------------|--------|
|
||||
| loss >= 15% OR RTT >= 200ms | Critical | Immediate downgrade signal |
|
||||
| loss >= 5% OR RTT >= 100ms | Degraded | Downgrade signal after 3 reports |
|
||||
| loss < 2% AND RTT < 80ms | Good | Stable signal |
|
||||
| loss < 1% AND RTT < 50ms for 30s | Excellent | Upgrade proposal |
|
||||
| loss < 0.5% AND RTT < 30ms for 60s | Studio | Studio upgrade proposal |
|
||||
|
||||
## Coordinated Switching Protocol
|
||||
|
||||
### Downgrade (fast, safety-first)
|
||||
|
||||
1. Relay detects degradation for ANY participant
|
||||
2. Relay sends `QualityUpdate { recommended_profile: DEGRADED }` to ALL participants
|
||||
3. ALL participants immediately switch encoder to the recommended profile
|
||||
4. No negotiation — downgrade is mandatory and instant
|
||||
|
||||
### Upgrade (slow, consensual)
|
||||
|
||||
1. Relay detects sustained good conditions for ALL participants (threshold: 30s stable)
|
||||
2. Relay sends `UpgradeProposal { target_profile, switch_timestamp }` to all
|
||||
3. Each client responds: `UpgradeAccept` or `UpgradeReject`
|
||||
4. If ALL accept within 5s → Relay sends `UpgradeConfirm { profile, switch_at_ms }`
|
||||
5. All clients switch encoder at the agreed timestamp (relative to session clock)
|
||||
6. If ANY rejects or times out → upgrade cancelled, stay on current profile
|
||||
|
||||
### Asymmetric Encoding (SFU optimization)
|
||||
|
||||
In SFU mode, each client encodes independently. The relay could allow:
|
||||
- Client A (strong connection): encode at 64k
|
||||
- Client B (weak connection): encode at 6k
|
||||
- Relay forwards A's 64k to B's decoder (auto-switch handles it)
|
||||
- B benefits from A's quality without needing to send at 64k
|
||||
|
||||
This requires NO protocol changes — just each client independently following the relay's recommendation for their own encoding quality. The decoder already handles any codec.
|
||||
|
||||
### Split Network Consideration
|
||||
|
||||
If participant A has great quality but participant C has terrible quality:
|
||||
- Option 1: **Match weakest link** — everyone encodes at C's level (current approach, simple)
|
||||
- Option 2: **Per-participant recommendations** — A encodes at 64k, C encodes at 6k. B (good connection) receives and decodes both. Works because decoders auto-switch per packet.
|
||||
- Option 3: **Relay transcoding** — relay re-encodes A's 64k as 6k for C. Adds CPU on relay, but saves bandwidth for C. Future feature.
|
||||
|
||||
Recommended: start with Option 1 (match weakest), add Option 2 later.
|
||||
|
||||
## Signal Messages (New/Modified)
|
||||
|
||||
```rust
|
||||
/// Quality signal from relay to client
|
||||
QualityDirective {
|
||||
/// Recommended profile to use for encoding
|
||||
recommended_profile: QualityProfile,
|
||||
/// Reason for the recommendation
|
||||
reason: QualityReason,
|
||||
}
|
||||
|
||||
enum QualityReason {
|
||||
/// Network conditions require this quality level
|
||||
NetworkCondition,
|
||||
/// Coordinated upgrade — all participants agreed
|
||||
CoordinatedUpgrade,
|
||||
/// Coordinated downgrade — weakest link determines level
|
||||
CoordinatedDowngrade,
|
||||
}
|
||||
|
||||
/// Upgrade proposal from relay
|
||||
UpgradeProposal {
|
||||
target_profile: QualityProfile,
|
||||
/// Milliseconds from now when the switch would happen
|
||||
switch_delay_ms: u32,
|
||||
}
|
||||
|
||||
/// Client response to upgrade proposal
|
||||
UpgradeResponse {
|
||||
accepted: bool,
|
||||
}
|
||||
|
||||
/// Confirmed upgrade — all clients switch at this time
|
||||
UpgradeConfirm {
|
||||
profile: QualityProfile,
|
||||
/// Session-relative timestamp to switch (ms since call start)
|
||||
switch_at_session_ms: u64,
|
||||
}
|
||||
```
|
||||
|
||||
## Relay-Side Implementation
|
||||
|
||||
### Per-Participant Quality Tracking
|
||||
|
||||
```rust
|
||||
struct ParticipantQuality {
|
||||
/// Sliding window of recent observations
|
||||
loss_samples: VecDeque<f32>, // last 30 seconds
|
||||
rtt_samples: VecDeque<u32>, // last 30 seconds
|
||||
jitter_samples: VecDeque<u32>,
|
||||
/// Current classification
|
||||
classification: QualityClass,
|
||||
/// How long current classification has been stable
|
||||
stable_since: Instant,
|
||||
}
|
||||
```
|
||||
|
||||
### Quality Monitor Task (on relay)
|
||||
|
||||
Runs alongside the SFU forwarding loop:
|
||||
1. Every 1 second, compute per-participant quality from QUIC connection stats
|
||||
2. Classify each participant
|
||||
3. If ANY participant degrades → send downgrade to ALL
|
||||
4. If ALL participants stable for threshold → propose upgrade
|
||||
5. Track upgrade negotiation state
|
||||
|
||||
### Integration with Existing Code
|
||||
|
||||
The relay already has access to:
|
||||
- `QuinnTransport::path_quality()` → loss, RTT, jitter, bandwidth estimates
|
||||
- `QualityReport` embedded in media packet headers
|
||||
- Per-session metrics in `RelayMetrics`
|
||||
|
||||
The quality monitor just needs to read these existing metrics and produce signals.
|
||||
|
||||
## Client-Side Implementation
|
||||
|
||||
### Handling Quality Signals
|
||||
|
||||
In the recv loop (both Android engine and desktop engine):
|
||||
```rust
|
||||
SignalMessage::QualityDirective { recommended_profile, .. } => {
|
||||
// Immediate: switch encoder to recommended profile
|
||||
encoder.set_profile(recommended_profile)?;
|
||||
fec_enc = create_encoder(&recommended_profile);
|
||||
frame_samples = frame_samples_for(&recommended_profile);
|
||||
info!(codec = ?recommended_profile.codec, "quality directive: switched");
|
||||
}
|
||||
```
|
||||
|
||||
### P2P Quality (simpler case)
|
||||
|
||||
For P2P calls (no relay), both clients directly observe quality:
|
||||
1. Each client runs its own `QualityAdapter` on the direct connection
|
||||
2. When quality changes, client proposes to peer via signal
|
||||
3. Simpler negotiation: only 2 parties, no relay middleman
|
||||
4. Same coordinated switching logic, just peer-to-peer signals
|
||||
|
||||
## Backporting P2P → Relay
|
||||
|
||||
The quality monitoring and codec switching logic is identical:
|
||||
- **P2P**: client observes quality directly → proposes switch to peer
|
||||
- **Relay**: relay observes quality → proposes switch to all clients
|
||||
|
||||
The only difference is WHO makes the decision (client vs relay) and HOW many participants need to agree (2 vs N).
|
||||
|
||||
Implementation strategy: build for P2P first (simpler, 2 parties), then wrap the same logic with relay-mediated signals for SFU mode.
|
||||
|
||||
## Milestones
|
||||
|
||||
| Phase | Scope | Effort |
|
||||
|-------|-------|--------|
|
||||
| 1 | Relay-side quality monitor (per-participant tracking) | 1 day |
|
||||
| 2 | Downgrade signal (immediate, match weakest) | 1 day |
|
||||
| 3 | Client handling of QualityDirective | 1 day (both engines) |
|
||||
| 4 | Upgrade proposal + negotiation protocol | 2 days |
|
||||
| 5 | P2P quality adaptation (direct observation) | 1 day |
|
||||
| 6 | Per-participant asymmetric encoding (Option 2) | 1 day |
|
||||
|
||||
## Implementation Status (2026-04-13)
|
||||
|
||||
Phases 1-2 are implemented. Phase 3 has a critical gap.
|
||||
|
||||
### What was built
|
||||
|
||||
- **`QualityDirective` signal** (`crates/wzp-proto/src/packet.rs`): New `SignalMessage` variant with `recommended_profile` and optional `reason`
|
||||
- **`ParticipantQuality`** (`crates/wzp-relay/src/room.rs`): Per-participant quality tracking using `AdaptiveQualityController`, created on join, removed on leave
|
||||
- **Weakest-link broadcast**: `observe_quality()` method computes room-wide worst tier, broadcasts `QualityDirective` to all participants when tier changes
|
||||
- **Desktop engine handling** (`desktop/src-tauri/src/engine.rs`): `AdaptiveQualityController` in recv task, `pending_profile` AtomicU8 bridge to send task, auto-mode profile switching based on **inbound quality reports**
|
||||
|
||||
### Phase 3 completed (2026-04-13)
|
||||
|
||||
Both engines now handle `QualityDirective` signals from the relay:
|
||||
- **Desktop** (`engine.rs`): both P2P and relay signal tasks match `QualityDirective`, extract `recommended_profile`, store index via `sig_pending_profile.store(idx, Release)`. Send task picks it up at the next frame boundary.
|
||||
- **Android** (`engine.rs`): signal task matches `QualityDirective`, stores via `pending_profile_recv.store(idx, Release)`.
|
||||
|
||||
Relay-coordinated codec switching is now end-to-end: relay monitors → broadcasts directive → clients switch.
|
||||
|
||||
### Phase remaining
|
||||
|
||||
- Phase 4: Upgrade proposal/negotiation protocol for quality recovery (task #28)
|
||||
Reference in New Issue
Block a user