Teaches the relay pair to route direct-call signaling across an
existing federation link. Alice on Relay A can now place a direct
call to Bob on Relay B if A and B are federation peers — the
wire protocol, call registry, and signal dispatch all learn to
track and route the cross-relay flow.
Phase 3.5's dual-path QUIC race then carries the media directly
peer-to-peer using the advertised reflex addrs, with zero
changes needed on the client side.
## Wire protocol (wzp-proto)
New `SignalMessage::FederatedSignalForward { inner, origin_relay_fp }`
envelope variant, appended at end of enum — JSON serde is
name-tagged so pre-Phase-4 relays just log "unknown variant" and
drop it. 2 new roundtrip tests (any-inner nesting + single
DirectCallOffer case).
## Call registry (wzp-relay)
`DirectCall.peer_relay_fp: Option<String>` — federation TLS fp
of the peer relay that forwarded the offer/answer for this call.
`None` on local calls, `Some` on cross-relay. Used by the answer
path to route the reply back through the same federation link
instead of trying (and failing) to deliver via local signal_hub.
New `set_peer_relay_fp` setter + 1 new unit test.
## FederationManager (wzp-relay)
Three new methods:
- `local_tls_fp()` — exposes the relay's own federation TLS fp
so main.rs can build `origin_relay_fp` fields.
- `broadcast_signal(msg) -> usize` — fan out any signal message
(in practice `FederatedSignalForward`) to every active peer
link, returning the reach count. Used when Relay A doesn't
know which peer has the target fingerprint.
- `send_signal_to_peer(fp, msg)` — targeted send for the reply
path where the registry already knows which peer relay to
hit.
Plus a new `cross_relay_signal_tx: Mutex<Option<Sender<...>>>`
field that `set_cross_relay_tx()` wires at startup so the
federation `handle_signal` can push unwrapped inner messages
into the main signal dispatcher.
## Federation handle_signal (wzp-relay)
New match arm for `FederatedSignalForward`:
- Loop prevention: drops forwards whose `origin_relay_fp` equals
this relay's own fp (prevents A→B→A echo loops without needing
TTL yet).
- Otherwise pulls the inner message out and pushes it through
`cross_relay_signal_tx` so the main loop's dispatcher task
handles it as if it had arrived locally.
## Main signal loop (wzp-relay)
### DirectCallOffer when target not local
Before falling through to Hangup, try the federation path:
- Wrap the offer in `FederatedSignalForward` with
`origin_relay_fp = this relay's tls_fp`
- `fm.broadcast_signal(forward)` — returns peer count
- If any peers reached, stash the call in local registry with
`caller_reflexive_addr` set, `peer_relay_fp` still None
(broadcast — the answer-side will identify itself when it
replies)
- Send `CallRinging` to caller immediately for UX feedback
- Only if no federation or no peers → legacy Hangup path
### DirectCallAnswer when peer is remote
- Registry lookup now reads both `peer_fingerprint` and
`peer_relay_fp` in one acquisition
- If `peer_relay_fp.is_some()`:
* Reject → forward a `Hangup` over federation via
`send_signal_to_peer` instead of local signal_hub
* Accept → wrap the raw answer in `FederatedSignalForward`,
route to the specific origin peer, then emit the LOCAL
CallSetup to our callee with `peer_direct_addr =
caller_reflexive_addr` (caller is remote; this side only
has the callee)
- If `peer_relay_fp.is_none()` → existing Phase 3 same-relay
path with both CallSetups (caller + callee)
### Cross-relay signal dispatcher task
New long-running task reading `(inner, origin_relay_fp)` from
`cross_relay_rx`. In Phase 4 MVP handles:
- `DirectCallOffer` — if target is local, create the call in
the registry with `peer_relay_fp = origin_relay_fp`, stash
caller addr, deliver offer to local callee. If target isn't
local, drop (no multi-hop in Phase 4 MVP).
- `DirectCallAnswer` — look up local caller by call_id, stash
callee addr, forward raw answer to local caller via
signal_hub, emit local CallSetup with `peer_direct_addr =
callee_reflexive_addr` (peer is local now; this side only
has the caller).
- `CallRinging` — best-effort forward to local caller for UX.
- `Hangup` — logged for now; Phase 4.1 will target by call_id.
## Integration tests
`crates/wzp-relay/tests/cross_relay_direct_call.rs` — 3 tests
that reproduce the main.rs cross-relay dispatcher logic inline
and assert the invariants without spinning up real binaries:
1. `cross_relay_offer_forwards_and_stashes_peer_relay_fp` —
Relay A gets Alice's offer, broadcasts. Relay B's dispatcher
creates the call with `peer_relay_fp = relay_a_tls_fp`.
2. `cross_relay_answer_crosswires_peer_direct_addrs` — full
round trip; both CallSetups (one on each relay) carry the
OTHER party's reflex addr.
3. `cross_relay_loop_prevention_drops_self_sourced_forward` —
explicit loop-prevention check.
Full workspace test goes from 413 → 419 passing. Clippy clean
on touched files.
## Non-goals (deferred to Phase 4.1+)
- Relay-mediated media fallback across federation — if P2P
direct fails (symmetric NAT on either side), the call errors
out with "no media path". Making the existing federation
media pipeline carry ephemeral call-<id> rooms is the Phase
4.1 lift.
- Multi-hop federation (A → B → C). Phase 4 MVP supports a
direct federation link between A and B only.
- Fingerprint → peer-relay routing gossip.
PRD: .taskmaster/docs/prd_phase4_cross_relay_p2p.txt
Tasks: 70-78 all completed
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Addressed every rustc warning surfaced by \`cargo check --workspace
--release --lib --bins\` on opus-DRED-v2. Split across three
categories:
## Real bugs surfaced by the audit (fix, don't silence)
- **crates/wzp-relay/src/federation.rs** — the per-peer RTT monitor
task computed \`rtt_ms\` every 5 s and threw it on the floor. The
\`wzp_federation_peer_rtt_ms\` gauge has been registered in
metrics.rs the whole time but was never receiving samples, leaving
the Grafana panel blank. Wired it up: the task now calls
\`fm_rtt.metrics.federation_peer_rtt_ms.with_label_values(&[&label_rtt]).set(rtt_ms)\`
on every sample. Fixes three warnings (\`rtt_ms\`, \`fm_rtt\`,
\`label_rtt\` were all captured for this task and all dead).
## Dead code removal
- **crates/wzp-relay/src/federation.rs** — removed \`local_delivery_seq:
AtomicU16\` field and its initializer. It was described in comments
as "per-room seq counter for federation media delivered to local
clients" but was declared, initialized to 0, and never read or
written anywhere else. Genuine half-wired feature; deletable with
zero behavior change.
- **crates/wzp-relay/src/room.rs** — removed \`let recv_start =
Instant::now()\` at the top of a recv loop that was never read.
Separate variable \`last_recv_instant\` already measures the actual
gap that's used for the \`max_recv_gap_ms\` stat.
- **crates/wzp-client/src/cli.rs** — removed \`let my_fp = fp.clone()\`
from the signal loop setup. Cloned but never used in any match arm.
## Stub-intent warnings (underscore + explanatory comment)
- **crates/wzp-relay/src/handshake.rs** — \`choose_profile\` hardcodes
\`QualityProfile::GOOD\` and ignores its \`supported\` parameter.
Comment already documented "Cap at GOOD (24k) for now — studio
tiers not yet tested for federation reliability". Renamed to
\`_supported\`, expanded the comment to explicitly note the future
plan (pick highest supported ≤ relay ceiling).
- **crates/wzp-relay/src/federation.rs** — \`forward_to_peers\` takes
\`room_name: &str\` but only uses \`room_hash\`. The caller
(handle_datagram) passes the name for caller-site symmetry with
other helpers; kept the param shape and underscored the binding
with a comment noting it's reserved for future per-name logging.
## Cosmetic fixes
- **crates/wzp-relay/src/event_log.rs** — dropped \`use std::sync::Arc\`
(unused).
- **crates/wzp-relay/src/signal_hub.rs** — trimmed \`use tracing::{info,
warn}\` to \`use tracing::info\`. Also removed unnecessary \`mut\` on
\`hub\` binding in the \`register_unregister\` test.
- **crates/wzp-relay/src/room.rs** — trimmed \`use tracing::{debug,
error, info, trace, warn}\` to \`{error, info, warn}\`. Also removed
unnecessary \`mut\` on \`mgr\` binding in the \`room_join_leave\` test.
- **crates/wzp-relay/src/main.rs** — removed unnecessary \`mut\` on the
\`config\` destructured binding from \`parse_args()\`; and dropped
\`ref caller_alias\` from the \`DirectCallOffer\` match pattern since
the relay just forwards the full \`msg\` (caller_alias is preserved
end-to-end, we don't need to read it on the relay).
- **crates/wzp-crypto/tests/featherchat_compat.rs** — dropped
\`CallSignalType\` from a \`use wzp_client::featherchat::{...}\`
(unused in the test body). Note: this test file has pre-existing
compile errors from SignalMessage schema drift unrelated to this
sweep; that's tracked separately.
## Crate-level annotation
- **crates/wzp-android/src/lib.rs** — added
\`#![allow(dead_code, unused_imports, unused_variables, unused_mut)]\`
with a doc block explaining the crate is dead code since the Tauri
mobile rewrite. The legacy Kotlin+JNI Android app that consumed
this crate was replaced by desktop/src-tauri (live Android recv
path) + crates/wzp-native (Oboe bridge). Rather than piecemeal
cleanup of a crate that shouldn't be maintained, the whole-crate
allow keeps CI clean until someone removes the crate entirely. Kills
all 6 wzp-android warnings (4 unused imports/vars, 1 unused \`mut\`
on a JNI env param, 1 dead \`command_rx\` field) in one line.
## Not touched
- **deps/featherchat/warzone/crates/warzone-protocol/src/x3dh.rs** —
3 unused-variable warnings in \`alice_spk_secret\`, \`alice_bundle\`,
\`bob_bundle_bytes\`. This is a vendored third-party submodule;
upstream's problem, not ours. Would need to be reported to
featherchat upstream if we care.
## Verification
- \`cargo check --workspace --release --lib --bins\` → 0 warnings, 0 errors
- \`cargo check --workspace --release --all-targets\` → only the 3
featherchat submodule warnings remain, plus the pre-existing 3
broken integration tests (SignalMessage schema drift from Phase 2,
tracked separately and explicitly out of scope).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Time-based dedup (2s TTL) replaces fixed-window dedup — consecutive
senders with same seq numbers no longer collide
- Raw byte forwarding for federation local delivery (no re-serialization)
- Jitter buffer resets on large backward seq jumps (>100)
- recv_media skips malformed datagrams instead of returning connection-closed
- SIGTERM handler for clean QUIC shutdown on wzp-client
- JSONL event log infrastructure (--event-log flag) for protocol analysis
- FEC disabled on GOOD profile for federation debugging (fec_ratio=0.0)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Federation media from different senders had conflicting seq numbers,
FEC block IDs, and Opus decoder state. The relay now assigns fresh
monotonic seq/fec_block/fec_symbol to all federation-delivered packets,
ensuring clients see a clean continuous stream regardless of sender changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When propagating GlobalRoomActive to other peers, use tagged participants
(with relay_label set to the originating relay) instead of the raw
untagged participants. This shows "Relay C" instead of "Relay B" when
C's participants are forwarded through hub B to A.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Dedup key now includes source peer fingerprint hash, preventing
packets from different senders with same room+seq from being dropped
as duplicates (was silently killing all multi-hop audio)
- Build scripts default to --pull (use --no-pull to skip)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Deduplicate remote participants by fingerprint in all merge sites
(canonical == raw room name caused double-lookup, doubling every remote participant)
- GlobalRoomInactive now propagates updated participant list to other peers
(hub relay B was not informing A when C's participants left)
- Add 15-second stale presence sweeper that purges remote participants
from peers that stop sending data (safety net for QUIC timeout delays)
- Add @Synchronized to WzpEngine.getStats/stopCall/destroy to prevent
TOCTOU race between stats polling coroutine and engine teardown (SIGSEGV)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a new federation link is established, announce not only LOCAL
global rooms but also rooms from OTHER peers (remote_participants).
This fixes multi-hop: when R2 connects to R3, R2 tells R3 about
R1's rooms that R2 learned about earlier.
Previously, only local rooms were announced on link setup. If R1
had a client but R2 had no clients, R2 wouldn't tell R3 about R1.
Also added diagnostic logging for room announcements on link setup.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three fixes for 3-relay chain (R1→R2→R3):
1. Room lookup in handle_datagram: hub relay (R2) has no local
participants, so active_rooms() was empty and datagrams were
silently dropped. Now also checks global_rooms config directly,
allowing hub relays to forward without local clients.
2. Multi-hop forwarding: removed active_rooms filter — forward to
ALL connected peers except source. The receiving peer decides
whether to deliver or forward further.
3. Android relay_label: native RoomMember now includes relay_label
from RoomUpdate signal. Kotlin UI reads it for relay grouping.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a remote relay's room goes inactive (all participants left),
the receiving relay now:
1. Clears remote_participants for that peer+room
2. Broadcasts updated RoomUpdate to local clients with the remote
participant removed
3. Updates federation_active_rooms metric
Previously, remote participants lingered in the participant list
after disconnect, causing ghost entries and stale media forwarding.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
RoomParticipant.relay_label identifies which relay a participant is
connected to. Local participants have None, federated participants
get tagged with the peer relay's label when storing remote_participants.
This enables clients to group participants by relay in the UI.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. CLI client now sends raw room names (no hash), matching Android
JNI and Desktop Tauri. All three clients are now consistent.
2. When a client joins a global room, the relay merges federated
remote participants into the initial RoomUpdate. Previously,
clients that joined after the GlobalRoomActive signal only saw
local participants. Now they see everyone immediately.
3. Added get_remote_participants() to FederationManager for querying
cached remote participants from all peer links.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wires up the existing RelayMetrics federation fields:
- wzp_federation_peer_status{peer} — 1=connected, 0=disconnected
- wzp_federation_packets_forwarded_total{peer,direction} — in/out counts
- wzp_federation_active_rooms — number of active federated rooms
These are critical for monitoring federation health and will feed into
the adaptive codec selection system (PRD-coordinated-codec.md).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
GlobalRoomActive signal now carries participant list from the
announcing relay. When received, the relay:
1. Stores remote participants per peer link
2. Broadcasts merged RoomUpdate to local clients (local + all remote)
This means clients on different relays can now SEE each other in the
participant list. Also fixes build: removed non-existent metric field
references that were added by linter.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Different clients send different room names:
- Android: raw "general" as SNI
- Desktop: hash_room_name("general") = "f09ae11d..." as SNI
Federation datagrams are tagged with an 8-byte room hash. Previously,
each relay computed the hash from the client-provided room name,
causing mismatches between relays with different client types.
Fix: resolve_global_room() maps any room name (raw or hashed) to the
canonical [[global_rooms]] name. global_room_hash() always uses the
canonical name for federation hashing. handle_datagram uses both raw
and canonical hash matching to find the local room.
Also: run_participant now receives the pre-computed federation_room_hash
so the egress uses the canonical hash, not the client-specific name.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The bug: when a local client joins a global room and sends media, the
egress task checked peer_links.active_rooms to decide where to forward.
But active_rooms tracks what PEERS announced (their rooms), not what
WE announced. So our own GlobalRoomActive signal went out but our
peer_links had empty active_rooms — media was dropped.
Fix: for locally-originated media, send to ALL connected federation
peers unconditionally. The receiving relay decides whether to deliver
to local participants (if it has the room) or forward further. This
is correct because federation peers are explicitly configured — if
they're connected, they should receive global room media.
Multi-hop forwarding (handle_datagram) still filters by active_rooms
to prevent loops — only forwards to peers that announced the room.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Added logging to trace federation media flow:
- media_task logs first + every 250th received datagram (count, len)
- handle_datagram multi-hop forward logs errors (was silently dropped)
- forward_to_peers logs when no peer matches
2-relay (A→B): WORKING — full audio received, 300 packets forwarded
3-relay (A→B→C): B receives datagrams from A but only 1 arrives —
remaining packets not received, likely a QUIC read_datagram issue
when handle_datagram holds locks during processing. Needs further
investigation into async lock contention or datagram buffering.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2-relay test: 5.0s audio, RMS 4748, PASS. Full pipeline verified:
- Room correctly identified as global (hash matching works)
- Federation egress channel created and connected
- GlobalRoomActive signals exchanged between peers
- 300 packets (250 source + 50 FEC) forwarded via tagged datagrams
- Client B on relay B received full 5-second tone from client A on relay A
Added debug logging: is_global check, egress channel creation, per-peer
forwarding with active_rooms diagnostic when no match found. Also logs
egress packet count (first + every 250th).
Multi-hop propagation: GlobalRoomActive signals forwarded to other peers
so A→B→C chain knows about rooms across the full mesh.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Major rewrite of relay federation replacing virtual participants with
a clean router model:
1. Global rooms: [[global_rooms]] in TOML config declares rooms that
are bridged across federation. Each relay is a router + local SFU.
2. Room events: RoomManager emits LocalJoin/LocalLeave via broadcast
channel when rooms transition between empty and non-empty.
3. GlobalRoomActive/Inactive signals: relays announce when they have
local participants in global rooms. Peers track active state and
forward media accordingly. Announcements propagate for multi-hop.
4. Media forwarding: separated from SFU loop. Local participant sends
via mpsc channel → egress task → forward_to_peers() → room-hash
tagged datagrams to active peer links. Inbound datagrams delivered
to local participants + forwarded to other active peers (multi-hop).
5. Loop prevention: don't forward back to source relay.
6. Room name hashing: is_global_room() checks both plain name and
hash (clients hash room names for SNI privacy).
Removed: ParticipantSender::Federation, federated_participants, virtual
participant join/leave, periodic room polling. Rooms now only contain
local participants.
Signaling tested: 3-relay chain (A→B←C) correctly propagates
GlobalRoomActive through B to both A and C. Media forwarding plumbing
in place but needs final debugging.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Added [[trusted]] config: relay B can accept inbound federation
from relay A by fingerprint alone, without knowing A's address.
A connects to B with [[peers]], B trusts A with [[trusted]].
- FederationHello signal: outbound connections send their TLS
fingerprint as first signal. The accepting relay verifies it
against [[peers]] (by IP) or [[trusted]] (by fingerprint).
- Tested 3-relay chain: A→B←C. Both A and C connect to B, B trusts
both. B correctly accepts both inbound connections. Room
announcements flow A→B and C→B.
- Remaining: B needs to announce rooms back to A and C on the same
connection so media can flow A→B→C. Currently A has no virtual
participant for B, so media doesn't reach B's SFU for forwarding.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Added debug logging to federation signal path. Fixed the announce/recv
flow: outbound link's announce_task sends FederationRoomJoin, peer's
inbound signal_task receives it and creates virtual participant.
Tested: two relays on localhost with mutual TOML config, client A
sends tone via relay A, client B records via relay B — audio received
through federation (0.1s/RMS 7291/PASS).
Room announcement delay is ~1s (poll interval). The full pipeline:
client join → room created → announce_task detects → sends signal →
peer receives → creates virtual participant → SFU loop forwards
media via room-hash-tagged datagrams → peer demuxes → local delivery.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Inbound federation connections now matched by source IP against
configured peer URLs (QUIC clients don't present TLS certs, so
fingerprint matching fails for inbound direction).
- Added periodic room announcement task (1s poll) that sends
FederationRoomJoin to peers when new rooms appear with local
participants. Handles rooms created after federation link is up.
- Added find_peer_by_addr() to FederationManager.
Federation link topology: each relay pair has 2 connections (outbound
from each side). Outbound sends signals, peer's inbound receives them.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1 of relay federation:
1. Signal messages: FederationRoomJoin/Leave/ParticipantUpdate added
to SignalMessage enum for relay-to-relay room coordination.
2. Room changes: ParticipantOrigin (Local/Federated) tracking, loop
prevention (federated media only forwards to local participants),
ParticipantSender::Federation with 8-byte room-hash prefixed
datagrams, merged participant lists (local + remote), new methods:
join_federated(), update_federated_participants(), local_senders(),
active_rooms(), local_participants().
3. FederationManager: connects to configured peers via QUIC with SNI
"_federation", reconnects with exponential backoff (5s-300s),
exchanges FederationRoomJoin signals, runs recv loops for both
signals and media datagrams, creates virtual participants in rooms.
4. Accept-side: _federation SNI handling in main.rs, unknown peer
gets helpful "add to relay.toml" log message, recognized peers
handed off to FederationManager.
TODO: TLS fingerprint verification — currently outbound connections
use client_config() which doesn't present a cert, so inbound
verification fails. Need mutual TLS or URL-based peer matching.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
