wz-phone/docs/ADMINISTRATION.md

WarzonePhone Relay Administration Guide

This document covers deploying, configuring, and operating wzp-relay instances, including federation setup, monitoring, and troubleshooting.

Relay Deployment

Binary

Build and run the relay directly:

# Build release binary
cargo build --release --bin wzp-relay

# Run with defaults (listen on 0.0.0.0:4433, room mode, no auth)
./target/release/wzp-relay

# Run with config file
./target/release/wzp-relay --config /etc/wzp/relay.toml

Remote Build (Linux)

The included build script provisions a temporary Hetzner Cloud VPS, builds all binaries, and downloads them:

# Requires: hcloud CLI authenticated, SSH key "wz" registered
./scripts/build-linux.sh
# Outputs to: target/linux-x86_64/

Produces: wzp-relay, wzp-client, wzp-client-audio, wzp-web, wzp-bench.

Docker

FROM rust:1.85 AS builder
WORKDIR /src
COPY . .
RUN cargo build --release --bin wzp-relay

FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/*
COPY --from=builder /src/target/release/wzp-relay /usr/local/bin/
EXPOSE 4433/udp
EXPOSE 9090/tcp
VOLUME /data
ENV HOME=/data
ENTRYPOINT ["wzp-relay"]
CMD ["--config", "/data/relay.toml", "--metrics-port", "9090"]

Build and run:

docker build -t wzp-relay .
docker run -d \
  --name wzp-relay \
  -p 4433:4433/udp \
  -p 9090:9090/tcp \
  -v /opt/wzp:/data \
  wzp-relay

systemd

Create /etc/systemd/system/wzp-relay.service:

[Unit]
Description=WarzonePhone Relay
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=wzp
Group=wzp
ExecStart=/usr/local/bin/wzp-relay --config /etc/wzp/relay.toml
Restart=always
RestartSec=5
LimitNOFILE=65536

# Security hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/wzp
PrivateTmp=yes

Environment=HOME=/var/lib/wzp
Environment=RUST_LOG=info

[Install]
WantedBy=multi-user.target

Setup:

# Create service user
useradd --system --home-dir /var/lib/wzp --create-home wzp

# Install binary and config
cp target/release/wzp-relay /usr/local/bin/
mkdir -p /etc/wzp
cp relay.toml /etc/wzp/

# Enable and start
systemctl daemon-reload
systemctl enable --now wzp-relay
journalctl -u wzp-relay -f

TOML Configuration Reference

All fields have defaults. A minimal config file only needs the fields you want to override.

Core Settings

Field	Type	Default	Description
listen_addr	string (socket addr)	"0.0.0.0:4433"	UDP address to listen on for incoming QUIC connections
remote_relay	string (socket addr)	none	Remote relay address for forward mode. Disables room mode when set
max_sessions	integer	100	Maximum concurrent client sessions
log_level	string	"info"	Logging level: trace, debug, info, warn, error

Jitter Buffer

Field	Type	Default	Description
jitter_target_depth	integer	50	Target buffer depth in packets (50 = 1 second at 20ms frames)
jitter_max_depth	integer	250	Maximum buffer depth in packets (250 = 5 seconds)

Authentication

Field	Type	Default	Description
auth_url	string	none	featherChat auth validation URL. When set, clients must send a bearer token as their first signal message. The relay validates it via POST <auth_url>

Metrics and Monitoring

Field	Type	Default	Description
metrics_port	integer	none	Port for the Prometheus HTTP metrics endpoint. Disabled if not set
probe_targets	array of socket addrs	[]	Peer relay addresses to probe for health monitoring (1 Ping/s each)
probe_mesh	boolean	false	Enable mesh mode for probe targets

Media Processing

Field	Type	Default	Description
trunking_enabled	boolean	false	Enable trunk batching for outgoing media. Packs multiple session packets into one QUIC datagram, reducing overhead

WebSocket / Browser Support

Field	Type	Default	Description
ws_port	integer	none	Port for WebSocket listener (browser clients). Disabled if not set
static_dir	string	none	Directory to serve static files (HTML/JS/WASM)

Federation

Field	Type	Default	Description
peers	array of PeerConfig	[]	Outbound federation peer relays
trusted	array of TrustedConfig	[]	Inbound federation trust list
global_rooms	array of GlobalRoomConfig	[]	Room names to bridge across federation

Debugging

Field	Type	Default	Description
debug_tap	string	none	Log packet headers for matching rooms. Use "*" for all rooms, or a specific room name

PeerConfig Fields

Field	Type	Required	Description
url	string	yes	Address of the peer relay (e.g., "193.180.213.68:4433")
fingerprint	string	yes	Expected TLS certificate fingerprint (hex with colons)
label	string	no	Human-readable label for logging

TrustedConfig Fields

Field	Type	Required	Description
fingerprint	string	yes	Expected TLS certificate fingerprint (hex with colons)
label	string	no	Human-readable label for logging

GlobalRoomConfig Fields

Field	Type	Required	Description
name	string	yes	Room name to bridge across federation (e.g., "android")

CLI Flags Reference

wzp-relay [--config <path>] [--listen <addr>] [--remote <addr>]
          [--auth-url <url>] [--metrics-port <port>]
          [--probe <addr>]... [--probe-mesh] [--mesh-status]
          [--trunking] [--global-room <name>]...
          [--debug-tap <room>]
          [--ws-port <port>] [--static-dir <dir>]

Flag	Description
--config <path>	Load configuration from TOML file. CLI flags override config file values
--listen <addr>	Listen address (default: 0.0.0.0:4433)
--remote <addr>	Remote relay for forwarding mode. Disables room mode
--auth-url <url>	featherChat auth endpoint (e.g., https://chat.example.com/v1/auth/validate)
--metrics-port <port>	Prometheus metrics HTTP port (e.g., 9090)
--probe <addr>	Peer relay to probe for health monitoring. Repeatable
--probe-mesh	Enable mesh mode for probes
--mesh-status	Print mesh health table and exit (diagnostic)
--trunking	Enable trunk batching for outgoing media
--global-room <name>	Declare a room as global (bridged across federation). Repeatable
--debug-tap <room>	Log packet headers for a room ("*" for all rooms)
--event-log <path>	Write JSONL protocol event log for federation debugging
--version, -V	Print build git hash and exit
--ws-port <port>	WebSocket listener port for browser clients
--static-dir <dir>	Directory to serve static files from
--help, -h	Print help and exit

CLI flags always override config file values when both are specified.

Federation Setup

Concepts

• [[peers]] -- outbound: relays we connect TO. Requires address + fingerprint
• [[trusted]] -- inbound: relays we accept connections FROM. Requires fingerprint only (they connect to us)
• [[global_rooms]] -- rooms bridged across all federated peers. Participants on different relays in the same global room hear each other

Getting Your Relay's Fingerprint

When a relay starts, it logs its TLS fingerprint:

INFO TLS certificate (deterministic from relay identity) tls_fingerprint="a5d6:e3c6:5ae7:185c:4eb1:af89:daed:4a43"
INFO federation: to peer with this relay, add to relay.toml:
INFO   [[peers]]
INFO   url = "193.180.213.68:4433"
INFO   fingerprint = "a5d6:e3c6:5ae7:185c:4eb1:af89:daed:4a43"

Share this information with the administrator of the peer relay.

Unknown Peer Connections

When an unknown relay tries to federate, the log shows:

WARN unknown relay wants to federate addr=10.0.0.5:12345 fp="7f2a:b391:0c44:..."
INFO   to accept, add to relay.toml:
INFO   [[trusted]]
INFO   fingerprint = "7f2a:b391:0c44:..."
INFO   label = "Relay at 10.0.0.5:12345"

Example Configurations

Single Relay (Minimal)

# /etc/wzp/relay.toml
# Minimal config -- all defaults, just enable metrics
metrics_port = 9090

Run:

wzp-relay --config /etc/wzp/relay.toml

Single Relay (Full Featured)

# /etc/wzp/relay.toml
listen_addr = "0.0.0.0:4433"
max_sessions = 200
log_level = "info"

# Metrics
metrics_port = 9090

# Authentication
auth_url = "https://chat.example.com/v1/auth/validate"

# Browser support
ws_port = 8080
static_dir = "/opt/wzp/web"

# Performance
trunking_enabled = true

# Jitter buffer tuning
jitter_target_depth = 50
jitter_max_depth = 250

Two-Relay Federation

Relay A (relay-a.toml on 193.180.213.68):

listen_addr = "0.0.0.0:4433"
metrics_port = 9090

# Outbound: connect to Relay B
[[peers]]
url = "10.0.0.5:4433"
fingerprint = "7f2a:b391:0c44:9e1d:a8b2:c5d7:e3f0:1234"
label = "Relay B (US)"

# Accept inbound from Relay B
[[trusted]]
fingerprint = "7f2a:b391:0c44:9e1d:a8b2:c5d7:e3f0:1234"
label = "Relay B (US)"

# Bridge these rooms
[[global_rooms]]
name = "android"

[[global_rooms]]
name = "general"

Relay B (relay-b.toml on 10.0.0.5):

listen_addr = "0.0.0.0:4433"
metrics_port = 9090

# Outbound: connect to Relay A
[[peers]]
url = "193.180.213.68:4433"
fingerprint = "a5d6:e3c6:5ae7:185c:4eb1:af89:daed:4a43"
label = "Relay A (EU)"

# Accept inbound from Relay A
[[trusted]]
fingerprint = "a5d6:e3c6:5ae7:185c:4eb1:af89:daed:4a43"
label = "Relay A (EU)"

# Same global rooms
[[global_rooms]]
name = "android"

[[global_rooms]]
name = "general"

Three-Relay Chain (Full Mesh)

For three relays (A, B, C) in full mesh federation, each relay needs peers and trusted entries for the other two:

Relay A (EU):

listen_addr = "0.0.0.0:4433"
metrics_port = 9090

# Probe all peers
probe_targets = ["10.0.0.5:4433", "10.0.0.9:4433"]
probe_mesh = true

# Peers
[[peers]]
url = "10.0.0.5:4433"
fingerprint = "7f2a:b391:0c44:9e1d:a8b2:c5d7:e3f0:1234"
label = "Relay B (US)"

[[peers]]
url = "10.0.0.9:4433"
fingerprint = "3c8e:d2a1:f7b5:6049:81c3:e9d4:a2f6:5678"
label = "Relay C (APAC)"

# Trust
[[trusted]]
fingerprint = "7f2a:b391:0c44:9e1d:a8b2:c5d7:e3f0:1234"
label = "Relay B (US)"

[[trusted]]
fingerprint = "3c8e:d2a1:f7b5:6049:81c3:e9d4:a2f6:5678"
label = "Relay C (APAC)"

# Global rooms
[[global_rooms]]
name = "android"

[[global_rooms]]
name = "general"

Relay B and Relay C follow the same pattern, listing the other two relays in their [[peers]] and [[trusted]] sections.

Monitoring

Prometheus Metrics

Enable with --metrics-port <port> or metrics_port in TOML. The relay exposes metrics at GET /metrics on the specified HTTP port.

Relay Metrics

Metric	Type	Labels	Description
wzp_relay_active_sessions	Gauge	--	Current active sessions
wzp_relay_active_rooms	Gauge	--	Current active rooms
wzp_relay_packets_forwarded_total	Counter	room	Total packets forwarded
wzp_relay_bytes_forwarded_total	Counter	room	Total bytes forwarded
wzp_relay_auth_attempts_total	Counter	result (ok/fail)	Auth validation attempts
wzp_relay_handshake_duration_seconds	Histogram	--	Crypto handshake time

Per-Session Metrics

Metric	Type	Labels	Description
wzp_relay_session_jitter_buffer_depth	Gauge	session_id	Buffer depth per session
wzp_relay_session_loss_pct	Gauge	session_id	Packet loss percentage
wzp_relay_session_rtt_ms	Gauge	session_id	Round-trip time
wzp_relay_session_underruns_total	Counter	session_id	Jitter buffer underruns
wzp_relay_session_overruns_total	Counter	session_id	Jitter buffer overruns

Inter-Relay Probe Metrics

Metric	Type	Labels	Description
wzp_probe_rtt_ms	Gauge	target	RTT to peer relay
wzp_probe_loss_pct	Gauge	target	Loss to peer relay
wzp_probe_jitter_ms	Gauge	target	Jitter to peer relay
wzp_probe_up	Gauge	target	1 if reachable, 0 if not

Prometheus Scrape Config

# prometheus.yml
scrape_configs:
  - job_name: 'wzp-relay'
    static_configs:
      - targets:
        - 'relay-a:9090'
        - 'relay-b:9090'
    scrape_interval: 10s

Grafana Dashboard

A pre-built dashboard is available at docs/grafana-dashboard.json. Import it into Grafana for:

1. Relay Health -- active sessions, rooms, packets/s, bytes/s
2. Call Quality -- per-session jitter depth, loss%, RTT, underruns over time
3. Inter-Relay Mesh -- latency heatmap, probe status, loss trends
4. Web Bridge -- active connections, frames bridged, auth failures

Event Log (Protocol Analyzer)

Use --event-log to write a JSONL event log that traces every federation media packet through the relay pipeline. Essential for debugging federation audio issues.

wzp-relay --config relay.toml --event-log /tmp/events.jsonl

Each media packet emits events at every decision point:

• federation_ingress — packet arrived from a peer relay
• local_deliver — packet delivered to local participants
• dedup_drop — packet dropped as duplicate
• rate_limit_drop — packet dropped by rate limiter
• room_not_found — packet for unknown room
• local_deliver_error — delivery to local client failed

Analyze with:

# Count events by type
cat events.jsonl | python3 -c "
import json, collections, sys
c = collections.Counter()
for l in sys.stdin: c[json.loads(l)['event']] += 1
for k,v in sorted(c.items(), key=lambda x:-x[1]): print(f'  {k}: {v}')
"

Remote Version Check

Verify a deployed relay's version without SSH:

wzp-client --version-check <relay-addr:port>

Debug Tap

Use --debug-tap to log packet headers for debugging:

# Log headers for room "android"
wzp-relay --debug-tap android

# Log headers for all rooms
wzp-relay --debug-tap '*'

Or in TOML:

debug_tap = "android"

Mesh Status

Print the current mesh health table (diagnostic):

wzp-relay --mesh-status

Authentication

featherChat Token Validation

When --auth-url is set, the relay requires clients to send an AuthToken signal message as their first message after QUIC connection. The relay validates the token by calling:

POST <auth_url>
Content-Type: application/json
Authorization: Bearer <token>

Expected response:

{
  "valid": true,
  "fingerprint": "a5d6:e3c6:...",
  "alias": "username"
}

If validation fails, the client is disconnected.

Without Authentication

When --auth-url is not set, any client can connect. The relay logs:

INFO auth disabled -- any client can connect (use --auth-url to enable)

Identity Persistence

Relay Identity File

The relay stores its identity seed at ~/.wzp/relay-identity (a 64-character hex string). This seed:

• Is generated automatically on first run
• Persists across restarts
• Derives the relay's Ed25519 signing key and X25519 key agreement key
• Derives the TLS certificate deterministically (same seed = same cert = same fingerprint)

If the identity file is corrupted, the relay generates a new one and logs a warning. This will change the relay's TLS fingerprint, requiring federation peers to update their config.

Backup

Back up the identity file to preserve the relay's fingerprint:

cp ~/.wzp/relay-identity /secure/backup/relay-identity

To restore, copy the file back before starting the relay.

Troubleshooting

Common Issues

Problem	Cause	Solution
"unknown argument" on startup	Unrecognized CLI flag	Check wzp-relay --help for valid flags
"failed to load config"	Invalid TOML syntax	Validate TOML file with toml-cli or similar
"auth failed" for all clients	Wrong auth_url or featherChat server down	Verify URL is reachable: curl -X POST <auth_url>
"session rejected"	Max sessions reached	Increase max_sessions in config
Clients cannot connect	Firewall blocking UDP 4433	Open UDP port 4433 in firewall
Federation "unknown relay wants to federate"	Peer's fingerprint not in [[trusted]]	Add the logged fingerprint to [[trusted]]
Federation "fingerprint mismatch"	Peer relay restarted with new identity	Update the fingerprint in [[peers]] config
Federation audio silent on consecutive connects	Dedup filter or jitter buffer state	Verify relay is running latest build with time-based dedup
Federation participant shows wrong relay label	Hub relay not propagating original labels	Update relay to latest build (label preservation fix)
Federation disconnect takes >15 seconds	QUIC idle timeout + stale sweeper	Normal: sweeper runs every 5s with 15s TTL. Use latest client with SIGTERM handler for instant disconnect
High packet loss between relays	Network congestion or misconfiguration	Check wzp_probe_loss_pct metric; consider relay chaining
Jitter buffer overruns	Packets arriving faster than playout	Increase jitter_max_depth
Jitter buffer underruns	Packets arriving too slowly or lost	Check network quality; increase jitter_target_depth
"probe connection closed"	Peer relay unreachable or crashed	Check peer relay status; will auto-reconnect
WebSocket clients cannot connect	ws_port not set	Add --ws-port <port> or ws_port in TOML
Browser mic access denied	Not using HTTPS	Use TLS termination in front of the relay or serve via wzp-web --tls

Log Level Tuning

Set RUST_LOG environment variable for fine-grained control:

# All relay logs at debug level
RUST_LOG=debug wzp-relay

# Only federation at trace, everything else at info
RUST_LOG=info,wzp_relay::federation=trace wzp-relay

# Quiet mode -- only warnings and errors
RUST_LOG=warn wzp-relay

Health Checks

# Check if relay is listening
nc -zu relay-host 4433

# Check metrics endpoint
curl -s http://relay-host:9090/metrics | head -20

# Check active sessions
curl -s http://relay-host:9090/metrics | grep wzp_relay_active_sessions

# Check federation probe health
curl -s http://relay-host:9090/metrics | grep wzp_probe_up

Build Pipelines

All production artifacts (Android APK, Linux x86_64 binaries, Windows .exe) are built on SepehrHomeserverdk using Docker, not on developer workstations. The pipelines are fire-and-forget: a local script invokes a tmux session on the remote, the build runs in a Docker container, and the artifact is uploaded to paste.dk.manko.yoga (rustypaste) with a notification sent to ntfy.sh/wzp on start and completion.

Docker images

Two long-lived images live on the remote:

Image	Used by	Base	Key contents
wzp-android-builder	Android APK (Tauri mobile + legacy Kotlin), Linux x86_64 relay/CLI	Debian bookworm	Rust stable with Android targets, cargo-ndk, NDK 26.1, Android SDK (API 34 + 35 + 36), JDK 17, Gradle 8.5, Node.js 20, cmake, ninja, tauri-cli 2.x
wzp-windows-builder	Windows x86_64 .exe	Debian bookworm	Rust stable with x86_64-pc-windows-msvc target, cargo-xwin (with pre-warmed MSVC CRT + Windows SDK cache), Node.js 20, cmake, ninja, clang, lld, nasm

Both images are rebuilt rarely — once the base toolchain is stable, rebuilds are only needed to pick up new dependencies or security patches.

Rebuilding an image (fire-and-forget, ~10 min on a warm base):

# Windows
./scripts/build-windows-docker.sh --image-build

# Android (upload and rebuild handled by the Android build script itself — see
# its --image-build flag or equivalent)

The --image-build flag uploads the local Dockerfile to the remote, kicks off docker build under nohup, and returns immediately. Monitor with:

ssh SepehrHomeserverdk 'tail -f /tmp/wzp-windows-image-build.log'

Pipeline: Android APK (Tauri Mobile)

./scripts/build-tauri-android.sh                  # Full: pull + build + upload + notify
./scripts/build-tauri-android.sh --no-pull        # Skip git fetch
./scripts/build-tauri-android.sh --clean          # Force-clean Rust target

• Branch: android-rewrite
• Image: wzp-android-builder
• Build command: cargo tauri android build --release
• Output: wzp-release.apk → uploaded to rustypaste
• Notifications: start + completion to ntfy.sh/wzp
• Remote artifact path: /mnt/storage/manBuilder/data/cache-android/target/…/release/app-release.apk

Pipeline: Linux x86_64 (relay + CLI + bench + web)

./scripts/build-linux-docker.sh                   # Fire-and-forget
./scripts/build-linux-docker.sh --no-pull         # Skip git fetch
./scripts/build-linux-docker.sh --clean           # Force-clean target
./scripts/build-linux-docker.sh --install         # Wait for completion and download locally

• Branch: feat/android-voip-client (script default — override by editing the script or passing an env var)
• Image: wzp-android-builder (shared, not a separate Linux-only image)
• Targets built: wzp-relay, wzp-client, wzp-client-audio (with --features audio), wzp-web, wzp-bench
• Output: wzp-linux-x86_64.tar.gz with all five binaries → uploaded to rustypaste
• Local landing dir (with --install): target/linux-x86_64/

Pipeline: Windows x86_64 (wzp-desktop.exe)

./scripts/build-windows-docker.sh                 # Full: pull + build + download locally
./scripts/build-windows-docker.sh --no-pull       # Skip git fetch
./scripts/build-windows-docker.sh --rust          # Force-clean target-windows cache
./scripts/build-windows-docker.sh --image-build   # Rebuild the Docker image (fire-and-forget)

• Branch: feat/desktop-audio-rewrite
• Image: wzp-windows-builder
• Build command: cargo xwin build --release --target x86_64-pc-windows-msvc --bin wzp-desktop
• Output: wzp-desktop.exe (~16 MB) → downloaded to target/windows-exe/wzp-desktop.exe, also uploaded to rustypaste
• Target cache volume: target-windows (separate from the Android target cache to avoid triple cross-contamination)
• Shared cache volumes: cargo-registry, cargo-git (shared with Android — both pipelines pull the same crates)

A/B-preserving workflow for testing audio backends: rename the prior .exe before re-running the build, so both coexist:

# Preserve prior build as the noAEC baseline
mv target/windows-exe/wzp-desktop.exe target/windows-exe/wzp-desktop-noAEC.exe
./scripts/build-windows-docker.sh
ls -la target/windows-exe/
# wzp-desktop-noAEC.exe  (previous build)
# wzp-desktop.exe        (new build)

Alternative pipeline: Windows via Hetzner Cloud VPS

For situations where Docker image rebuilds would be disruptive, or for one-shot debug builds on a clean machine:

./scripts/build-windows-cloud.sh                  # Full: create VM → build → download → destroy
./scripts/build-windows-cloud.sh --prepare        # Create VM + install deps, don't build
./scripts/build-windows-cloud.sh --build          # Build on existing VM
./scripts/build-windows-cloud.sh --transfer       # Download .exe from existing VM
./scripts/build-windows-cloud.sh --destroy        # Delete the VM
WZP_KEEP_VM=1 ./scripts/build-windows-cloud.sh    # Don't auto-destroy after successful build

• Provider: Hetzner Cloud
• Default server type: cx33 (8 GB RAM, 8 vCPU — cx23 with 4 GB OOMs on the tauri+rustls cross-compile)
• Image: ubuntu-24.04
• SSH key: must be named wz in Hetzner and loaded in the local ssh-agent
• Reminder: set WZP_KEEP_VM=1 for multi-build sessions, then remember to --destroy at end of day so the VM isn't left running overnight. This is tracked in the auto-memory as feedback_keep_windows_builder_vm.md.

Notifications

All pipelines post to https://ntfy.sh/wzp. Subscribe from your phone via the ntfy.sh app to get push notifications on build start/success/failure. Messages include the short git hash and the rustypaste URL on success:

WZP Windows build OK [03a80a3] (16M)
https://paste.dk.manko.yoga/<uuid>/wzp-desktop.exe

Rustypaste credentials

Build pipelines read rusty_address and rusty_auth_token from the .env file at /mnt/storage/manBuilder/.env on SepehrHomeserverdk. Local scripts that upload directly (build-windows-cloud.sh when run in --transfer mode) read from ~/.wzp/rustypaste.env with the same variable names. Both files must be kept in sync manually if rotated.