featherChat/warzone/docs/SERVER.md

# Warzone Server -- Administration Guide

**Version 0.0.21**

---

## 1. Building

### Local Build

From the workspace root:

```bash
# Debug
cargo build -p warzone-server

# Release (recommended for deployment)
cargo build -p warzone-server --release
```

Binary output: `target/release/warzone-server`.

### Cross-Compile for Linux (x86_64)

The `scripts/build-linux.sh` script spins up a Hetzner Cloud VPS, builds
Linux release binaries, and pulls them back to `target/linux-x86_64/`.

```bash
# Full pipeline: build + deploy to all production servers + destroy VM
./scripts/build-linux.sh --ship

# Step-by-step:
./scripts/build-linux.sh --prepare     # create VM, install deps, upload source
./scripts/build-linux.sh --build       # compile release binaries on the VM
./scripts/build-linux.sh --transfer    # download binaries to target/linux-x86_64/
./scripts/build-linux.sh --destroy     # delete the VM

# Or all three build steps at once (VM persists):
./scripts/build-linux.sh --all
```

### Minimum Rust Version

Rust 1.75 or later (`rust-version = "1.75"` in `Cargo.toml`).

---

## 2. Running

### Basic

```bash
# Defaults: bind 0.0.0.0:7700, data in ./warzone-data
./warzone-server

# Custom bind address and data directory
./warzone-server --bind 0.0.0.0:7700 --data-dir ./data

# With federation enabled
./warzone-server --federation federation.json
```

### CLI Flags

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--bind` | `-b` | `0.0.0.0:7700` | Address and port to listen on |
| `--data-dir` | `-d` | `./warzone-data` | Directory for the sled database |
| `--federation` | `-f` | *(none)* | Path to federation JSON config file |
| `--enable-bots` | | *(off)* | Enable Bot API and auto-create BotFather on startup |

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `RUST_LOG` | `warn` (production) | Log filter. Examples: `info`, `warzone_server=debug`, `trace` |
| `WZP_RELAY_ADDR` | *(none)* | WZP voice relay address advertised to clients |

### Per-Instance Configuration (`server.env`)

Each server instance can use a `server.env` file for per-instance settings.
Place it in the working directory or alongside the binary. This allows
different instances to have different configurations (e.g., bots enabled on
one server but not another).

Example `server.env`:
```
RUST_LOG=info
WZP_RELAY_ADDR=relay.example.com:3478
ENABLE_BOTS=true
```

### systemd Service

A production-ready unit file is provided at `deploy/warzone-server.service`:

```ini
[Unit]
Description=Warzone Messenger Server (featherChat)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=warzone
Group=warzone
WorkingDirectory=/home/warzone
ExecStart=/home/warzone/warzone-server --bind 0.0.0.0:7700 --data-dir /home/warzone/data --federation /home/warzone/federation.json
Restart=always
RestartSec=3
LimitNOFILE=65536

# Security hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/home/warzone/data
PrivateTmp=yes

Environment=RUST_LOG=warn,warzone_server::federation=info

[Install]
WantedBy=multi-user.target
```

Install and enable:

```bash
sudo cp deploy/warzone-server.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now warzone-server
```

---

## 3. Configuration

### Federation JSON

Enable federation by passing `--federation <path>` on startup. The config
file specifies the local server identity, peer connection details, and a
shared secret for authentication.

**Format** (see `federation.example.json`):

```json
{
  "server_id": "alpha",
  "shared_secret": "change-me-to-a-long-random-string-shared-between-both-servers",
  "peer": {
    "id": "bravo",
    "url": "http://10.0.0.2:7700"
  },
  "presence_interval_secs": 5
}
```

| Field | Description |
|-------|-------------|
| `server_id` | Unique name for this server (e.g. `"alpha"`) |
| `shared_secret` | Pre-shared secret; must match on both sides |
| `peer.id` | The remote server's `server_id` |
| `peer.url` | HTTP base URL of the remote server |
| `presence_interval_secs` | How often to broadcast online-user lists (default 5) |

---

## 4. Federation

Federation connects two Warzone servers over a persistent WebSocket so
their users can communicate transparently.

### How It Works

- On startup, each server opens an outgoing WebSocket to its peer at
  `/v1/federation/ws` and authenticates with the shared secret.
- The connection auto-reconnects on failure.
- Presence (online fingerprints) is synced on the configured interval.
- Messages to users on the remote server are forwarded automatically.

### Federated Features

| Feature | Behavior |
|---------|----------|
| **Key lookup proxy** | If a key bundle is not found locally, the server queries the peer |
| **Message forwarding** | Messages addressed to a remote fingerprint are relayed over the WS |
| **Alias resolution** | `/v1/resolve/:address` checks the peer if the alias is not local |
| **Presence sync** | Each server broadcasts its online fingerprints to the peer |

### Two-Server Setup

**Server A** (`alpha`, e.g. `mequ`):

```json
{
  "server_id": "alpha",
  "shared_secret": "s3cret-shared-between-both",
  "peer": { "id": "bravo", "url": "http://bravo-host:7700" },
  "presence_interval_secs": 5
}
```

**Server B** (`bravo`, e.g. `kh3rad3ree`):

```json
{
  "server_id": "bravo",
  "shared_secret": "s3cret-shared-between-both",
  "peer": { "id": "alpha", "url": "http://alpha-host:7700" },
  "presence_interval_secs": 5
}
```

Both files use the same `shared_secret`. Each server's `peer.id` matches
the other server's `server_id`.

### Federation Status Endpoint

```bash
curl http://localhost:7700/v1/federation/status
```

Returns JSON with connection state, peer info, and presence data.

---

## 4b. Bot System

### Enabling Bots

Start the server with `--enable-bots` to activate bot functionality. Without
this flag, all bot endpoints return 403.

```bash
./warzone-server --bind 0.0.0.0:7700 --enable-bots
```

### BotFather Auto-Creation

On first start with `--enable-bots`, the server auto-creates the `@botfather`
bot. The BotFather token is printed to the server logs. Users interact with
`@botfather` to register new bots.

### Per-Instance Bot Toggle

Bot support can be enabled independently per server instance:

| Instance | Bots | Config |
|----------|------|--------|
| mequ | Disabled | No `--enable-bots` flag |
| kh3rad3ree | Enabled | `--enable-bots` flag set |

### Bot Webhook Delivery

When a bot has a webhook configured (via `setWebhook`), incoming messages are
delivered live to the webhook URL via HTTP POST instead of being queued for
`getUpdates` polling. This is integrated into the standard message routing
pipeline -- `deliver_or_queue` checks for webhook configuration before
queueing.

---

## 5. API Reference

All endpoints are prefixed with `/v1`. The web UI is served at `/`.

### Notation

- **Auth** = requires `Authorization: Bearer <token>` header (write routes).
- **Public** = no authentication needed (read routes).

---

### Health

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| GET | `/v1/health` | Public | Health check; returns `{"status":"ok"}` |

---

### Keys

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| POST | `/v1/keys/register` | Public | Register a pre-key bundle |
| POST | `/v1/keys/replenish` | Public | Upload additional one-time pre-keys |
| GET | `/v1/keys/:fingerprint` | Public | Fetch a key bundle (falls back to federation peer) |
| GET | `/v1/keys/list` | Public | List all registered fingerprints |
| GET | `/v1/keys/:fingerprint/otpk-count` | Public | Remaining one-time pre-key count |
| GET | `/v1/keys/:fingerprint/devices` | Public | List devices for a fingerprint |

---

### Messages

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| POST | `/v1/messages/send` | Auth | Send an encrypted message blob |
| GET | `/v1/messages/poll/:fingerprint` | Public | Poll queued messages |
| DELETE | `/v1/messages/:id/ack` | Public | Acknowledge (delete) a message |

---

### Groups

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| POST | `/v1/groups/create` | Auth | Create a group |
| POST | `/v1/groups/:name/join` | Auth | Join a group |
| POST | `/v1/groups/:name/send` | Auth | Send a message to a group |
| POST | `/v1/groups/:name/leave` | Auth | Leave a group |
| POST | `/v1/groups/:name/kick` | Auth | Kick a member from a group |
| GET | `/v1/groups` | Public | List all groups |
| GET | `/v1/groups/:name` | Public | Get group details |
| GET | `/v1/groups/:name/members` | Public | List members (includes online status) |

---

### Aliases

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| POST | `/v1/alias/register` | Auth | Register a human-readable alias |
| POST | `/v1/alias/unregister` | Auth | Remove your alias |
| POST | `/v1/alias/recover` | Auth | Transfer alias to a new fingerprint |
| POST | `/v1/alias/renew` | Auth | Renew alias expiry |
| POST | `/v1/alias/admin-remove` | Auth | Admin-remove an alias |
| GET | `/v1/alias/resolve/:name` | Public | Resolve alias to fingerprint |
| GET | `/v1/alias/list` | Public | List all registered aliases |
| GET | `/v1/alias/whois/:fingerprint` | Public | Reverse-lookup: fingerprint to alias |

---

### Calls (WZP)

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| POST | `/v1/calls/initiate` | Auth | Start a 1:1 call |
| POST | `/v1/calls/:id/end` | Auth | End an active call |
| POST | `/v1/calls/missed` | Auth | Get missed calls for a fingerprint |
| POST | `/v1/groups/:name/call` | Auth | Initiate a group call |
| GET | `/v1/calls/:id` | Public | Get call details |
| GET | `/v1/calls/active` | Public | List active calls |

---

### Devices

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| POST | `/v1/devices/:id/kick` | Auth | Disconnect a specific device |
| POST | `/v1/devices/revoke-all` | Auth | Disconnect all devices (optional keep one) |
| GET | `/v1/devices` | Auth | List your connected devices |

---

### Presence

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| POST | `/v1/presence/batch` | Auth | Batch-query presence for multiple fingerprints |
| GET | `/v1/presence/:fingerprint` | Public | Check if a fingerprint is online |

---

### Friends

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| POST | `/v1/friends` | Auth | Save friend list (encrypted blob) |
| GET | `/v1/friends` | Auth | Retrieve saved friend list |

---

### Resolve

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| GET | `/v1/resolve/:address` | Public | Universal resolve: ETH address, alias, or fingerprint. Checks federation peer if not found locally. |

---

### WZP Voice Relay

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| GET | `/v1/wzp/relay-config` | Public | Get the WZP relay address for voice calls |

---

### Federation

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| GET | `/v1/federation/status` | Public | Federation connection status and peer info |
| GET | `/v1/federation/ws` | Internal | WebSocket endpoint for server-to-server communication |

---

### Bot API

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| POST | `/v1/bot/register` | Auth | Register a bot; returns an API token |
| GET | `/v1/bot/:token/getMe` | Token | Bot identity info |
| POST | `/v1/bot/:token/getUpdates` | Token | Long-poll for new messages (Telegram-compatible) |
| POST | `/v1/bot/:token/sendMessage` | Token | Send a message as the bot (Telegram-compatible) |

Bot tokens are scoped to the bot's fingerprint. The `getUpdates` and
`sendMessage` endpoints follow the Telegram Bot API conventions so existing
Telegram bot libraries can be adapted with minimal changes.

---

### WebSocket

| Path | Description |
|------|-------------|
| `/v1/ws/:fingerprint` | Real-time message delivery. Clients receive instant push of new messages. |

---

### Web UI

| Path | Description |
|------|-------------|
| `/` | Single-page WASM web client |
| `/wasm/warzone_wasm.js` | WASM JavaScript bindings |
| `/wasm/warzone_wasm_bg.wasm` | WASM binary |

---

## Voice Calls (WZP Integration)

featherChat supports voice calls via the WarzonePhone (WZP) audio relay. Three components work together:

### Components

| Component | Binary | Port | Purpose |
|-----------|--------|------|---------|
| featherChat server | `warzone-server` | 7700 | Signaling (offer/answer/hangup) + auth tokens |
| WZP relay | `wzp-relay` | 4433 | QUIC audio relay (SFU) |
| WZP web bridge | `wzp-web` | 8080 | Browser WebSocket ↔ QUIC bridge |

### Running

```bash
# 1. WZP relay (QUIC audio)
./wzp-relay --listen 0.0.0.0:4433 --auth-url http://127.0.0.1:7700/v1/auth/validate

# 2. WZP web bridge (browser ↔ relay)
./wzp-web --port 8080 --relay 127.0.0.1:4433 --auth-url http://127.0.0.1:7700/v1/auth/validate

# 3. featherChat server (with relay address)
WZP_RELAY_ADDR=127.0.0.1:8080 ./warzone-server
```

### TLS Requirements

| Scenario | TLS needed? | Why |
|----------|-------------|-----|
| localhost dev | No | Browser allows mic on localhost without HTTPS |
| LAN/remote | wzp-web needs TLS | Browsers require HTTPS for `getUserMedia()` on non-localhost |
| Production | All three should use TLS | Security best practice |

For production TLS on wzp-web:
```bash
./wzp-web --port 8080 --relay 127.0.0.1:4433 --auth-url http://127.0.0.1:7700/v1/auth/validate --cert /path/to/cert.pem --key /path/to/key.pem
```

### Auth Flow

1. User clicks Call -> signaling via featherChat WebSocket
2. Call accepted -> both clients fetch `GET /v1/wzp/relay-config`
3. Server returns `{ relay_addr, token, expires_in: 300 }`
4. Clients connect WebSocket to `ws://relay_addr/ws/ROOM`
5. First message: `{"type":"auth","token":"<token>"}`
6. wzp-web validates token against featherChat `/v1/auth/validate`
7. Audio flows: mic -> PCM -> WS -> wzp-web -> QUIC -> wzp-relay -> peer

---

## 6. Database

The server uses **sled** (embedded key-value store). All data lives under
the `--data-dir` directory.

### Trees

| Tree | Purpose |
|------|---------|
| `keys` | Pre-key bundles (public keys only) |
| `messages` | Queued encrypted message blobs |
| `groups` | Group metadata and membership |
| `aliases` | Human-readable alias mappings |
| `tokens` | Authentication tokens (device sessions) |
| `calls` | Call records (1:1 and group) |
| `missed_calls` | Missed call notifications |
| `friends` | Encrypted friend lists |
| `eth_addresses` | Ethereum address to fingerprint mappings |

### Data Directory Structure

```
warzone-data/
  db                    # sled database file
  conf                  # sled config
  blobs/                # sled blob storage
  snap.*/               # sled snapshots
```

The entire directory should be treated as a unit for backup. Stop the server
before copying, or use filesystem-level snapshots (LVM, ZFS, btrfs).

---

## 7. Security

### Auth Middleware

All write (POST) endpoints require a bearer token in the `Authorization`
header. Tokens are issued during key registration and tied to a fingerprint.
Read (GET) endpoints are public.

### Rate Limiting

- **200 concurrent requests** (tower `ConcurrencyLimitLayer`)
- **5 WebSocket connections per fingerprint** (multi-device cap)

### Device Management

Users can list connected devices, kick individual devices, or revoke all
sessions via the `/v1/devices` endpoints. The `revoke-all` endpoint accepts
an optional `keep_device_id` to keep the current device active.

### What the Server Can See

| Data | Visible |
|------|---------|
| Message plaintext | No (E2E encrypted blobs) |
| Sender/recipient fingerprints | Yes |
| Message size and timing | Yes |
| Public pre-key bundles | Yes (public by design) |
| IP addresses | Yes (from HTTP) |

---

## 8. Monitoring

### Logging

Control verbosity with `RUST_LOG`:

```bash
RUST_LOG=warn ./warzone-server              # production default
RUST_LOG=info ./warzone-server              # request-level logging
RUST_LOG=warzone_server=debug ./warzone-server  # server internals
RUST_LOG=trace ./warzone-server             # everything
```

With systemd:

```bash
journalctl -u warzone-server -f
```

### Health Check

```bash
curl http://localhost:7700/v1/health
```

### Federation Status

```bash
curl http://localhost:7700/v1/federation/status
```

Returns connection state, peer identity, and synced presence data.

---

## 9. Deploy Scripts

The `scripts/build-linux.sh` script handles the full build and deploy
lifecycle via Hetzner Cloud VMs.

### Key Commands

| Command | Description |
|---------|-------------|
| `--ship` | Full pipeline: build on VM, deploy to all production servers, destroy VM |
| `--update-all` | Upload pre-built binaries to all production servers and restart |
| `--update <user@host>` | Update a single production server |
| `--status` | Check service status and federation on all production servers |
| `--logs [user@host]` | Tail `journalctl` logs (defaults to first production server) |

### Typical Deploy Workflow

```bash
# One command: build, deploy everywhere, clean up
./scripts/build-linux.sh --ship

# Or step by step:
./scripts/build-linux.sh --all          # build (VM persists)
./scripts/build-linux.sh --update-all   # deploy binaries
./scripts/build-linux.sh --destroy      # clean up VM
./scripts/build-linux.sh --status       # verify
```

---

## 10. Backup and Recovery

### Backup

```bash
systemctl stop warzone-server
cp -r /home/warzone/data /backup/warzone-$(date +%Y%m%d)
systemctl start warzone-server
```

Do not copy the sled directory while the server is running without
filesystem-level snapshots.

### Recovery

1. Stop the server.
2. Replace the data directory with the backup.
3. Start the server.

Messages queued after the backup was taken are permanently lost. All
messages are E2E encrypted and cannot be recovered from any other source.