nick/nick-doc
3
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs: sync vault with codebase state (2026-06-12)

Browse Source 
- Update backend, frontend, scanner, deployment, amanat-assist service docs
- Update System Overview, Scanner Architecture, Telegram Mini App flow
- Update 10 - Services/README.md
- Add Tenant data model, Tenant API reference, Tenant Storefront Flow
- Add Multi-Shop Branch Project Scan (2026-06-10)
- Add tenant.md service doc
- Append activity log entry
- Reflects archived/search/stats route fix and new E2E test suite

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Siavash Sameni
2026-06-12 11:42:18 +04:00
parent 18073afb52
commit e52ffce48a
18 changed files with 2619 additions and 1102 deletions
Download Patch File Download Diff File Expand all files Collapse all files

515
10 - Services/deployment.md
View File

@@ -5,7 +5,7 @@ tags: [services, deployment, infrastructure, docker]
# Deployment
The `deployment/` sub-project contains all Docker Compose definitions, Caddyfile configurations, Gatus monitoring config, and environment templates for running the Amanat escrow platform. Two compose files exist side-by-side reflecting a legacy setup and the current live stack.
The `deployment/` sub-project contains Docker Compose definitions, reverse-proxy configs, Gatus monitoring, and migration bundles for running the Amanat escrow platform. It covers three distinct stacks: a **legacy compose** (reference only), the **dev-amn active dev stack** (`dev.amn.gg`), and the **escrow-multi white-label stack** (`multi.amn.gg`).
---
@@ -13,124 +13,198 @@ The `deployment/` sub-project contains all Docker Compose definitions, Caddyfile
| File | Status | Host | Notes |
|---|---|---|---|
| `deployment/docker-compose.yml` | Legacy | Any | nginx + traefik_public network, images from `git.manko.yoga` registry |
| `deployment/dev-amn/docker-compose.yml` | **Active** | `89.58.32.32` | shared-web + infra-caddy ingress, images from `git.tbs.amn.gg/escrow` |
| `deployment/docker-compose.yml` | **Legacy / reference** | Any | nginx + traefik_public network; images from `git.manko.yoga` registry. Do not deploy from this. |
| `deployment/dev-amn/docker-compose.yml` | **Active** | `89.58.32.32` | `shared-web` + infra-caddy ingress; images from `git.tbs.amn.gg/escrow` |
| `deployment/escrow-multi/docker-compose.yml` | **Active multi-shop** | `89.58.32.32` | Isolated stack for `multi.amn.gg`; images tagged `:multi`; fresh Postgres/Redis; Drizzle migrations |
The `dev-amn` stack is the authoritative deployment. It runs under Arcane project **devEscrow** (`77c10db2…`) on the ARM64 host at `89.58.32.32`. All operational decisions, env var edits, and container restarts target this stack.
The `dev-amn` stack is the authoritative dev deployment. The `escrow-multi` stack is the only valid target for `feature/white-label-shops` branch work.
The legacy compose (`deployment/docker-compose.yml`) is kept for historical reference. It uses an nginx sidecar, Traefik labels, and images from the old `git.manko.yoga` registry. Do not deploy from it.
> [!warning] Branch / stack isolation
> Work on `feature/white-label-shops` must NEVER touch `escrow-dev` — no restart, redeploy, or env change. Work on `main` must NEVER touch `escrow-multi`. Each stack must have its own `TELEGRAM_BOT_TOKEN` (different bots). See [[deploy_architecture_two_stacks]].
---
## 2. Services
### 2.1 dev-amn stack (active, `dev.amn.gg`)
| Service | Image | Internal Port | Role |
|---|---|---|---|
| `backend` | `git.tbs.amn.gg/escrow/backend:dev` | 5001 | Express 5 API + Socket.IO + admin seed |
| `frontend` | `git.tbs.amn.gg/escrow/frontend:dev` | 8083 | Next.js SSR app |
| `refscanner` | `git.tbs.amn.gg/escrow/scanner:dev` | 8080 | In-house AMN payment scanner (SQLite) |
| `postgres` | `postgres:18-alpine` | 5432 | Primary datastore (auth, marketplace, PG stores) |
| `redis` | `redis:8-alpine` | 6379 | Session cache, rate-limit counters, job queues |
| `mongodb` | `mongo:8.0-noble` | 27017 | Legacy datastore — dev boot parity only, retired in prod |
| `gatus` | `twinproduction/gatus:latest` | 8080 (mapped 8084) | Uptime monitoring + Telegram alerting |
| `postgres` | `postgres:18-alpine` | 5432 | Primary datastore (auth + all 8 Postgres domain stores) |
| `redis` | `redis:8-alpine` | 6379 | Session cache, rate-limit counters, pub/sub |
| `mongodb` | `mongo:8.0-noble` | 27017 | Legacy datastore — dev boot parity only; retire once remaining reads migrated |
> **Note on refscanner:** The in-house scanner (`provider: "amn.scanner"`) persists state in a SQLite file at `/data/scanner.db` inside the container. It does not expose a port on `shared-web`; the backend calls it via the `default` bridge by container alias `refscanner`.
### 2.2 escrow-multi stack (`multi.amn.gg`)
> **Note on mongodb:** The Mongo container is retained for dev stack parity because `MONGODB_URI` is still present in the env. It will be removed once the backend's remaining Mongo reads are migrated to Postgres. See [[mongo-to-pg-migration-guide]] and [[mongo_retirement_status]].
| Service | Image | Internal Port | Role |
|---|---|---|---|
| `migrate` | `node:22-alpine` | n/a | One-shot Drizzle migration runner |
| `backend` | `git.tbs.amn.gg/escrow/backend:multi` | 5001 | Express API, tenant services, storefront API, tenant bot webhook |
| `frontend` | `git.tbs.amn.gg/escrow/frontend:multi` | 8083 | Next.js for `multi.amn.gg`, tenant subdomains, dashboards |
| `postgres` | `postgres:18-alpine` | 5432 | Isolated multi-stack database (`escrow_multi`) |
| `redis` | `redis:8-alpine` | 6379 | Isolated multi-stack cache/session/pub-sub |
### 2.3 Legacy compose services (`deployment/docker-compose.yml`)
> These are documented for reference only. Do not deploy from this file.
| Service | Image | Host Port | Role |
|---|---|---|---|
| `nginx` | `nginx:alpine` | 80 (via Traefik) | Reverse proxy in front of backend and frontend |
| `nickDev-marketplace` | `git.manko.yoga/manawenuz/escrow-backend:dev` | — | Backend (legacy registry) |
| `mongodb` | `mongo:8.0-noble` | — | Mongo datastore |
| `postgres` | `postgres:18-alpine` | — | Postgres datastore |
| `redis` | `redis:8-alpine` | — | Cache/sessions |
| `nickDev-frontend` | `git.manko.yoga/manawenuz/escrow-frontend:dev` | 8083 | Frontend (legacy registry) |
| `gatus` | `twinproduction/gatus:latest` | 8084→8080 | Uptime monitoring + Telegram alerting |
---
## 3. Architecture Diagram
### dev-amn (active)
```
Internet (HTTPS 443 / HTTP 80)
│
▼
┌───────────────────────────┐
│ Cloudflare CDN / Proxy │
│ amn.gg / dev.amn.gg │
└─────────────┬─────────────┘
│
▼ (origin)
┌─────────────────────────────────────────────────┐
│ Host: 89.58.32.32 │
│ │
│ ┌────────────────────────────────────────────┐ │
│ │ infra-caddy (Arcane project "infra") │ │
│ │ ports 80:80, 443:443 on host │ │
│ │ reads Caddyfile at │ │
│ │ /opt/arcane/data/projects/infra/Caddyfile │ │
│ └───┬───────────────────────────┬────────────┘ │
│ │ /api/* /socket.io/* │ /* │
│ │ /uploads/* │ │
│ ▼ ▼ │
│ ┌────────────┐ ┌────────────────┐ │
│ │ backend │ │ frontend │ │
│ │ :5001 │ │ :8083 │ │
│ │ shared-web │ │ shared-web │ │
│ └──┬──┬──┬───┘ └────────────────┘ │
│ │ │ │ │
│ │ │ └──────────────────────┐ │
│ │ │ ▼ │
│ │ │ ┌────────────────────┐ │
│ │ │ │ refscanner │ │
│ │ │ │ :8080 (default │ │
│ │ │ │ bridge only) │ │
│ │ │ └────────────────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌───────────────┐ │
│ │ postgres │ │ redis │ │ mongodb │ │
│ │ :5432 │ │ :6379 │ │ :27017 │ │
│ │ (default │ │ (default │ │ (default only,│ │
│ │ only) │ │ only) │ │ legacy) │ │
│ └──────────┘ └──────────┘ └───────────────┘ │
│ │
│ ┌────────────────────────────────────────────┐ │
│ │ gatus :8084 (mapped from :8080) │ │
│ │ monitors dev.amn.gg + amn.gg + external │ │
│ └────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────┘
┌───────────────────────────────┐
│ Cloudflare CDN / Proxy │
│ amn.gg / dev.amn.gg │
└─────────────┬─────────────────┘
│ (origin request)
▼
┌─────────────────────────────────────────────────────┐
│ Host: 89.58.32.32 │
│ │
│ ┌──────────────────────────────────────────────┐ │
│ │ infra-caddy (Arcane project "infra") │ │
│ │ ports 80:80, 443:443 bound to host │ │
│ │ Caddyfile: /opt/arcane/data/projects/ │ │
│ │ infra/Caddyfile │ │
│ └───────┬─────────────────────────┬────────────┘ │
│ │ /api/* /socket.io/* │ /* │
│ │ /uploads/* │ │
│ ▼ ▼ │
│ ┌───────────────┐ ┌────────────────────┐ │
│ │ backend │ │ frontend │ │
│ │ :5001 │ │ :8083 │ │
│ │ shared-web │ │ shared-web │ │
│ └──┬──┬────┬────┘ └────────────────────┘ │
│ │ │ │ │
│ │ │ └────────────────────┐ │
│ │ │ ▼ │
│ │ │ ┌──────────────────────┐ │
│ │ │ │ refscanner │ │
│ │ │ │ :8080 │ │
│ │ │ │ (default only) │ │
│ │ │ └──────────────────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌────────────────┐ │
│ │ postgres │ │ redis │ │ mongodb │ │
│ │ :5432 │ │ :6379 │ │ :27017 │ │
│ │ (default │ │ (default │ │ (default only, │ │
│ │ only) │ │ only) │ │ legacy) │ │
│ └──────────┘ └──────────┘ └────────────────┘ │
│ │
└─────────────────────────────────────────────────────┘
Networks:
shared-web ─── external, attached: backend, frontend
default ─── internal bridge: all services
shared-web (external) ─ backend + frontend (reachable by infra-caddy)
default (bridge) ─ all containers on the stack
```
### Legacy compose (reference only)
```
Internet (HTTPS 443)
│
▼
┌─────────────────────────┐
│ Traefik (external) │
│ escrowdev.ch.manko. │
│ yoga → nginx:80 │
│ gatus.ch.manko.yoga → │
│ gatus:8080 │
└────────────┬────────────┘
│
▼
┌────────────────────────────────────────┐
│ nginx (traefik_public + default) │
│ nickDev-nginx │
│ conf: /var/data/escrowDev/nginx/ │
└────────┬──────────────────┬────────────┘
│ /api /socket.io │ /*
▼ ▼
┌────────────────┐ ┌────────────────────┐
│ nickDev- │ │ nickDev-frontend │
│ marketplace │ │ :8083 │
│ backend │ │ (watchtower) │
│ (watchtower) │ └────────────────────┘
└──┬──┬──────────┘
│ │
▼ ▼
┌──────────┐ ┌──────────┐ ┌────────────┐
│ mongodb │ │ postgres │ │ redis │
│ :27017 │ │ :5432 │ │ :6379 │
└──────────┘ └──────────┘ └────────────┘
┌────────────────────────────────────────┐
│ gatus :8084→:8080 (traefik_public) │
└────────────────────────────────────────┘
```
---
## 4. Networks
| Network | Type | Services Attached | Purpose |
|---|---|---|---|
| `default` (bridge) | Internal | All services | Container-to-container communication |
| `shared-web` | External (pre-existing) | `backend`, `frontend` | Allows infra-caddy to proxy by container name |
| `traefik_public` | External (legacy only) | nginx, gatus (legacy compose) | Old Traefik-based ingress on `git.manko.yoga` host |
| Network | Type | Present in | Services Attached | Purpose |
|---|---|---|---|---|
| `default` (bridge) | Internal auto | dev-amn, legacy | All services | Container-to-container communication |
| `shared-web` | External (pre-existing) | dev-amn, escrow-multi | `backend`, `frontend` | Allows infra-caddy to proxy by container name |
| `traefik_public` | External (pre-existing) | Legacy compose only | `nginx`, `gatus` | Old Traefik-based ingress on `git.manko.yoga` host |
**Key rules:**
- `postgres`, `redis`, `mongodb` are on `default` only — no external exposure.
- `postgres`, `redis`, `mongodb` are on `default` only — never externally reachable.
- `refscanner` is on `default` only; backend reaches it via alias `refscanner:8080`.
- Any new public-facing service must join `shared-web` AND get a Caddyfile block. See [[Shared Infra (89.58.32.32)]] and section 6 below.
- `shared-web` must exist on the host before `docker compose up`. It is created by the `infra` project.
- Any new public-facing service must join `shared-web` AND get a Caddyfile vhost block.
- `shared-web` must exist on the host before `docker compose up` — it is created by the Arcane `infra` project.
---
## 5. Volumes and Bind Mounts
All data volumes in the `dev-amn` stack use relative bind mounts under `./data/` (resolved to `/opt/arcane/data/projects/escrow-dev/data/` on the server):
### dev-amn stack
All data volumes use relative bind mounts under `./data/` (resolved to `/opt/arcane/data/projects/escrow-dev/data/` on the server):
| Service | Host Path | Container Path | Notes |
|---|---|---|---|
| `backend` | `./data/uploads` | `/app/uploads` | User-uploaded files |
| `backend` | `./data/uploads` | `/app/uploads` | User-uploaded files (served via `/uploads/*`) |
| `refscanner` | `./data/scanner` | `/data` | SQLite DB at `/data/scanner.db` |
| `postgres` | `./data/postgres` | `/var/lib/postgresql/data` | `PGDATA` subdir workaround: actual data at `./data/postgres/pgdata` |
| `redis` | `./data/redis` | `/data` | Persistence dump |
| `mongodb` | `./data/mongo` | `/data/db` | Legacy; can be deleted once Mongo retired |
| `gatus` | `./gatus/config.yaml` | `/config/config.yaml` (ro) | Monitoring config — part of repo |
| `postgres` | `./data/postgres` | `/var/lib/postgresql/data` | `PGDATA=/var/lib/postgresql/data/pgdata` (subdir workaround) |
| `redis` | `./data/redis` | `/data` | RDB persistence dump |
| `mongodb` | `./data/mongo` | `/data/db` | Legacy; can be removed once Mongo retired |
**Postgres volume note:** `postgres:18` introduced a version-scoped data directory layout and refuses to init directly into a volume root that already contains files from a different layout. The compose file sets `PGDATA=/var/lib/postgresql/data/pgdata` to place actual data in a subdirectory of the mount, avoiding init conflicts.
The Gatus config (`deployment/gatus/config.yaml`) is bind-mounted read-only into the gatus container at `/config/config.yaml`. It lives in the repo, not in `./data/`.
**Legacy compose** (`deployment/docker-compose.yml`) uses absolute host paths under `/var/data/escrowDev/` and does not share volumes with the dev-amn stack.
> **Postgres volume note:** `postgres:18` uses a version-scoped data directory layout and refuses to init into a volume root that already contains files from a different layout. `PGDATA` is set to a subdirectory (`/var/lib/postgresql/data/pgdata`) inside the mount to avoid init conflicts.
### Legacy compose bind mounts (`/var/data/escrowDev/`)
| Service | Host Path | Container Path |
|---|---|---|
| `nginx` | `/var/data/escrowDev/nginx/nginx.conf` | `/etc/nginx/nginx.conf` (ro) |
| `nginx` | `/var/data/escrowDev/nginx/logs` | `/var/log/nginx` |
| `nginx` / `backend` | `/var/data/escrowDev/uploads` | `/uploads` / `/app/uploads` |
| `mongodb` | `/var/data/escrowDev/mongodb_data` | `/data/db` |
| `mongodb` | `/var/data/escrowDev/mongo-init` | `/docker-entrypoint-initdb.d` |
| `postgres` | `/var/data/escrowDev/postgres_data` | `/var/lib/postgresql` |
| `redis` | `/var/data/escrowDev/redis_data` | `/data` |
---
@@ -138,11 +212,12 @@ All data volumes in the `dev-amn` stack use relative bind mounts under `./data/`
Ingress for `89.58.32.32` is handled exclusively by **infra-caddy** — the Caddy container in the Arcane project `infra`. It owns host ports 80 and 443. No service should bind those ports directly.
### Current Caddyfile block (dev.amn.gg)
### Caddyfile block for dev.amn.gg
Located at `/opt/arcane/data/projects/infra/Caddyfile` on the server (and mirrored in `deployment/dev-amn/Caddyfile` for reference):
Live location on server: `/opt/arcane/data/projects/infra/Caddyfile`
Reference copy in repo: `deployment/dev-amn/Caddyfile`
```
```caddy
{
email manwe@manko.yoga
auto_https disable_redirects
@@ -158,25 +233,46 @@ dev.amn.gg {
}
```
- `auto_https disable_redirects` — Cloudflare proxy sits in front; Caddy should not force HTTP→HTTPS redirects at origin.
- Routes by path prefix: `/api/*`, `/socket.io/*`, `/uploads/*` go to `backend:5001`; everything else to `frontend:8083`.
- Container names are resolved via the `shared-web` network.
- `auto_https disable_redirects` — Cloudflare proxy sits in front; Caddy must not force HTTP→HTTPS redirects at origin.
- Routes by path prefix: `/api/*`, `/socket.io/*`, `/uploads/*` → `backend:5001`; everything else → `frontend:8083`.
- Container names resolve via the `shared-web` network (both `backend` and `frontend` join it).
### Adding a new public service
1. Add the service to `deployment/dev-amn/docker-compose.yml` with `networks: shared-web: {}`.
2. Edit `/opt/arcane/data/projects/infra/Caddyfile` on the server — add a new vhost block or path matcher.
3. Reload Caddy (no restart needed):
3. Reload Caddy without restarting:
```bash
docker exec infra-caddy caddy reload --config /etc/caddy/Caddyfile
ssh -i ~/CascadeProjects/wzp root@89.58.32.32 \
"docker exec infra-caddy caddy reload --config /etc/caddy/Caddyfile"
```
4. Verify via `curl -I https://dev.amn.gg/<new-path>`.
4. Verify: `curl -I https://dev.amn.gg/<new-path>`
See also: [[Shared Infra (89.58.32.32)]]
### Legacy compose: Traefik labels
In the legacy compose, nginx and gatus expose themselves to Traefik via Docker labels:
```yaml
# nginx
traefik.http.routers.escrowDev.rule=Host(`escrowdev.ch.manko.yoga`)
traefik.http.routers.escrowDev.entrypoints=https
traefik.http.services.escrowDev.loadbalancer.server.port=80
# gatus
traefik.http.routers.gatus.rule=Host(`gatus.ch.manko.yoga`)
traefik.http.routers.gatus.entrypoints=https
traefik.http.services.gatus.loadbalancer.server.port=8080
```
---
## 7. Gatus Monitoring
Gatus runs as a sidecar in the `default` network. Config is at `deployment/gatus/config.yaml` (bind-mounted read-only). Alerts are delivered via Telegram.
Gatus runs as a sidecar in the `default` network. Config is at `deployment/gatus/config.yaml` (bind-mounted read-only). Alerts are delivered via Telegram to `GATUS_TELEGRAM_CHAT_ID`.
In the legacy compose, Gatus is exposed on host port `8084` (mapped from container `:8080`) and publicly accessible via Traefik at `gatus.ch.manko.yoga`.
### Alert policy
@@ -187,69 +283,80 @@ Gatus runs as a sidecar in the `default` network. Config is at `deployment/gatus
| Send on resolved | Yes |
| Alert channel | Telegram (`GATUS_TELEGRAM_BOT_TOKEN` / `GATUS_TELEGRAM_CHAT_ID`) |
Prod endpoints use `failure-threshold: 2` (faster alerting).
### Monitored endpoints
| Name | Group | URL | Interval | Key Conditions |
|---|---|---|---|---|
| `backend-dev-version` | backend-dev | `https://dev.amn.gg/api/version` | 60s | HTTP 200, body.version not empty |
| `backend-dev-health` | backend-dev | `https://dev.amn.gg/api/health` | 30s | HTTP 200, all PG store modes = postgres, redis ok, RN chain+token registry loaded |
| `backend-prod-version` | backend-prod | `https://amn.gg/api/version` | 60s | HTTP 200, body.version not empty (failure-threshold 2) |
| `backend-prod-health` | backend-prod | `https://amn.gg/api/health` | 30s | HTTP 200, db/postgres/redis/RN registries ok (failure-threshold 2) |
| `frontend-dev` | frontend | `https://dev.amn.gg/` | 60s | HTTP 200, response < 3000ms |
| `frontend-prod` | frontend | `https://amn.gg/` | 60s | HTTP 200, response < 3000ms (failure-threshold 2) |
| `rn-api-reachable` | external | `https://api.request.network/v2/health` | 5m | HTTP 200/401/404 (accepts auth errors — just checks reachability) |
| `backend-dev-version` | backend-dev | `https://dev.amn.gg/api/version` | 60s | HTTP 200, `body.version` not empty |
| `backend-dev-health` | backend-dev | `https://dev.amn.gg/api/health` | 30s | HTTP 200, all 8 PG store modes = postgres, redis ok, RN chain+token registry loaded |
| `backend-prod-version` | backend-prod | `https://amn.gg/api/version` | 60s | HTTP 200, `body.version` not empty |
| `backend-prod-health` | backend-prod | `https://amn.gg/api/health` | 30s | HTTP 200, db/postgres/redis/RN registries ok |
| `frontend-dev` | frontend | `https://dev.amn.gg/` | 60s | HTTP 200, response time < 3000ms |
| `frontend-prod` | frontend | `https://amn.gg/` | 60s | HTTP 200, response time < 3000ms |
| `rn-api-reachable` | external | `https://api.request.network/v2/health` | 5m | HTTP 200/401/404 (checks reachability only) |
| `chainalysis-public-api` | external | `https://public.chainalysis.com/api/v1/address/0x000…` | 5m | HTTP 200 or 404 |
| `bsc-rpc-publicnode` | external | `https://bsc-rpc.publicnode.com` (POST) | 2m | HTTP 200, `result == "0x38"` (BSC mainnet chain ID) |
The `backend-dev-health` check validates that **all 8 domain stores are running on Postgres** (`auth`, `config`, `address`, `category`, `levelConfig`, `shopSettings`, `review`, `notification`). A failure here means a store mode regression or a broken `PG_URL`.
The `backend-dev-health` endpoint validates **all 8 domain stores running on Postgres**: `auth`, `config`, `address`, `category`, `levelConfig`, `shopSettings`, `review`, `notification`. A failure here means a store mode regression or a broken `PG_URL`.
Gatus dashboard is accessible at `:8084` on the host (not publicly proxied by default — access via SSH tunnel or add a Caddyfile block if needed).
Gatus dashboard: `:8084` on the host locally (not publicly proxied by default — access via SSH tunnel, or add a Caddyfile block if public exposure is needed).
---
## 8. Environment Variables
All vars are passed to containers via `.env` at the stack root (`deployment/dev-amn/.env` on the server, `deployment/.env` in the repo as the live dev reference). The file is `chmod 600` and never committed.
All vars are injected via `.env` at the stack root. The server file is `chmod 600` and never committed. The `deployment/.env` in the repo serves as the live dev reference / template.
### Backend
### 8.1 Runtime / Node
| Variable | Description | Example / Default |
| Variable | Description | Default |
|---|---|---|
| `NODE_ENV` | Runtime environment | `production` |
| `PORT` | Express listen port | `5001` |
| `TRUST_PROXY` | Express trust-proxy (required behind Caddy) | `true` |
| `TRUST_PROXY` | Express trust-proxy (required behind Caddy/nginx) | `true` |
| `DEBUG` | Debug namespaces | _(empty)_ |
| `LOG_LEVEL` | Winston log level | `info` |
#### Database
### 8.2 Database — Postgres
| Variable | Description | Example |
| Variable | Description | Default in compose |
|---|---|---|
| `MONGODB_URI` | Mongo connection string | `mongodb://admin:pass@mongodb:27017/marketplace?authSource=admin` |
| `MONGO_INITDB_ROOT_USERNAME` | Mongo root user | `admin` |
| `MONGO_INITDB_ROOT_PASSWORD` | Mongo root password | — |
| `MONGO_INITDB_DATABASE` | Mongo init database | `marketplace` |
| `DB_NAME` | Mongo database name used by app | `amn-db` |
| `PG_URL` | Postgres DSN | `postgres://amanat:pass@amanat-postgres:5432/amanat_dev` |
| `PG_URL` | Postgres DSN | `postgres://amanat:amanat_local@postgres:5432/amanat_dev` |
| `POSTGRES_USER` | Postgres superuser | `amanat` |
| `POSTGRES_PASSWORD` | Postgres superuser password | — |
| `POSTGRES_DB` | Postgres database name | `amanat_dev` |
| `AUTO_SEED_ON_START` | Run seed on boot | `true` |
#### Store modes (dual-write seam)
### 8.3 Database — Mongo (legacy)
| Variable | Description | Default |
|---|---|---|
| `AUTH_STORE` | Auth domain store backend | `postgres` |
| `CONFIG_STORE` | Config domain | `postgres` |
| `ADDRESS_STORE` | Address domain | `postgres` |
| `CATEGORY_STORE` | Category domain | `postgres` |
| `LEVEL_CONFIG_STORE` | Level config domain | `postgres` |
| `SHOP_SETTINGS_STORE` | Shop settings domain | `postgres` |
| `REVIEW_STORE` | Review domain | `postgres` |
| `NOTIFICATION_STORE` | Notification domain | `postgres` |
| `MONGODB_URI` | Mongo connection string | `mongodb://admin:pass@mongodb:27017/marketplace?authSource=admin` |
| `MONGO_INITDB_ROOT_USERNAME` | Mongo root user | `admin` |
| `MONGO_INITDB_ROOT_PASSWORD` | Mongo root password | `changeme_local` |
| `MONGO_INITDB_DATABASE` | Mongo init DB | `marketplace` |
| `DB_NAME` | Mongo database name used by app | `amn-db` |
#### Auth / Sessions
### 8.4 Store modes (dual-write seam)
All default to `postgres` in the dev-amn compose. Changing any to `mongo` re-routes that domain's reads/writes to MongoDB.
| Variable | Domain | Default |
|---|---|---|
| `AUTH_STORE` | Auth / user accounts | `postgres` |
| `CONFIG_STORE` | App config | `postgres` |
| `ADDRESS_STORE` | User addresses | `postgres` |
| `CATEGORY_STORE` | Marketplace categories | `postgres` |
| `LEVEL_CONFIG_STORE` | Gamification level config | `postgres` |
| `SHOP_SETTINGS_STORE` | Per-shop settings | `postgres` |
| `REVIEW_STORE` | Product / seller reviews | `postgres` |
| `NOTIFICATION_STORE` | User notifications | `postgres` |
See [[mongo_retirement_status]] and [[mongo-to-pg-migration-guide]].
### 8.5 Auth / Sessions
| Variable | Description |
|---|---|
@@ -257,14 +364,14 @@ All vars are passed to containers via `.env` at the stack root (`deployment/dev-
| `JWT_EXPIRES_IN` | Access token TTL (e.g. `7d`) |
| `REFRESH_TOKEN_EXPIRES_IN` | Refresh token TTL (e.g. `30d`) |
#### Redis
### 8.6 Redis
| Variable | Description |
|---|---|
| `REDIS_URI` | Redis connection string (includes password) |
| `REDIS_PASSWORD` | Redis auth password (standalone, if not in URI) |
| `REDIS_PASSWORD` | Redis auth password (standalone form) |
#### URLs / CORS
### 8.7 URLs / CORS
| Variable | Description |
|---|---|
@@ -274,23 +381,23 @@ All vars are passed to containers via `.env` at the stack root (`deployment/dev-
| `BACKEND_URL` | Backend origin |
| `CORS_ORIGIN` | Allowed CORS origin |
#### File uploads
### 8.8 File uploads
| Variable | Description | Default |
|---|---|---|
| `UPLOAD_PATH` | Upload directory inside container | `/app/uploads` |
| `MAX_FILE_SIZE` | Max upload bytes | `52428800` (50 MB) |
#### Rate limiting
### 8.9 Rate limiting
| Variable | Description | Default |
|---|---|---|
| `RATE_LIMIT_WINDOW_MS` | Window for rate limiter | `900000` (15 min) |
| `RATE_LIMIT_MAX_REQUESTS` | Max requests per window | `100` |
> GET `/api/payment/:id` must bypass `paymentLimiter` — see [[backend_rate_limits]].
> GET `/api/payment/:id` must bypass `paymentLimiter` (30 req/15 min) — see [[backend_rate_limits]].
#### SMTP
### 8.10 SMTP
| Variable | Description |
|---|---|
@@ -301,7 +408,7 @@ All vars are passed to containers via `.env` at the stack root (`deployment/dev-
| `SMTP_PASS` | SMTP password |
| `SMTP_FROM` | From address |
#### WebAuthn (Passkeys)
### 8.11 WebAuthn (Passkeys)
| Variable | Description |
|---|---|
@@ -309,7 +416,7 @@ All vars are passed to containers via `.env` at the stack root (`deployment/dev-
| `WEBAUTHN_RP_NAME` | Relying party display name |
| `WEBAUTHN_RP_ORIGIN` | Relying party origin URL |
#### Admin seed
### 8.12 Admin seed
| Variable | Description |
|---|---|
@@ -318,14 +425,14 @@ All vars are passed to containers via `.env` at the stack root (`deployment/dev-
| `ADMIN_FIRST_NAME` | Admin first name |
| `ADMIN_LAST_NAME` | Admin last name |
#### Google OAuth
### 8.13 Google OAuth
| Variable | Description |
|---|---|
| `GOOGLE_CLIENT_ID` | Google OAuth client ID |
| `GOOGLE_CLIENT_SECRET` | Google OAuth client secret |
#### OpenAI
### 8.14 OpenAI
| Variable | Description |
|---|---|
@@ -334,13 +441,13 @@ All vars are passed to containers via `.env` at the stack root (`deployment/dev-
| `OPENAI_MAX_TOKENS` | Max tokens per request |
| `OPENAI_TEMPERATURE` | Sampling temperature |
#### Sentry
### 8.15 Sentry
| Variable | Description |
|---|---|
| `SENTRY_DSN` | Sentry ingest DSN |
#### Wallets / Blockchain
### 8.16 Wallets / Blockchain
| Variable | Description |
|---|---|
@@ -349,7 +456,7 @@ All vars are passed to containers via `.env` at the stack root (`deployment/dev-
| `ADMIN_PAYOUT_WALLET_ADDRESS` | Admin payout destination |
| `RECEIVER_WALLET_ADDRESS` | Default receiver wallet |
#### DePay
### 8.17 DePay
| Variable | Description |
|---|---|
@@ -359,7 +466,7 @@ All vars are passed to containers via `.env` at the stack root (`deployment/dev-
| `DEPAY_ALLOWED_TOKENS` | Allowed payment tokens |
| `DEPAY_PUBLIC_KEY` | DePay public key (PEM) |
#### SHKeeper
### 8.18 SHKeeper
| Variable | Description |
|---|---|
@@ -375,24 +482,25 @@ All vars are passed to containers via `.env` at the stack root (`deployment/dev-
| `SHKEEPER_CALLBACK_SECRET` | Callback verification secret |
| `SHKEEPER_WEBHOOK_SECRET` | Webhook verification secret |
#### Request Network
### 8.19 Request Network
| Variable | Description |
|---|---|
| `REQUEST_NETWORK_ENABLED` | Enable RN provider |
| `REQUEST_NETWORK_WEBHOOK_SECRET` | Webhook signature secret |
| `REQUEST_NETWORK_WEBHOOK_CALLBACK_URL` | Public callback URL |
| `REQUEST_NETWORK_PAYMENT_CURRENCY` | Currency (e.g. `USDC`) |
| `REQUEST_NETWORK_NETWORK` | Chain (e.g. `bsc`) |
| `REQUEST_NETWORK_MERCHANT_REFERENCE` | Merchant address reference |
| `REQUEST_NETWORK_API_BASE_URL` | RN API root |
| `REQUEST_NETWORK_API_KEY` | RN API key |
| `REQUEST_NETWORK_ORIGIN` | Origin header sent to RN |
| `REQUEST_NETWORK_ALLOW_TEST_WEBHOOKS` | Allow test events (default `false`) |
| Variable | Description | Default |
|---|---|---|
| `REQUEST_NETWORK_ENABLED` | Enable RN provider | — |
| `REQUEST_NETWORK_WEBHOOK_SECRET` | Webhook signature secret | — |
| `REQUEST_NETWORK_WEBHOOK_CALLBACK_URL` | Public callback URL | — |
| `REQUEST_NETWORK_PAYMENT_CURRENCY` | Currency (e.g. `USDC`) | — |
| `REQUEST_NETWORK_NETWORK` | Chain (e.g. `bsc`) | — |
| `REQUEST_NETWORK_MERCHANT_REFERENCE` | Merchant address reference | — |
| `REQUEST_NETWORK_API_BASE_URL` | RN API root | — |
| `REQUEST_NETWORK_API_KEY` | RN API key | — |
| `REQUEST_NETWORK_ORIGIN` | Origin header sent to RN | — |
| `REQUEST_NETWORK_ALLOW_TEST_WEBHOOKS` | Allow test events | `false` |
> RN webhook discriminator is `payload.event` (not `eventType`) — see [[rn_webhook_event_field]].
> RN proxy addresses differ per chain (not canonical CREATE2 addresses) — see [[rn_proxy_addresses_per_chain]].
#### Transaction safety
### 8.20 Transaction safety
| Variable | Description | Default |
|---|---|---|
@@ -402,7 +510,7 @@ All vars are passed to containers via `.env` at the stack root (`deployment/dev-
| `TRANSACTION_SAFETY_MIN_CONFIRMATIONS` | Min block confirmations | `12` |
| `TRANSACTION_SAFETY_AML_PROVIDER` | AML provider (`none`, `chainalysis`) | `none` |
#### Payment routing
### 8.21 Payment routing
| Variable | Description |
|---|---|
@@ -411,23 +519,25 @@ All vars are passed to containers via `.env` at the stack root (`deployment/dev-
| `PAYMENT_PROVIDER_MODE` | `live` or `test` |
| `PAYMENT_ROLLBACK_PROVIDER` | Fallback provider |
#### Telegram
### 8.22 Telegram
| Variable | Description |
|---|---|
| `TELEGRAM_FEATURE_ENABLED` | Enable Telegram integration |
| `TELEGRAM_MINIAPP_ENABLED` | Enable Mini App |
| `TELEGRAM_WEBHOOK_ENABLED` | Enable webhook receiver |
| `TELEGRAM_BOT_TOKEN` | Main bot token |
| `TELEGRAM_BOT_TOKEN` | Main bot token (`@amnescrow_Bot` for dev) |
| `TELEGRAM_WEBHOOK_SECRET_TOKEN` | Webhook secret for validation |
| `TELEGRAM_INITDATA_MAX_AGE_SEC` | Max age for initData |
| `TELEGRAM_INITDATA_REPLAY_WINDOW_MS` | Replay protection window |
| `TELEGRAM_WEBHOOK_REPLAY_WINDOW_MS` | Webhook replay protection window |
| `TELEGRAM_SESSION_TTL_SEC` | Session TTL |
| `TG_NOTIFY_BOT_TOKEN` | Ops/monitoring bot token (amnGG_MonitorBot) |
| `TG_NOTIFY_BOT_TOKEN` | Ops/monitoring bot token (`amnGG_MonitorBot`) |
| `TG_NOTIFY_CHATS` | Comma-separated chat IDs for ops notifications |
#### Pangolin / Newt (VPN mesh — optional)
> Each stack (dev, multi) must have a **different `TELEGRAM_BOT_TOKEN`** — sharing a bot token kills one stack's webhook when the other registers. See [[escrow_multi_woodpecker_deploy]] and stack isolation warning above.
### 8.23 Pangolin / Newt (optional VPN mesh)
| Variable | Description |
|---|---|
@@ -435,13 +545,7 @@ All vars are passed to containers via `.env` at the stack root (`deployment/dev-
| `NEWT_ID` | Newt node ID |
| `NEWT_SECRET` | Newt node secret |
#### Testnet chains
| Variable | Description |
|---|---|
| `ENABLE_TESTNET_CHAINS` | Expose testnet chain configs | Set to `true` in dev-amn compose override |
### Frontend (NEXT_PUBLIC_*)
### 8.24 Frontend (NEXT_PUBLIC_*)
| Variable | Description |
|---|---|
@@ -458,7 +562,7 @@ All vars are passed to containers via `.env` at the stack root (`deployment/dev-
| `NEXT_PUBLIC_PASSKEY_RP_NAME` | WebAuthn RP name |
| `NEXT_PUBLIC_PASSKEY_RP_ID` | WebAuthn RP ID |
| `NEXT_PUBLIC_PASSKEY_ORIGIN` | WebAuthn origin |
| `NEXT_PUBLIC_BACKEND_URL` | Backend origin (used for direct calls) |
| `NEXT_PUBLIC_BACKEND_URL` | Backend origin (direct calls) |
| `NEXT_PUBLIC_DEPAY_INTEGRATION_ID` | DePay integration ID |
| `NEXT_PUBLIC_IS_DEVELOPMENT` | Development flag |
| `NEXT_PUBLIC_ENABLE_DEBUG` | Enable client debug logging |
@@ -466,7 +570,9 @@ All vars are passed to containers via `.env` at the stack root (`deployment/dev-
| `NEXT_PUBLIC_TELEGRAM_BOT_ID` | Telegram bot numeric ID |
| `BUILD_STATIC_EXPORT` | Enable `next export` mode (`false` for SSR) |
### Gatus
> `NEXT_PUBLIC_*` vars are baked into the frontend bundle at build time. Never put secrets in them. Frontend env changes require a fresh image build and redeploy.
### 8.25 Gatus
| Variable | Description |
|---|---|
@@ -477,9 +583,9 @@ All vars are passed to containers via `.env` at the stack root (`deployment/dev-
## 9. Deploy Workflow
### 9.1 Normal image update (CI-driven)
### 9.1 Normal image update (CI-driven — dev-amn)
Woodpecker CI builds `backend` and `frontend` images, pushes tags to `git.tbs.amn.gg/escrow/` on merge to `dev`, then triggers an Arcane GitOps sync which pulls the new image and recreates the container.
Woodpecker CI builds backend and frontend images, pushes to `git.tbs.amn.gg/escrow/`, then triggers an Arcane GitOps sync which pulls the new image and recreates the container.
```
git push origin dev
@@ -487,31 +593,52 @@ git push origin dev
└─► docker push git.tbs.amn.gg/escrow/backend:dev
└─► docker push git.tbs.amn.gg/escrow/frontend:dev
└─► arcane-cli gitops sync cf6c9eab… (or watchtower polls)
└─► escrow-backend container restarted with new image
└─► escrow-frontend container restarted with new image
└─► escrow-backend restarted with new image
└─► escrow-frontend restarted with new image
```
> Always bump the version in `package.json` + lock before pushing, otherwise the CI build may not register as a new deploy. See [[version_bump_before_ci]].
> Always bump `package.json` version before pushing. See [[version_bump_before_ci]].
### 9.2 Manual deploy (backend hotfix — no registry)
### 9.2 escrow-multi deploys (white-label stack)
For urgent backend fixes without a full CI cycle, use the local-build pattern (the dev stack has `pull_policy: always` but the override `docker-compose.override.yml` sets `pull_policy: never` for the `escrow-backend-local:dev` image path):
**Always use Woodpecker. Never use manual rsync/docker-build/ssh for escrow-multi.**
```bash
# 1. Copy changed files to build tree on server
# 1. Make changes, bump version in package.json
# 2. Commit
git commit -m "fix: description (vX.Y.Z)"
# 3. Push to Forgejo (remote is "forgejo", not "origin")
git push forgejo feature/white-label-shops
# 4. Monitor Woodpecker pipeline
source ~/CascadeProjects/escrow/.env
WOODPECKER_SERVER=$WOODPECKER_SERVER WOODPECKER_TOKEN=$WOODPECKER_TOKEN \
woodpecker-cli pipeline ls escrow/backend
```
Frontend is a separate Woodpecker project (`escrow/frontend`). Both push targets trigger their respective pipelines.
### 9.3 Manual hotfix deploy (backend only — no registry cycle)
For urgent fixes without a full CI cycle, build locally on the server:
```bash
# 1. Copy changed files to build tree
scp -i ~/CascadeProjects/wzp src/services/auth/authRoutes.ts \
root@89.58.32.32:/tmp/escrow-backend-build/src/services/auth/
# 2. Rebuild image on server (~3 min, ARM64)
ssh -i ~/CascadeProjects/wzp root@89.58.32.32 \
"cd /tmp/escrow-backend-build && docker build -f Dockerfile.prod -t escrow-backend-local:dev ."
"cd /tmp/escrow-backend-build && docker build -f Dockerfile.prod \
-t escrow-backend-local:dev ."
# 3. Restart the backend container
ssh -i ~/CascadeProjects/wzp root@89.58.32.32 \
"cd /opt/arcane/data/projects/escrow-dev && docker compose up -d backend"
```
### 9.3 Bringing the stack up/down
The `docker-compose.override.yml` at `/opt/arcane/data/projects/escrow-dev/docker-compose.override.yml` sets `pull_policy: never` for `escrow-backend-local:dev` so watchtower never clobbers it.
### 9.4 Bringing the stack up/down
```bash
# via Arcane CLI (preferred)
@@ -526,9 +653,7 @@ ssh -i ~/CascadeProjects/wzp root@89.58.32.32 \
"cd /opt/arcane/data/projects/escrow-dev && docker compose down"
```
### 9.4 Reloading Caddy after Caddyfile edits
Edit `/opt/arcane/data/projects/infra/Caddyfile` on the server, then:
### 9.5 Reloading Caddy after Caddyfile edits
```bash
ssh -i ~/CascadeProjects/wzp root@89.58.32.32 \
@@ -537,17 +662,17 @@ ssh -i ~/CascadeProjects/wzp root@89.58.32.32 \
No container restart needed.
### 9.5 Updating env vars
### 9.6 Updating env vars
1. Edit `.env` on the server: `/opt/arcane/data/projects/escrow-dev/.env`
2. Restart affected service:
2. Restart affected container:
```bash
ssh -i ~/CascadeProjects/wzp root@89.58.32.32 \
"cd /opt/arcane/data/projects/escrow-dev && docker compose up -d backend"
```
Frontend env vars baked at build time (via `NEXT_PUBLIC_*`) require a fresh image rebuild.
3. Frontend `NEXT_PUBLIC_*` vars are baked at build time — they require a fresh image build and full redeploy via CI.
### 9.6 Verifying a deploy
### 9.7 Verifying a deploy
```bash
# Check running containers
@@ -556,7 +681,7 @@ arcane-cli project status devEscrow
# Check backend version
curl https://dev.amn.gg/api/version
# Check health (all stores + registries)
# Check health (all stores + RN registries)
curl https://dev.amn.gg/api/health | jq .
# Tail backend logs
@@ -564,48 +689,48 @@ ssh -i ~/CascadeProjects/wzp root@89.58.32.32 \
"docker logs -f escrow-backend --tail 100"
```
> CI ✓ green does NOT guarantee the new image was pushed to the registry. Always verify `curl /api/version` returns the expected version. See [[woodpecker_silent_build_fail]].
> CI ✓ green does NOT guarantee the new image was pushed. Always verify `curl /api/version` returns the expected version. See [[woodpecker_silent_build_fail]].
---
## 10. Dev vs Prod Differences
| Aspect | dev-amn (dev.amn.gg) | Prod (amn.gg) |
| Aspect | dev-amn (`dev.amn.gg`) | Prod (`amn.gg`) |
|---|---|---|
| Compose file | `deployment/dev-amn/docker-compose.yml` | Separate prod stack (not in this repo) |
| Image registry | `git.tbs.amn.gg/escrow` | Same registry, prod tags |
| Image tag | `:dev` | `:latest` or versioned |
| MongoDB | Present (dev parity) | Retired |
| MongoDB | Present (dev parity — retired in prod) | Not present |
| `ENABLE_TESTNET_CHAINS` | `true` (compose override) | Not set / `false` |
| `NODE_ENV` | `production` (same) | `production` |
| `NODE_ENV` | `production` | `production` |
| `NEXT_PUBLIC_IS_DEVELOPMENT` | `false` | `false` |
| `PAYMENT_PROVIDER_MODE` | `live` | `live` |
| `REQUEST_NETWORK_ALLOW_TEST_WEBHOOKS` | can be `true` for RN testing | `false` |
| `REQUEST_NETWORK_ALLOW_TEST_WEBHOOKS` | May be `true` for RN testing | `false` |
| `TRANSACTION_SAFETY_MIN_CONFIRMATIONS` | `12` (default) | `12` (default) |
| Gatus monitoring | Monitors both dev + prod endpoints | Shared Gatus instance |
| TLS | Cloudflare proxy → Caddy (`disable_redirects`) | Same |
| Version bump | Required before CI push | Required |
| Watchtower labels | Present in legacy compose | Prod stack may differ |
| Gatus monitoring | Monitors both dev + prod endpoints | N/A (shared gatus instance) |
| TLS | Cloudflare proxy → Caddy (disable_redirects) | Same |
| Version bump requirement | Required before CI push | Required |
---
## 11. Secret Management
**The `.env` file on the server is the single source of runtime secrets. It is never committed.**
The `.env` file on the server is the single source of runtime secrets. It is never committed.
- Location on server: `/opt/arcane/data/projects/escrow-dev/.env`
- Permissions: `chmod 600` owned by root
- Reference template: `deployment/.env` (in repo — contains live dev values, treated as low-sensitivity dev config; rotate before prod use)
- `.gitleaks.toml` in `deployment/` configures secret scanning exclusions for the repo
- **Server location:** `/opt/arcane/data/projects/escrow-dev/.env`
- **Permissions:** `chmod 600`, owned by root
- **Repo template:** `deployment/.env` — contains live dev values, treated as low-sensitivity dev config; rotate all values before using in production
### Rules
1. Never commit `.env` or any file containing real tokens, passwords, or private keys.
2. Never pass secrets as Dockerfile `ARG`/`ENV` at build time — they appear in image layers. All secrets are runtime-injected via `env_file`.
3. `NEXT_PUBLIC_*` vars are baked into the frontend bundle at build time. Do not place secrets in any `NEXT_PUBLIC_` variable.
4. Wallet addresses (e.g. `ESCROW_WALLET_ADDRESS`) are public on-chain but still kept out of the repo for operational hygiene.
5. For new deployments: copy `deployment/.env` to the server, fill in real values, then `chmod 600`.
6. Gatus bot token and chat ID go into the same `.env` — they are read by the gatus container via `environment:` directives.
7. Telegram bot tokens are high-value secrets — rotate immediately if accidentally pushed.
3. `NEXT_PUBLIC_*` vars are baked into the frontend bundle. Never place secrets in them.
4. Wallet addresses are public on-chain but still kept out of the repo for operational hygiene.
5. For new deployments: copy `deployment/.env` to the server, fill in real values, `chmod 600`.
6. Gatus vars (`GATUS_TELEGRAM_BOT_TOKEN`, `GATUS_TELEGRAM_CHAT_ID`) go into the same `.env`.
7. Telegram bot tokens are high-value — rotate immediately if accidentally pushed.
### Sensitive variable groups
@@ -614,7 +739,7 @@ ssh -i ~/CascadeProjects/wzp root@89.58.32.32 \
| JWT | `JWT_SECRET` | Full session forgery |
| DB credentials | `POSTGRES_PASSWORD`, `REDIS_PASSWORD`, `MONGO_INITDB_ROOT_PASSWORD` | Database access |
| Payment webhook secrets | `REQUEST_NETWORK_WEBHOOK_SECRET`, `DEPAY_WEBHOOK_SECRET`, `SHKEEPER_CALLBACK_SECRET`, `SHKEEPER_WEBHOOK_SECRET` | Fake payment injection |
| Bot tokens | `TELEGRAM_BOT_TOKEN`, `TG_NOTIFY_BOT_TOKEN` | Bot takeover |
| Bot tokens | `TELEGRAM_BOT_TOKEN`, `TG_NOTIFY_BOT_TOKEN` | Bot takeover / webhook hijack |
| OAuth secrets | `GOOGLE_CLIENT_SECRET` | OAuth impersonation |
| API keys | `OPENAI_API_KEY`, `REQUEST_NETWORK_API_KEY`, `SHKEEPER_API_KEY` | Billing / data access |
| Sentry DSN | `SENTRY_DSN` | Error data exfiltration |