nick-doc/10 - Services/deployment.md

---
title: Deployment
tags: [services, deployment, infrastructure, docker]
---

# Deployment

The `deployment/` sub-project contains all Docker Compose definitions, Caddyfile configurations, Gatus monitoring config, and environment templates for running the Amanat escrow platform. Two compose files exist side-by-side reflecting a legacy setup and the current live stack.

---

## 1. Overview

| File | Status | Host | Notes |
|---|---|---|---|
| `deployment/docker-compose.yml` | Legacy | Any | nginx + traefik_public network, images from `git.manko.yoga` registry |
| `deployment/dev-amn/docker-compose.yml` | **Active** | `89.58.32.32` | shared-web + infra-caddy ingress, images from `git.tbs.amn.gg/escrow` |

The `dev-amn` stack is the authoritative deployment. It runs under Arcane project **devEscrow** (`77c10db2…`) on the ARM64 host at `89.58.32.32`. All operational decisions, env var edits, and container restarts target this stack.

The legacy compose (`deployment/docker-compose.yml`) is kept for historical reference. It uses an nginx sidecar, Traefik labels, and images from the old `git.manko.yoga` registry. Do not deploy from it.

---

## 2. Services

| Service | Image | Internal Port | Role |
|---|---|---|---|
| `backend` | `git.tbs.amn.gg/escrow/backend:dev` | 5001 | Express 5 API + Socket.IO + admin seed |
| `frontend` | `git.tbs.amn.gg/escrow/frontend:dev` | 8083 | Next.js SSR app |
| `refscanner` | `git.tbs.amn.gg/escrow/scanner:dev` | 8080 | In-house AMN payment scanner (SQLite) |
| `postgres` | `postgres:18-alpine` | 5432 | Primary datastore (auth, marketplace, PG stores) |
| `redis` | `redis:8-alpine` | 6379 | Session cache, rate-limit counters, job queues |
| `mongodb` | `mongo:8.0-noble` | 27017 | Legacy datastore — dev boot parity only, retired in prod |
| `gatus` | `twinproduction/gatus:latest` | 8080 (mapped 8084) | Uptime monitoring + Telegram alerting |

> **Note on refscanner:** The in-house scanner (`provider: "amn.scanner"`) persists state in a SQLite file at `/data/scanner.db` inside the container. It does not expose a port on `shared-web`; the backend calls it via the `default` bridge by container alias `refscanner`.

> **Note on mongodb:** The Mongo container is retained for dev stack parity because `MONGODB_URI` is still present in the env. It will be removed once the backend's remaining Mongo reads are migrated to Postgres. See [[mongo-to-pg-migration-guide]] and [[mongo_retirement_status]].

---

## 3. Architecture Diagram

```
Internet (HTTPS 443 / HTTP 80)
        │
        ▼
┌───────────────────────────┐
│   Cloudflare CDN / Proxy  │
│   amn.gg / dev.amn.gg     │
└─────────────┬─────────────┘
              │
              ▼ (origin)
┌─────────────────────────────────────────────────┐
│  Host: 89.58.32.32                              │
│                                                 │
│  ┌────────────────────────────────────────────┐ │
│  │  infra-caddy  (Arcane project "infra")     │ │
│  │  ports 80:80, 443:443 on host              │ │
│  │  reads Caddyfile at                        │ │
│  │  /opt/arcane/data/projects/infra/Caddyfile │ │
│  └───┬───────────────────────────┬────────────┘ │
│      │ /api/* /socket.io/*       │ /*           │
│      │ /uploads/*                │              │
│      ▼                           ▼              │
│  ┌────────────┐          ┌────────────────┐     │
│  │  backend   │          │   frontend     │     │
│  │ :5001      │          │   :8083        │     │
│  │ shared-web │          │   shared-web   │     │
│  └──┬──┬──┬───┘          └────────────────┘     │
│     │  │  │                                     │
│     │  │  └──────────────────────┐              │
│     │  │                         ▼              │
│     │  │            ┌────────────────────┐      │
│     │  │            │    refscanner      │      │
│     │  │            │    :8080 (default  │      │
│     │  │            │    bridge only)    │      │
│     │  │            └────────────────────┘      │
│     │  │                                        │
│     ▼  ▼                                        │
│  ┌──────────┐  ┌──────────┐  ┌───────────────┐ │
│  │ postgres │  │  redis   │  │   mongodb     │ │
│  │  :5432   │  │  :6379   │  │   :27017      │ │
│  │ (default │  │ (default │  │ (default only,│ │
│  │  only)   │  │  only)   │  │   legacy)     │ │
│  └──────────┘  └──────────┘  └───────────────┘ │
│                                                 │
│  ┌────────────────────────────────────────────┐ │
│  │   gatus :8084 (mapped from :8080)          │ │
│  │   monitors dev.amn.gg + amn.gg + external  │ │
│  └────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────┘

Networks:
  shared-web  ─── external, attached: backend, frontend
  default     ─── internal bridge: all services
```

---

## 4. Networks

| Network | Type | Services Attached | Purpose |
|---|---|---|---|
| `default` (bridge) | Internal | All services | Container-to-container communication |
| `shared-web` | External (pre-existing) | `backend`, `frontend` | Allows infra-caddy to proxy by container name |
| `traefik_public` | External (legacy only) | nginx, gatus (legacy compose) | Old Traefik-based ingress on `git.manko.yoga` host |

**Key rules:**
- `postgres`, `redis`, `mongodb` are on `default` only — no external exposure.
- `refscanner` is on `default` only; backend reaches it via alias `refscanner:8080`.
- Any new public-facing service must join `shared-web` AND get a Caddyfile block. See [[Shared Infra (89.58.32.32)]] and section 6 below.
- `shared-web` must exist on the host before `docker compose up`. It is created by the `infra` project.

---

## 5. Volumes and Bind Mounts

All data volumes in the `dev-amn` stack use relative bind mounts under `./data/` (resolved to `/opt/arcane/data/projects/escrow-dev/data/` on the server):

| Service | Host Path | Container Path | Notes |
|---|---|---|---|
| `backend` | `./data/uploads` | `/app/uploads` | User-uploaded files |
| `refscanner` | `./data/scanner` | `/data` | SQLite DB at `/data/scanner.db` |
| `postgres` | `./data/postgres` | `/var/lib/postgresql/data` | `PGDATA` subdir workaround: actual data at `./data/postgres/pgdata` |
| `redis` | `./data/redis` | `/data` | Persistence dump |
| `mongodb` | `./data/mongo` | `/data/db` | Legacy; can be deleted once Mongo retired |
| `gatus` | `./gatus/config.yaml` | `/config/config.yaml` (ro) | Monitoring config — part of repo |

**Postgres volume note:** `postgres:18` introduced a version-scoped data directory layout and refuses to init directly into a volume root that already contains files from a different layout. The compose file sets `PGDATA=/var/lib/postgresql/data/pgdata` to place actual data in a subdirectory of the mount, avoiding init conflicts.

**Legacy compose** (`deployment/docker-compose.yml`) uses absolute host paths under `/var/data/escrowDev/` and does not share volumes with the dev-amn stack.

---

## 6. Reverse Proxy (infra-caddy) Integration

Ingress for `89.58.32.32` is handled exclusively by **infra-caddy** — the Caddy container in the Arcane project `infra`. It owns host ports 80 and 443. No service should bind those ports directly.

### Current Caddyfile block (dev.amn.gg)

Located at `/opt/arcane/data/projects/infra/Caddyfile` on the server (and mirrored in `deployment/dev-amn/Caddyfile` for reference):

```
{
    email manwe@manko.yoga
    auto_https disable_redirects
}

dev.amn.gg {
    encode zstd gzip

    @backend path /api/* /socket.io/* /uploads/*
    reverse_proxy @backend backend:5001

    reverse_proxy frontend:8083
}
```

- `auto_https disable_redirects` — Cloudflare proxy sits in front; Caddy should not force HTTP→HTTPS redirects at origin.
- Routes by path prefix: `/api/*`, `/socket.io/*`, `/uploads/*` go to `backend:5001`; everything else to `frontend:8083`.
- Container names are resolved via the `shared-web` network.

### Adding a new public service

1. Add the service to `deployment/dev-amn/docker-compose.yml` with `networks: shared-web: {}`.
2. Edit `/opt/arcane/data/projects/infra/Caddyfile` on the server — add a new vhost block or path matcher.
3. Reload Caddy (no restart needed):
   ```bash
   docker exec infra-caddy caddy reload --config /etc/caddy/Caddyfile
   ```
4. Verify via `curl -I https://dev.amn.gg/<new-path>`.

---

## 7. Gatus Monitoring

Gatus runs as a sidecar in the `default` network. Config is at `deployment/gatus/config.yaml` (bind-mounted read-only). Alerts are delivered via Telegram.

### Alert policy

| Setting | Value |
|---|---|
| Default failure threshold | 3 consecutive failures |
| Default success threshold | 2 consecutive successes |
| Send on resolved | Yes |
| Alert channel | Telegram (`GATUS_TELEGRAM_BOT_TOKEN` / `GATUS_TELEGRAM_CHAT_ID`) |

### Monitored endpoints

| Name | Group | URL | Interval | Key Conditions |
|---|---|---|---|---|
| `backend-dev-version` | backend-dev | `https://dev.amn.gg/api/version` | 60s | HTTP 200, body.version not empty |
| `backend-dev-health` | backend-dev | `https://dev.amn.gg/api/health` | 30s | HTTP 200, all PG store modes = postgres, redis ok, RN chain+token registry loaded |
| `backend-prod-version` | backend-prod | `https://amn.gg/api/version` | 60s | HTTP 200, body.version not empty (failure-threshold 2) |
| `backend-prod-health` | backend-prod | `https://amn.gg/api/health` | 30s | HTTP 200, db/postgres/redis/RN registries ok (failure-threshold 2) |
| `frontend-dev` | frontend | `https://dev.amn.gg/` | 60s | HTTP 200, response < 3000ms |
| `frontend-prod` | frontend | `https://amn.gg/` | 60s | HTTP 200, response < 3000ms (failure-threshold 2) |
| `rn-api-reachable` | external | `https://api.request.network/v2/health` | 5m | HTTP 200/401/404 (accepts auth errors — just checks reachability) |
| `chainalysis-public-api` | external | `https://public.chainalysis.com/api/v1/address/0x000…` | 5m | HTTP 200 or 404 |
| `bsc-rpc-publicnode` | external | `https://bsc-rpc.publicnode.com` (POST) | 2m | HTTP 200, `result == "0x38"` (BSC mainnet chain ID) |

The `backend-dev-health` check validates that **all 8 domain stores are running on Postgres** (`auth`, `config`, `address`, `category`, `levelConfig`, `shopSettings`, `review`, `notification`). A failure here means a store mode regression or a broken `PG_URL`.

Gatus dashboard is accessible at `:8084` on the host (not publicly proxied by default — access via SSH tunnel or add a Caddyfile block if needed).

---

## 8. Environment Variables

All vars are passed to containers via `.env` at the stack root (`deployment/dev-amn/.env` on the server, `deployment/.env` in the repo as the live dev reference). The file is `chmod 600` and never committed.

### Backend

| Variable | Description | Example / Default |
|---|---|---|
| `NODE_ENV` | Runtime environment | `production` |
| `PORT` | Express listen port | `5001` |
| `TRUST_PROXY` | Express trust-proxy (required behind Caddy) | `true` |
| `DEBUG` | Debug namespaces | _(empty)_ |
| `LOG_LEVEL` | Winston log level | `info` |

#### Database

| Variable | Description | Example |
|---|---|---|
| `MONGODB_URI` | Mongo connection string | `mongodb://admin:pass@mongodb:27017/marketplace?authSource=admin` |
| `MONGO_INITDB_ROOT_USERNAME` | Mongo root user | `admin` |
| `MONGO_INITDB_ROOT_PASSWORD` | Mongo root password | — |
| `MONGO_INITDB_DATABASE` | Mongo init database | `marketplace` |
| `DB_NAME` | Mongo database name used by app | `amn-db` |
| `PG_URL` | Postgres DSN | `postgres://amanat:pass@amanat-postgres:5432/amanat_dev` |
| `POSTGRES_USER` | Postgres superuser | `amanat` |
| `POSTGRES_PASSWORD` | Postgres superuser password | — |
| `POSTGRES_DB` | Postgres database name | `amanat_dev` |
| `AUTO_SEED_ON_START` | Run seed on boot | `true` |

#### Store modes (dual-write seam)

| Variable | Description | Default |
|---|---|---|
| `AUTH_STORE` | Auth domain store backend | `postgres` |
| `CONFIG_STORE` | Config domain | `postgres` |
| `ADDRESS_STORE` | Address domain | `postgres` |
| `CATEGORY_STORE` | Category domain | `postgres` |
| `LEVEL_CONFIG_STORE` | Level config domain | `postgres` |
| `SHOP_SETTINGS_STORE` | Shop settings domain | `postgres` |
| `REVIEW_STORE` | Review domain | `postgres` |
| `NOTIFICATION_STORE` | Notification domain | `postgres` |

#### Auth / Sessions

| Variable | Description |
|---|---|
| `JWT_SECRET` | JWT signing secret |
| `JWT_EXPIRES_IN` | Access token TTL (e.g. `7d`) |
| `REFRESH_TOKEN_EXPIRES_IN` | Refresh token TTL (e.g. `30d`) |

#### Redis

| Variable | Description |
|---|---|
| `REDIS_URI` | Redis connection string (includes password) |
| `REDIS_PASSWORD` | Redis auth password (standalone, if not in URI) |

#### URLs / CORS

| Variable | Description |
|---|---|
| `BASE_URL` | Canonical origin (`https://dev.amn.gg`) |
| `API_URL` | API base URL |
| `FRONTEND_URL` | Frontend origin |
| `BACKEND_URL` | Backend origin |
| `CORS_ORIGIN` | Allowed CORS origin |

#### File uploads

| Variable | Description | Default |
|---|---|---|
| `UPLOAD_PATH` | Upload directory inside container | `/app/uploads` |
| `MAX_FILE_SIZE` | Max upload bytes | `52428800` (50 MB) |

#### Rate limiting

| Variable | Description | Default |
|---|---|---|
| `RATE_LIMIT_WINDOW_MS` | Window for rate limiter | `900000` (15 min) |
| `RATE_LIMIT_MAX_REQUESTS` | Max requests per window | `100` |

> GET `/api/payment/:id` must bypass `paymentLimiter` — see [[backend_rate_limits]].

#### SMTP

| Variable | Description |
|---|---|
| `SMTP_HOST` | SMTP server hostname |
| `SMTP_PORT` | SMTP port |
| `SMTP_SECURE` | TLS (`true`/`false`) |
| `SMTP_USER` | SMTP username |
| `SMTP_PASS` | SMTP password |
| `SMTP_FROM` | From address |

#### WebAuthn (Passkeys)

| Variable | Description |
|---|---|
| `WEBAUTHN_RP_ID` | Relying party ID (domain) |
| `WEBAUTHN_RP_NAME` | Relying party display name |
| `WEBAUTHN_RP_ORIGIN` | Relying party origin URL |

#### Admin seed

| Variable | Description |
|---|---|
| `ADMIN_EMAIL` | Bootstrap admin email |
| `ADMIN_PASSWORD` | Bootstrap admin password |
| `ADMIN_FIRST_NAME` | Admin first name |
| `ADMIN_LAST_NAME` | Admin last name |

#### Google OAuth

| Variable | Description |
|---|---|
| `GOOGLE_CLIENT_ID` | Google OAuth client ID |
| `GOOGLE_CLIENT_SECRET` | Google OAuth client secret |

#### OpenAI

| Variable | Description |
|---|---|
| `OPENAI_API_KEY` | OpenAI API key |
| `OPENAI_DEFAULT_MODEL` | Default model (e.g. `gpt-4`) |
| `OPENAI_MAX_TOKENS` | Max tokens per request |
| `OPENAI_TEMPERATURE` | Sampling temperature |

#### Sentry

| Variable | Description |
|---|---|
| `SENTRY_DSN` | Sentry ingest DSN |

#### Wallets / Blockchain

| Variable | Description |
|---|---|
| `ESCROW_WALLET_ADDRESS` | Platform escrow wallet |
| `BSC_USDT_CONTRACT` | BSC USDT token contract address |
| `ADMIN_PAYOUT_WALLET_ADDRESS` | Admin payout destination |
| `RECEIVER_WALLET_ADDRESS` | Default receiver wallet |

#### DePay

| Variable | Description |
|---|---|
| `DEPAY_INTEGRATION_ID` | DePay integration UUID |
| `DEPAY_WEBHOOK_SECRET` | Webhook verification secret |
| `DEPAY_NETWORKS` | Enabled chains (e.g. `bsc`) |
| `DEPAY_ALLOWED_TOKENS` | Allowed payment tokens |
| `DEPAY_PUBLIC_KEY` | DePay public key (PEM) |

#### SHKeeper

| Variable | Description |
|---|---|
| `SHKEEPER_API_KEY` | SHKeeper API key |
| `SHKEEPER_BASE_URL` | SHKeeper service base URL |
| `SHKEEPER_API_URL` | Payment request endpoint |
| `SHKEEPER_ENVIRONMENT` | `production` or `sandbox` |
| `SHKEEPER_WALLET_ID` | Destination wallet |
| `SHKEEPER_NETWORKS` | Enabled chains |
| `SHKEEPER_ALLOWED_TOKENS` | Allowed tokens |
| `SHKEEPER_FORCE_REAL` | Bypass test mode |
| `SHKEEPER_TOKEN` | Token type (e.g. `USDT`) |
| `SHKEEPER_CALLBACK_SECRET` | Callback verification secret |
| `SHKEEPER_WEBHOOK_SECRET` | Webhook verification secret |

#### Request Network

| Variable | Description |
|---|---|
| `REQUEST_NETWORK_ENABLED` | Enable RN provider |
| `REQUEST_NETWORK_WEBHOOK_SECRET` | Webhook signature secret |
| `REQUEST_NETWORK_WEBHOOK_CALLBACK_URL` | Public callback URL |
| `REQUEST_NETWORK_PAYMENT_CURRENCY` | Currency (e.g. `USDC`) |
| `REQUEST_NETWORK_NETWORK` | Chain (e.g. `bsc`) |
| `REQUEST_NETWORK_MERCHANT_REFERENCE` | Merchant address reference |
| `REQUEST_NETWORK_API_BASE_URL` | RN API root |
| `REQUEST_NETWORK_API_KEY` | RN API key |
| `REQUEST_NETWORK_ORIGIN` | Origin header sent to RN |
| `REQUEST_NETWORK_ALLOW_TEST_WEBHOOKS` | Allow test events (default `false`) |

> RN webhook discriminator is `payload.event` (not `eventType`) — see [[rn_webhook_event_field]].

#### Transaction safety

| Variable | Description | Default |
|---|---|---|
| `TRANSACTION_SAFETY_ENABLED` | Enable on-chain verification | `true` |
| `TRANSACTION_SAFETY_REQUIRE_TX_HASH` | Require tx hash | `true` |
| `TRANSACTION_SAFETY_REQUIRE_TRANSFER_MATCH` | Require transfer match | `true` |
| `TRANSACTION_SAFETY_MIN_CONFIRMATIONS` | Min block confirmations | `12` |
| `TRANSACTION_SAFETY_AML_PROVIDER` | AML provider (`none`, `chainalysis`) | `none` |

#### Payment routing

| Variable | Description |
|---|---|
| `PAYMENT_PROVIDER` | Active provider |
| `PAYMENT_ENABLED_PROVIDERS` | Comma-separated enabled providers |
| `PAYMENT_PROVIDER_MODE` | `live` or `test` |
| `PAYMENT_ROLLBACK_PROVIDER` | Fallback provider |

#### Telegram

| Variable | Description |
|---|---|
| `TELEGRAM_FEATURE_ENABLED` | Enable Telegram integration |
| `TELEGRAM_MINIAPP_ENABLED` | Enable Mini App |
| `TELEGRAM_WEBHOOK_ENABLED` | Enable webhook receiver |
| `TELEGRAM_BOT_TOKEN` | Main bot token |
| `TELEGRAM_WEBHOOK_SECRET_TOKEN` | Webhook secret for validation |
| `TELEGRAM_INITDATA_MAX_AGE_SEC` | Max age for initData |
| `TELEGRAM_INITDATA_REPLAY_WINDOW_MS` | Replay protection window |
| `TELEGRAM_WEBHOOK_REPLAY_WINDOW_MS` | Webhook replay protection window |
| `TELEGRAM_SESSION_TTL_SEC` | Session TTL |
| `TG_NOTIFY_BOT_TOKEN` | Ops/monitoring bot token (amnGG_MonitorBot) |
| `TG_NOTIFY_CHATS` | Comma-separated chat IDs for ops notifications |

#### Pangolin / Newt (VPN mesh — optional)

| Variable | Description |
|---|---|
| `PANGOLIN_ENDPOINT` | Pangolin tunnel endpoint |
| `NEWT_ID` | Newt node ID |
| `NEWT_SECRET` | Newt node secret |

#### Testnet chains

| Variable | Description |
|---|---|
| `ENABLE_TESTNET_CHAINS` | Expose testnet chain configs | Set to `true` in dev-amn compose override |

### Frontend (NEXT_PUBLIC_*)

| Variable | Description |
|---|---|
| `NEXT_PUBLIC_API_URL` | Backend API URL (browser-visible) |
| `NEXT_PUBLIC_SOCKET_URL` | Socket.IO server URL |
| `NEXT_PUBLIC_WALLETCONNECT_PROJECT_ID` | WalletConnect project ID |
| `NEXT_PUBLIC_ALCHEMY_API_KEY_MAINNET` | Alchemy mainnet key |
| `NEXT_PUBLIC_ALCHEMY_API_KEY_SEPOLIA` | Alchemy Sepolia key |
| `NEXT_PUBLIC_ALCHEMY_API_KEY_POLYGON` | Alchemy Polygon key |
| `NEXT_PUBLIC_ESCROW_WALLET_ADDRESS` | Escrow wallet (shown in UI) |
| `NEXT_PUBLIC_APP_NAME` | App display name |
| `NEXT_PUBLIC_APP_VERSION` | App version string |
| `NEXT_PUBLIC_MAPBOX_API_KEY` | Mapbox key (address autocomplete) |
| `NEXT_PUBLIC_PASSKEY_RP_NAME` | WebAuthn RP name |
| `NEXT_PUBLIC_PASSKEY_RP_ID` | WebAuthn RP ID |
| `NEXT_PUBLIC_PASSKEY_ORIGIN` | WebAuthn origin |
| `NEXT_PUBLIC_BACKEND_URL` | Backend origin (used for direct calls) |
| `NEXT_PUBLIC_DEPAY_INTEGRATION_ID` | DePay integration ID |
| `NEXT_PUBLIC_IS_DEVELOPMENT` | Development flag |
| `NEXT_PUBLIC_ENABLE_DEBUG` | Enable client debug logging |
| `NEXT_PUBLIC_APP_URL` | Canonical app URL |
| `NEXT_PUBLIC_TELEGRAM_BOT_ID` | Telegram bot numeric ID |
| `BUILD_STATIC_EXPORT` | Enable `next export` mode (`false` for SSR) |

### Gatus

| Variable | Description |
|---|---|
| `GATUS_TELEGRAM_BOT_TOKEN` | Telegram bot for alert delivery |
| `GATUS_TELEGRAM_CHAT_ID` | Target chat ID for alerts |

---

## 9. Deploy Workflow

### 9.1 Normal image update (CI-driven)

Woodpecker CI builds `backend` and `frontend` images, pushes tags to `git.tbs.amn.gg/escrow/` on merge to `dev`, then triggers an Arcane GitOps sync which pulls the new image and recreates the container.

```
git push origin dev
  └─► Woodpecker build pipeline
        └─► docker push git.tbs.amn.gg/escrow/backend:dev
        └─► docker push git.tbs.amn.gg/escrow/frontend:dev
        └─► arcane-cli gitops sync cf6c9eab…  (or watchtower polls)
              └─► escrow-backend container restarted with new image
              └─► escrow-frontend container restarted with new image
```

> Always bump the version in `package.json` + lock before pushing, otherwise the CI build may not register as a new deploy. See [[version_bump_before_ci]].

### 9.2 Manual deploy (backend hotfix — no registry)

For urgent backend fixes without a full CI cycle, use the local-build pattern (the dev stack has `pull_policy: always` but the override `docker-compose.override.yml` sets `pull_policy: never` for the `escrow-backend-local:dev` image path):

```bash
# 1. Copy changed files to build tree on server
scp -i ~/CascadeProjects/wzp src/services/auth/authRoutes.ts \
    root@89.58.32.32:/tmp/escrow-backend-build/src/services/auth/

# 2. Rebuild image on server (~3 min, ARM64)
ssh -i ~/CascadeProjects/wzp root@89.58.32.32 \
  "cd /tmp/escrow-backend-build && docker build -f Dockerfile.prod -t escrow-backend-local:dev ."

# 3. Restart the backend container
ssh -i ~/CascadeProjects/wzp root@89.58.32.32 \
  "cd /opt/arcane/data/projects/escrow-dev && docker compose up -d backend"
```

### 9.3 Bringing the stack up/down

```bash
# via Arcane CLI (preferred)
arcane-cli project start devEscrow
arcane-cli project stop devEscrow

# via SSH + docker compose (direct)
ssh -i ~/CascadeProjects/wzp root@89.58.32.32 \
  "cd /opt/arcane/data/projects/escrow-dev && docker compose up -d"

ssh -i ~/CascadeProjects/wzp root@89.58.32.32 \
  "cd /opt/arcane/data/projects/escrow-dev && docker compose down"
```

### 9.4 Reloading Caddy after Caddyfile edits

Edit `/opt/arcane/data/projects/infra/Caddyfile` on the server, then:

```bash
ssh -i ~/CascadeProjects/wzp root@89.58.32.32 \
  "docker exec infra-caddy caddy reload --config /etc/caddy/Caddyfile"
```

No container restart needed.

### 9.5 Updating env vars

1. Edit `.env` on the server: `/opt/arcane/data/projects/escrow-dev/.env`
2. Restart affected service:
   ```bash
   ssh -i ~/CascadeProjects/wzp root@89.58.32.32 \
     "cd /opt/arcane/data/projects/escrow-dev && docker compose up -d backend"
   ```
   Frontend env vars baked at build time (via `NEXT_PUBLIC_*`) require a fresh image rebuild.

### 9.6 Verifying a deploy

```bash
# Check running containers
arcane-cli project status devEscrow

# Check backend version
curl https://dev.amn.gg/api/version

# Check health (all stores + registries)
curl https://dev.amn.gg/api/health | jq .

# Tail backend logs
ssh -i ~/CascadeProjects/wzp root@89.58.32.32 \
  "docker logs -f escrow-backend --tail 100"
```

> CI ✓ green does NOT guarantee the new image was pushed to the registry. Always verify `curl /api/version` returns the expected version. See [[woodpecker_silent_build_fail]].

---

## 10. Dev vs Prod Differences

| Aspect | dev-amn (dev.amn.gg) | Prod (amn.gg) |
|---|---|---|
| Compose file | `deployment/dev-amn/docker-compose.yml` | Separate prod stack (not in this repo) |
| Image registry | `git.tbs.amn.gg/escrow` | Same registry, prod tags |
| Image tag | `:dev` | `:latest` or versioned |
| MongoDB | Present (dev parity) | Retired |
| `ENABLE_TESTNET_CHAINS` | `true` (compose override) | Not set / `false` |
| `NODE_ENV` | `production` (same) | `production` |
| `NEXT_PUBLIC_IS_DEVELOPMENT` | `false` | `false` |
| `PAYMENT_PROVIDER_MODE` | `live` | `live` |
| `REQUEST_NETWORK_ALLOW_TEST_WEBHOOKS` | can be `true` for RN testing | `false` |
| Watchtower labels | Present in legacy compose | Prod stack may differ |
| Gatus monitoring | Monitors both dev + prod endpoints | N/A (shared gatus instance) |
| TLS | Cloudflare proxy → Caddy (disable_redirects) | Same |
| Version bump requirement | Required before CI push | Required |

---

## 11. Secret Management

**The `.env` file on the server is the single source of runtime secrets. It is never committed.**

- Location on server: `/opt/arcane/data/projects/escrow-dev/.env`
- Permissions: `chmod 600` owned by root
- Reference template: `deployment/.env` (in repo — contains live dev values, treated as low-sensitivity dev config; rotate before prod use)
- `.gitleaks.toml` in `deployment/` configures secret scanning exclusions for the repo

### Rules

1. Never commit `.env` or any file containing real tokens, passwords, or private keys.
2. Never pass secrets as Dockerfile `ARG`/`ENV` at build time — they appear in image layers. All secrets are runtime-injected via `env_file`.
3. `NEXT_PUBLIC_*` vars are baked into the frontend bundle at build time. Do not place secrets in any `NEXT_PUBLIC_` variable.
4. Wallet addresses (e.g. `ESCROW_WALLET_ADDRESS`) are public on-chain but still kept out of the repo for operational hygiene.
5. For new deployments: copy `deployment/.env` to the server, fill in real values, then `chmod 600`.
6. Gatus bot token and chat ID go into the same `.env` — they are read by the gatus container via `environment:` directives.
7. Telegram bot tokens are high-value secrets — rotate immediately if accidentally pushed.

### Sensitive variable groups

| Group | Variables | Risk if leaked |
|---|---|---|
| JWT | `JWT_SECRET` | Full session forgery |
| DB credentials | `POSTGRES_PASSWORD`, `REDIS_PASSWORD`, `MONGO_INITDB_ROOT_PASSWORD` | Database access |
| Payment webhook secrets | `REQUEST_NETWORK_WEBHOOK_SECRET`, `DEPAY_WEBHOOK_SECRET`, `SHKEEPER_CALLBACK_SECRET`, `SHKEEPER_WEBHOOK_SECRET` | Fake payment injection |
| Bot tokens | `TELEGRAM_BOT_TOKEN`, `TG_NOTIFY_BOT_TOKEN` | Bot takeover |
| OAuth secrets | `GOOGLE_CLIENT_SECRET` | OAuth impersonation |
| API keys | `OPENAI_API_KEY`, `REQUEST_NETWORK_API_KEY`, `SHKEEPER_API_KEY` | Billing / data access |
| Sentry DSN | `SENTRY_DSN` | Error data exfiltration |