--- title: Deployment tags: [operations] --- # Deployment How the production stack runs and gets updated on the live host. The stack is fully containerised and self-updates via Watchtower from the Gitea container registry. --- ## 1. Topology ``` ┌─────────────────────────┐ HTTPS 443 ──────────►│ External SSL term. │ │ (DNS amn.gg, dev.amn.gg)│ └────────────┬────────────┘ │ HTTP 80 (in-VPC) ▼ ┌──────────────────────────────────┐ │ Nginx container │ │ (nickapp-nginx, port 80) │ └─┬───────────────────┬────────────┘ │ │ │ / │ /api /socket.io ▼ ▼ ┌─────────────────────┐ ┌──────────────────────────┐ │ nickapp-frontend │ │ nickapp-backend │ │ Next.js, port 8083 │ │ Express 5, port 5001 │ └─────────────────────┘ └──────┬────────────┬──────┘ │ │ ▼ ▼ ┌──────────┐ ┌──────────┐ │ mongodb │ │ redis │ │ 8.2 │ │ 8 │ └──────────┘ └──────────┘ ┌──────────────────────────────────┐ │ Watchtower │ │ Polls registry → restarts │ │ containers labelled enable=true │ └──────────────────────────────────┘ ``` All containers run on the **`default`** Docker network defined by `docker-compose.production.yml`. Watchtower runs as a sidecar container on the same host. DNS resolves both `amn.gg` and `dev.amn.gg` to the production host's public IP. SSL termination happens **outside** the compose stack (typically via the hosting provider's edge or a host-level reverse proxy), and traffic is forwarded as HTTP to the `nginx` container on port `80` (mapped to host `8083`). --- ## 2. Compose file `backend/docker-compose.production.yml` is the single source of truth. Services: | Service | Image | Ports | Volumes | Notes | |---------|-------|-------|---------|-------| | `nginx` | `nginx:alpine` | `8083:80` | `./nginx/nginx.conf`, `./nginx/logs`, `./uploads` (served as `/uploads`) | Reverse proxy | | `nickapp-backend` | `nickapp-backend:latest` (build from `Dockerfile.prod`) | not exposed externally | `./uploads:/app/uploads` | Labelled for Watchtower | | `nickapp-frontend` | `nickapp-frontend:latest` (build from `../frontend/Dockerfile`) | `expose: 8083` | — | Labelled for Watchtower | | `mongodb` | `mongo:8.2` | not exposed | `mongodb_data:/data/db`, `./mongo-init:/docker-entrypoint-initdb.d` | Healthcheck via `mongosh ping` | | `redis` | `redis:8-alpine` | not exposed | `redis_data:/data` | Started with `--requirepass "$REDIS_PASSWORD"` | Healthchecks are configured for backend (`curl /health`), frontend (`curl /`), Mongo (`mongosh ping`), and Redis (`redis-cli -a $REDIS_PASSWORD ping`). See [[Monitoring]]. Watchtower polls images labelled `com.centurylinklabs.watchtower.enable=true` — currently `nickapp-backend` and `nickapp-frontend`. MongoDB and Redis are **not** auto-updated. --- ## 3. Registry & images | Image | Registry path | |-------|---------------| | Backend prod | `git.manko.yoga/manawenuz/escrow-backend:latest` | | Backend dev | `git.manko.yoga/manawenuz/escrow-backend:dev` | | Backend tagged | `git.manko.yoga/manawenuz/escrow-backend:` | | Frontend | `git.manko.yoga/manawenuz/escrow-frontend:latest` and `:` | `docker-compose.production.yml` currently builds locally on first up (`build: context: .`). Once images are in the registry the file can be switched to `image: git.manko.yoga/manawenuz/escrow-backend:latest` to let Watchtower pull straight from there. > [!tip] To pin a specific version while debugging, edit the compose file to `image: git.manko.yoga/manawenuz/escrow-backend:2.6.3` and re-run `docker compose up -d`. Remove the Watchtower label or the agent will undo it on next poll. --- ## 4. Watchtower Watchtower runs as its own container (managed outside the compose file) with `WATCHTOWER_LABEL_ENABLE=true` so it only touches services that opt in. On each poll cycle (default 5 minutes, configurable via `WATCHTOWER_POLL_INTERVAL`) it: 1. Pulls the latest digest for each enabled service's image. 2. Compares to the running container's digest. 3. If different, stops the container, removes it, and starts a new one from the new image, preserving all named volumes. Configuration knobs typically set on the host: ```bash docker run -d --name watchtower \ --restart unless-stopped \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /root/.docker/config.json:/config.json \ # so it can pull from the private Gitea registry -e WATCHTOWER_POLL_INTERVAL=300 \ -e WATCHTOWER_LABEL_ENABLE=true \ -e WATCHTOWER_CLEANUP=true \ -e WATCHTOWER_INCLUDE_RESTARTING=true \ containrrr/watchtower ``` The `~/.docker/config.json` must have a valid login for `git.manko.yoga` (created via `docker login git.manko.yoga -u manawenuz`). --- ## 5. First-time deploy (cold start) > [!warning] Run these steps on a fresh production host. They are destructive on an existing one. See [[Backup & Recovery]] before touching live data. ### Prerequisites on the host - Ubuntu 22.04+ (or any systemd Linux), Docker Engine 24+, `docker compose` plugin - `git` installed - DNS `amn.gg` + `dev.amn.gg` already pointing here - An SSL terminator (Caddy / Nginx / Cloudflare) reverse-proxying to host port `8083` - Registry login: `docker login git.manko.yoga -u manawenuz` ### Steps ```bash # 1. Clone both repos as siblings (compose references ../frontend) cd /opt git clone ssh://git@git.manko.yoga:222/nick/backend.git git clone ssh://git@git.manko.yoga:222/nick/frontend.git cd backend git checkout main # 2. Create the production .env sudo nano .env # fill from Environment Variables doc; production values, real secrets # 3. Provision the nginx config + uploads dir mkdir -p nginx/logs uploads mongo-init sudo cp /path/to/nginx.conf nginx/nginx.conf # (the nginx.conf forwards /api/* and /socket.io/* to nickapp-backend:5001, # forwards /uploads/* to /uploads (volume), and everything else to nickapp-frontend:8083) # 4. Build & start the stack docker compose -f docker-compose.production.yml up --build -d # 5. Verify docker compose -f docker-compose.production.yml ps docker compose -f docker-compose.production.yml logs -f --tail=200 curl -fsS http://localhost:8083/api/health | jq . # 6. Seed initial data (optional — if AUTO_SEED_ON_START=true is set, it's already done) docker compose -f docker-compose.production.yml exec nickapp-backend node dist/scripts/seedCategories.js # 7. Start Watchtower (one-time) docker run -d --name watchtower --restart unless-stopped \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /root/.docker/config.json:/config.json \ -e WATCHTOWER_POLL_INTERVAL=300 \ -e WATCHTOWER_LABEL_ENABLE=true \ -e WATCHTOWER_CLEANUP=true \ containrrr/watchtower ``` ### SSL / TLS Termination happens at the edge — outside the compose stack. The two common setups: - **Caddy on the host** forwarding `amn.gg` and `dev.amn.gg` to `127.0.0.1:8083`. Caddy handles Let's Encrypt automatically. - **Cloudflare Full (strict)** in front of the host. Use Cloudflare Origin certificates on the host's Caddy/Nginx. Either way, the compose stack itself sees only HTTP on port 80 inside the nginx container. The `nginx.conf` should set `proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto` and the backend already trusts the proxy when `NODE_ENV=production` (see `trust proxy` block in `src/app.ts`). --- ## 6. Routine deploy (after first deploy) The normal flow is **fully automatic**: 1. Developer merges PR to `main` (see [[Git Workflow]]). 2. Gitea Actions runs `.gitea/workflows/docker-build-no-cache.yml` (backend) or `deploy.yml` (frontend). The workflow builds the production image and pushes `:latest` + `:` to the registry. See [[CI-CD Pipeline]]. 3. Watchtower polls the registry, sees a new digest, restarts the container. 4. Healthcheck on the new container passes after `start_period=40s`, traffic resumes. Total time from merge to live: **5–10 minutes** depending on Watchtower poll interval and image size. ### Force an immediate deploy If you don't want to wait for the poll: ```bash # On the production host: cd /opt/backend docker login git.manko.yoga -u manawenuz # if creds expired docker compose -f docker-compose.production.yml pull nickapp-backend nickapp-frontend docker compose -f docker-compose.production.yml up -d nickapp-backend nickapp-frontend ``` The `up -d` will detect changed images and restart only the affected containers. ### Roll back ```bash # Find available versions docker images git.manko.yoga/manawenuz/escrow-backend # Pin to the previous tag in the compose file sed -i 's|escrow-backend:latest|escrow-backend:2.6.2|' docker-compose.production.yml # Re-up docker compose -f docker-compose.production.yml up -d nickapp-backend # Disable Watchtower for the affected service until you're ready to resume docker compose ... restart # no-op if you removed the watchtower label ``` > [!warning] Watchtower will undo a pin to a non-`latest` tag on its next poll if the container still has the `watchtower.enable=true` label. Either remove the label temporarily or pause Watchtower (`docker stop watchtower`). --- ## 7. Logs ```bash # All services docker compose -f docker-compose.production.yml logs -f --tail=300 # Single service docker compose -f docker-compose.production.yml logs -f nickapp-backend # Nginx access log tail -f /opt/backend/nginx/logs/access.log ``` Backend logs are also captured by Sentry breadcrumbs when an error occurs — see [[Monitoring]]. --- ## 8. Maintenance window Plan a 5-minute window when bumping major versions or running migrations: ```bash # Announce + drain # (set a maintenance banner in the frontend if possible) # Take a backup first ./scripts/backup-mongo.sh # or per Backup & Recovery # Pull new images, restart docker compose -f docker-compose.production.yml pull docker compose -f docker-compose.production.yml up -d # Verify curl -fsS https://amn.gg/api/health ``` If anything goes sideways, follow [[Incident Response]].