nick-doc/08 - Operations/Deployment.md

title, tags

title	tags
Deployment	operations

Deployment

How the production stack runs and gets updated on the live host. The stack is fully containerised and self-updates via Watchtower from the Gitea container registry.

---

1. Topology

                                  ┌─────────────────────────┐
            HTTPS  443 ──────────►│   External SSL term.    │
                                  │ (DNS amn.gg, dev.amn.gg)│
                                  └────────────┬────────────┘
                                               │ HTTP 80 (in-VPC)
                                               ▼
                            ┌──────────────────────────────────┐
                            │           Nginx container        │
                            │       (nickapp-nginx, port 80)   │
                            └─┬───────────────────┬────────────┘
                              │                   │
                              │ /                 │ /api  /socket.io
                              ▼                   ▼
                ┌─────────────────────┐  ┌──────────────────────────┐
                │ nickapp-frontend    │  │ nickapp-backend          │
                │ Next.js, port 8083  │  │ Express 5, port 5001     │
                └─────────────────────┘  └──────┬────────────┬──────┘
                                                │            │
                                                ▼            ▼
                                         ┌──────────┐  ┌──────────┐
                                         │ mongodb  │  │  redis   │
                                         │  8.2     │  │  8       │
                                         └──────────┘  └──────────┘

                            ┌──────────────────────────────────┐
                            │            Watchtower            │
                            │ Polls registry → restarts        │
                            │ containers labelled enable=true  │
                            └──────────────────────────────────┘

All containers run on the default Docker network defined by docker-compose.production.yml. Watchtower runs as a sidecar container on the same host.

DNS resolves both amn.gg and dev.amn.gg to the production host's public IP. SSL termination happens outside the compose stack (typically via the hosting provider's edge or a host-level reverse proxy), and traffic is forwarded as HTTP to the nginx container on port 80 (mapped to host 8083).

---

2. Compose file

backend/docker-compose.production.yml is the single source of truth. Services:

Service	Image	Ports	Volumes	Notes
nginx	nginx:alpine	8083:80	./nginx/nginx.conf, ./nginx/logs, ./uploads (served as /uploads)	Reverse proxy
nickapp-backend	nickapp-backend:latest (build from Dockerfile.prod)	not exposed externally	./uploads:/app/uploads	Labelled for Watchtower
nickapp-frontend	nickapp-frontend:latest (build from ../frontend/Dockerfile)	expose: 8083	—	Labelled for Watchtower
mongodb	mongo:8.2	not exposed	mongodb_data:/data/db, ./mongo-init:/docker-entrypoint-initdb.d	Healthcheck via mongosh ping
redis	redis:8-alpine	not exposed	redis_data:/data	Started with --requirepass "$REDIS_PASSWORD"

Healthchecks are configured for backend (curl /health), frontend (curl /), Mongo (mongosh ping), and Redis (redis-cli -a $REDIS_PASSWORD ping). See Monitoring.

Watchtower polls images labelled com.centurylinklabs.watchtower.enable=true — currently nickapp-backend and nickapp-frontend. MongoDB and Redis are not auto-updated.

---

3. Registry & images

Image	Registry path
Backend prod	git.manko.yoga/manawenuz/escrow-backend:latest
Backend dev	git.manko.yoga/manawenuz/escrow-backend:dev
Backend tagged	git.manko.yoga/manawenuz/escrow-backend:<package-version>
Frontend	git.manko.yoga/manawenuz/escrow-frontend:latest and :<version>

docker-compose.production.yml currently builds locally on first up (build: context: .). Once images are in the registry the file can be switched to image: git.manko.yoga/manawenuz/escrow-backend:latest to let Watchtower pull straight from there.

[!tip] To pin a specific version while debugging, edit the compose file to image: git.manko.yoga/manawenuz/escrow-backend:2.6.3 and re-run docker compose up -d. Remove the Watchtower label or the agent will undo it on next poll.

---

4. Watchtower

Watchtower runs as its own container (managed outside the compose file) with WATCHTOWER_LABEL_ENABLE=true so it only touches services that opt in. On each poll cycle (default 5 minutes, configurable via WATCHTOWER_POLL_INTERVAL) it:

1. Pulls the latest digest for each enabled service's image.
2. Compares to the running container's digest.
3. If different, stops the container, removes it, and starts a new one from the new image, preserving all named volumes.

Configuration knobs typically set on the host:

docker run -d --name watchtower \
  --restart unless-stopped \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /root/.docker/config.json:/config.json \   # so it can pull from the private Gitea registry
  -e WATCHTOWER_POLL_INTERVAL=300 \
  -e WATCHTOWER_LABEL_ENABLE=true \
  -e WATCHTOWER_CLEANUP=true \
  -e WATCHTOWER_INCLUDE_RESTARTING=true \
  containrrr/watchtower

The ~/.docker/config.json must have a valid login for git.manko.yoga (created via docker login git.manko.yoga -u manawenuz).

---

5. First-time deploy (cold start)

[!warning] Run these steps on a fresh production host. They are destructive on an existing one. See Backup & Recovery before touching live data.

Prerequisites on the host

• Ubuntu 22.04+ (or any systemd Linux), Docker Engine 24+, docker compose plugin
• git installed
• DNS amn.gg + dev.amn.gg already pointing here
• An SSL terminator (Caddy / Nginx / Cloudflare) reverse-proxying to host port 8083
• Registry login: docker login git.manko.yoga -u manawenuz

Steps

# 1. Clone both repos as siblings (compose references ../frontend)
cd /opt
git clone ssh://git@git.manko.yoga:222/nick/backend.git
git clone ssh://git@git.manko.yoga:222/nick/frontend.git
cd backend
git checkout main

# 2. Create the production .env
sudo nano .env       # fill from Environment Variables doc; production values, real secrets

# 3. Provision the nginx config + uploads dir
mkdir -p nginx/logs uploads mongo-init
sudo cp /path/to/nginx.conf nginx/nginx.conf
# (the nginx.conf forwards /api/* and /socket.io/* to nickapp-backend:5001,
#  forwards /uploads/* to /uploads (volume), and everything else to nickapp-frontend:8083)

# 4. Build & start the stack
docker compose -f docker-compose.production.yml up --build -d

# 5. Verify
docker compose -f docker-compose.production.yml ps
docker compose -f docker-compose.production.yml logs -f --tail=200
curl -fsS http://localhost:8083/api/health | jq .

# 6. Seed initial data (optional — if AUTO_SEED_ON_START=true is set, it's already done)
docker compose -f docker-compose.production.yml exec nickapp-backend node dist/scripts/seedCategories.js

# 7. Start Watchtower (one-time)
docker run -d --name watchtower --restart unless-stopped \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /root/.docker/config.json:/config.json \
  -e WATCHTOWER_POLL_INTERVAL=300 \
  -e WATCHTOWER_LABEL_ENABLE=true \
  -e WATCHTOWER_CLEANUP=true \
  containrrr/watchtower

SSL / TLS

Termination happens at the edge — outside the compose stack. The two common setups:

• Caddy on the host forwarding amn.gg and dev.amn.gg to 127.0.0.1:8083. Caddy handles Let's Encrypt automatically.
• Cloudflare Full (strict) in front of the host. Use Cloudflare Origin certificates on the host's Caddy/Nginx.

Either way, the compose stack itself sees only HTTP on port 80 inside the nginx container. The nginx.conf should set proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto and the backend already trusts the proxy when NODE_ENV=production (see trust proxy block in src/app.ts).

---

6. Routine deploy (after first deploy)

The normal flow is fully automatic:

1. Developer merges PR to main (see Git Workflow).
2. Gitea Actions runs .gitea/workflows/docker-build-no-cache.yml (backend) or deploy.yml (frontend). The workflow builds the production image and pushes :latest + :<version> to the registry. See CI-CD Pipeline.
3. Watchtower polls the registry, sees a new digest, restarts the container.
4. Healthcheck on the new container passes after start_period=40s, traffic resumes.

Total time from merge to live: 5–10 minutes depending on Watchtower poll interval and image size.

Force an immediate deploy

If you don't want to wait for the poll:

# On the production host:
cd /opt/backend
docker login git.manko.yoga -u manawenuz       # if creds expired
docker compose -f docker-compose.production.yml pull nickapp-backend nickapp-frontend
docker compose -f docker-compose.production.yml up -d nickapp-backend nickapp-frontend

The up -d will detect changed images and restart only the affected containers.

Roll back

# Find available versions
docker images git.manko.yoga/manawenuz/escrow-backend

# Pin to the previous tag in the compose file
sed -i 's|escrow-backend:latest|escrow-backend:2.6.2|' docker-compose.production.yml

# Re-up
docker compose -f docker-compose.production.yml up -d nickapp-backend

# Disable Watchtower for the affected service until you're ready to resume
docker compose ... restart   # no-op if you removed the watchtower label

[!warning] Watchtower will undo a pin to a non-latest tag on its next poll if the container still has the watchtower.enable=true label. Either remove the label temporarily or pause Watchtower (docker stop watchtower).

---

7. Logs

# All services
docker compose -f docker-compose.production.yml logs -f --tail=300

# Single service
docker compose -f docker-compose.production.yml logs -f nickapp-backend

# Nginx access log
tail -f /opt/backend/nginx/logs/access.log

Backend logs are also captured by Sentry breadcrumbs when an error occurs — see Monitoring.

---

8. Maintenance window

Plan a 5-minute window when bumping major versions or running migrations:

# Announce + drain
# (set a maintenance banner in the frontend if possible)

# Take a backup first
./scripts/backup-mongo.sh    # or per Backup & Recovery

# Pull new images, restart
docker compose -f docker-compose.production.yml pull
docker compose -f docker-compose.production.yml up -d

# Verify
curl -fsS https://amn.gg/api/health

If anything goes sideways, follow Incident Response.