nick-doc/08 - Operations/Deployment.md

---
title: Deployment
tags: [operations]
---

# Deployment

How the production stack runs and gets updated on the live host. The stack is fully containerised and self-updates via Watchtower from the Gitea container registry.

---

## 1. Topology

```
                                  ┌─────────────────────────┐
            HTTPS  443 ──────────►│   External SSL term.    │
                                  │ (DNS amn.gg, dev.amn.gg)│
                                  └────────────┬────────────┘
                                               │ HTTP 80 (in-VPC)
                                               ▼
                            ┌──────────────────────────────────┐
                            │           Nginx container        │
                            │       (nickapp-nginx, port 80)   │
                            └─┬───────────────────┬────────────┘
                              │                   │
                              │ /                 │ /api  /socket.io
                              ▼                   ▼
                ┌─────────────────────┐  ┌──────────────────────────┐
                │ nickapp-frontend    │  │ nickapp-backend          │
                │ Next.js, port 8083  │  │ Express 5, port 5001     │
                └─────────────────────┘  └──────┬────────────┬──────┘
                                                │            │
                                                ▼            ▼
                                         ┌──────────┐  ┌──────────┐
                                         │ mongodb  │  │  redis   │
                                         │  8.2     │  │  8       │
                                         └──────────┘  └──────────┘

                            ┌──────────────────────────────────┐
                            │            Watchtower            │
                            │ Polls registry → restarts        │
                            │ containers labelled enable=true  │
                            └──────────────────────────────────┘
```

All containers run on the **`default`** Docker network defined by `docker-compose.production.yml`. Watchtower runs as a sidecar container on the same host.

DNS resolves both `amn.gg` and `dev.amn.gg` to the production host's public IP. SSL termination happens **outside** the compose stack (typically via the hosting provider's edge or a host-level reverse proxy), and traffic is forwarded as HTTP to the `nginx` container on port `80` (mapped to host `8083`).

---

## 2. Compose file

`backend/docker-compose.production.yml` is the single source of truth. Services:

| Service | Image | Ports | Volumes | Notes |
|---------|-------|-------|---------|-------|
| `nginx` | `nginx:alpine` | `8083:80` | `./nginx/nginx.conf`, `./nginx/logs`, `./uploads` (served as `/uploads`) | Reverse proxy |
| `nickapp-backend` | `nickapp-backend:latest` (build from `Dockerfile.prod`) | not exposed externally | `./uploads:/app/uploads` | Labelled for Watchtower |
| `nickapp-frontend` | `nickapp-frontend:latest` (build from `../frontend/Dockerfile`) | `expose: 8083` | — | Labelled for Watchtower |
| `mongodb` | `mongo:8.2` | not exposed | `mongodb_data:/data/db`, `./mongo-init:/docker-entrypoint-initdb.d` | Healthcheck via `mongosh ping` |
| `redis` | `redis:8-alpine` | not exposed | `redis_data:/data` | Started with `--requirepass "$REDIS_PASSWORD"` |

Healthchecks are configured for backend (`curl /health`), frontend (`curl /`), Mongo (`mongosh ping`), and Redis (`redis-cli -a $REDIS_PASSWORD ping`). See [[Monitoring]].

Watchtower polls images labelled `com.centurylinklabs.watchtower.enable=true` — currently `nickapp-backend` and `nickapp-frontend`. MongoDB and Redis are **not** auto-updated.

---

## 3. Registry & images

| Image | Registry path |
|-------|---------------|
| Backend prod | `git.manko.yoga/manawenuz/escrow-backend:latest` |
| Backend dev | `git.manko.yoga/manawenuz/escrow-backend:dev` |
| Backend tagged | `git.manko.yoga/manawenuz/escrow-backend:<package-version>` |
| Frontend | `git.manko.yoga/manawenuz/escrow-frontend:latest` and `:<version>` |

`docker-compose.production.yml` currently builds locally on first up (`build: context: .`). Once images are in the registry the file can be switched to `image: git.manko.yoga/manawenuz/escrow-backend:latest` to let Watchtower pull straight from there.

> [!tip] To pin a specific version while debugging, edit the compose file to `image: git.manko.yoga/manawenuz/escrow-backend:2.6.3` and re-run `docker compose up -d`. Remove the Watchtower label or the agent will undo it on next poll.

---

## 4. Watchtower

Watchtower runs as its own container (managed outside the compose file) with `WATCHTOWER_LABEL_ENABLE=true` so it only touches services that opt in. On each poll cycle (default 5 minutes, configurable via `WATCHTOWER_POLL_INTERVAL`) it:

1. Pulls the latest digest for each enabled service's image.
2. Compares to the running container's digest.
3. If different, stops the container, removes it, and starts a new one from the new image, preserving all named volumes.

Configuration knobs typically set on the host:

```bash
docker run -d --name watchtower \
  --restart unless-stopped \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /root/.docker/config.json:/config.json \   # so it can pull from the private Gitea registry
  -e WATCHTOWER_POLL_INTERVAL=300 \
  -e WATCHTOWER_LABEL_ENABLE=true \
  -e WATCHTOWER_CLEANUP=true \
  -e WATCHTOWER_INCLUDE_RESTARTING=true \
  containrrr/watchtower
```

The `~/.docker/config.json` must have a valid login for `git.manko.yoga` (created via `docker login git.manko.yoga -u manawenuz`).

---

## 5. First-time deploy (cold start)

> [!warning] Run these steps on a fresh production host. They are destructive on an existing one. See [[Backup & Recovery]] before touching live data.

### Prerequisites on the host

- Ubuntu 22.04+ (or any systemd Linux), Docker Engine 24+, `docker compose` plugin
- `git` installed
- DNS `amn.gg` + `dev.amn.gg` already pointing here
- An SSL terminator (Caddy / Nginx / Cloudflare) reverse-proxying to host port `8083`
- Registry login: `docker login git.manko.yoga -u manawenuz`

### Steps

```bash
# 1. Clone both repos as siblings (compose references ../frontend)
cd /opt
git clone ssh://git@git.manko.yoga:222/nick/backend.git
git clone ssh://git@git.manko.yoga:222/nick/frontend.git
cd backend
git checkout main

# 2. Create the production .env
sudo nano .env       # fill from Environment Variables doc; production values, real secrets

# 3. Provision the nginx config + uploads dir
mkdir -p nginx/logs uploads mongo-init
sudo cp /path/to/nginx.conf nginx/nginx.conf
# (the nginx.conf forwards /api/* and /socket.io/* to nickapp-backend:5001,
#  forwards /uploads/* to /uploads (volume), and everything else to nickapp-frontend:8083)

# 4. Build & start the stack
docker compose -f docker-compose.production.yml up --build -d

# 5. Verify
docker compose -f docker-compose.production.yml ps
docker compose -f docker-compose.production.yml logs -f --tail=200
curl -fsS http://localhost:8083/api/health | jq .

# 6. Seed initial data (optional — if AUTO_SEED_ON_START=true is set, it's already done)
docker compose -f docker-compose.production.yml exec nickapp-backend node dist/scripts/seedCategories.js

# 7. Start Watchtower (one-time)
docker run -d --name watchtower --restart unless-stopped \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /root/.docker/config.json:/config.json \
  -e WATCHTOWER_POLL_INTERVAL=300 \
  -e WATCHTOWER_LABEL_ENABLE=true \
  -e WATCHTOWER_CLEANUP=true \
  containrrr/watchtower
```

### SSL / TLS

Termination happens at the edge — outside the compose stack. The two common setups:

- **Caddy on the host** forwarding `amn.gg` and `dev.amn.gg` to `127.0.0.1:8083`. Caddy handles Let's Encrypt automatically.
- **Cloudflare Full (strict)** in front of the host. Use Cloudflare Origin certificates on the host's Caddy/Nginx.

Either way, the compose stack itself sees only HTTP on port 80 inside the nginx container. The `nginx.conf` should set `proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto` and the backend already trusts the proxy when `NODE_ENV=production` (see `trust proxy` block in `src/app.ts`).

---

## 6. Routine deploy (after first deploy)

The normal flow is **fully automatic**:

1. Developer merges PR to `main` (see [[Git Workflow]]).
2. Gitea Actions runs `.gitea/workflows/docker-build-no-cache.yml` (backend) or `deploy.yml` (frontend). The workflow builds the production image and pushes `:latest` + `:<version>` to the registry. See [[CI-CD Pipeline]].
3. Watchtower polls the registry, sees a new digest, restarts the container.
4. Healthcheck on the new container passes after `start_period=40s`, traffic resumes.

Total time from merge to live: **5–10 minutes** depending on Watchtower poll interval and image size.

### Force an immediate deploy

If you don't want to wait for the poll:

```bash
# On the production host:
cd /opt/backend
docker login git.manko.yoga -u manawenuz       # if creds expired
docker compose -f docker-compose.production.yml pull nickapp-backend nickapp-frontend
docker compose -f docker-compose.production.yml up -d nickapp-backend nickapp-frontend
```

The `up -d` will detect changed images and restart only the affected containers.

### Roll back

```bash
# Find available versions
docker images git.manko.yoga/manawenuz/escrow-backend

# Pin to the previous tag in the compose file
sed -i 's|escrow-backend:latest|escrow-backend:2.6.2|' docker-compose.production.yml

# Re-up
docker compose -f docker-compose.production.yml up -d nickapp-backend

# Disable Watchtower for the affected service until you're ready to resume
docker compose ... restart   # no-op if you removed the watchtower label
```

> [!warning] Watchtower will undo a pin to a non-`latest` tag on its next poll if the container still has the `watchtower.enable=true` label. Either remove the label temporarily or pause Watchtower (`docker stop watchtower`).

---

## 7. Logs

```bash
# All services
docker compose -f docker-compose.production.yml logs -f --tail=300

# Single service
docker compose -f docker-compose.production.yml logs -f nickapp-backend

# Nginx access log
tail -f /opt/backend/nginx/logs/access.log
```

Backend logs are also captured by Sentry breadcrumbs when an error occurs — see [[Monitoring]].

---

## 8. Maintenance window

Plan a 5-minute window when bumping major versions or running migrations:

```bash
# Announce + drain
# (set a maintenance banner in the frontend if possible)

# Take a backup first
./scripts/backup-mongo.sh    # or per Backup & Recovery

# Pull new images, restart
docker compose -f docker-compose.production.yml pull
docker compose -f docker-compose.production.yml up -d

# Verify
curl -fsS https://amn.gg/api/health
```

If anything goes sideways, follow [[Incident Response]].