nick/nick-doc
3
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Initial commit: nick docs

Browse Source
This commit is contained in:
moojttaba
2026-05-23 20:35:34 +03:30
commit 0da235ae27
90 changed files with 18268 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

255
08 - Operations/Deployment.md Normal file
View File

@@ -0,0 +1,255 @@
---
title: Deployment
tags: [operations]
---
# Deployment
How the production stack runs and gets updated on the live host. The stack is fully containerised and self-updates via Watchtower from the Gitea container registry.
---
## 1. Topology
```
┌─────────────────────────┐
HTTPS 443 ──────────►│ External SSL term. │
│ (DNS amn.gg, dev.amn.gg)│
└────────────┬────────────┘
│ HTTP 80 (in-VPC)
▼
┌──────────────────────────────────┐
│ Nginx container │
│ (nickapp-nginx, port 80) │
└─┬───────────────────┬────────────┘
│ │
│ / │ /api /socket.io
▼ ▼
┌─────────────────────┐ ┌──────────────────────────┐
│ nickapp-frontend │ │ nickapp-backend │
│ Next.js, port 8083 │ │ Express 5, port 5001 │
└─────────────────────┘ └──────┬────────────┬──────┘
│ │
▼ ▼
┌──────────┐ ┌──────────┐
│ mongodb │ │ redis │
│ 8.2 │ │ 8 │
└──────────┘ └──────────┘
┌──────────────────────────────────┐
│ Watchtower │
│ Polls registry → restarts │
│ containers labelled enable=true │
└──────────────────────────────────┘
```
All containers run on the **`default`** Docker network defined by `docker-compose.production.yml`. Watchtower runs as a sidecar container on the same host.
DNS resolves both `amn.gg` and `dev.amn.gg` to the production host's public IP. SSL termination happens **outside** the compose stack (typically via the hosting provider's edge or a host-level reverse proxy), and traffic is forwarded as HTTP to the `nginx` container on port `80` (mapped to host `8083`).
---
## 2. Compose file
`backend/docker-compose.production.yml` is the single source of truth. Services:
| Service | Image | Ports | Volumes | Notes |
|---------|-------|-------|---------|-------|
| `nginx` | `nginx:alpine` | `8083:80` | `./nginx/nginx.conf`, `./nginx/logs`, `./uploads` (served as `/uploads`) | Reverse proxy |
| `nickapp-backend` | `nickapp-backend:latest` (build from `Dockerfile.prod`) | not exposed externally | `./uploads:/app/uploads` | Labelled for Watchtower |
| `nickapp-frontend` | `nickapp-frontend:latest` (build from `../frontend/Dockerfile`) | `expose: 8083` | — | Labelled for Watchtower |
| `mongodb` | `mongo:8.2` | not exposed | `mongodb_data:/data/db`, `./mongo-init:/docker-entrypoint-initdb.d` | Healthcheck via `mongosh ping` |
| `redis` | `redis:8-alpine` | not exposed | `redis_data:/data` | Started with `--requirepass "$REDIS_PASSWORD"` |
Healthchecks are configured for backend (`curl /health`), frontend (`curl /`), Mongo (`mongosh ping`), and Redis (`redis-cli -a $REDIS_PASSWORD ping`). See [[Monitoring]].
Watchtower polls images labelled `com.centurylinklabs.watchtower.enable=true` — currently `nickapp-backend` and `nickapp-frontend`. MongoDB and Redis are **not** auto-updated.
---
## 3. Registry & images
| Image | Registry path |
|-------|---------------|
| Backend prod | `git.manko.yoga/manawenuz/escrow-backend:latest` |
| Backend dev | `git.manko.yoga/manawenuz/escrow-backend:dev` |
| Backend tagged | `git.manko.yoga/manawenuz/escrow-backend:<package-version>` |
| Frontend | `git.manko.yoga/manawenuz/escrow-frontend:latest` and `:<version>` |
`docker-compose.production.yml` currently builds locally on first up (`build: context: .`). Once images are in the registry the file can be switched to `image: git.manko.yoga/manawenuz/escrow-backend:latest` to let Watchtower pull straight from there.
> [!tip] To pin a specific version while debugging, edit the compose file to `image: git.manko.yoga/manawenuz/escrow-backend:2.6.3` and re-run `docker compose up -d`. Remove the Watchtower label or the agent will undo it on next poll.
---
## 4. Watchtower
Watchtower runs as its own container (managed outside the compose file) with `WATCHTOWER_LABEL_ENABLE=true` so it only touches services that opt in. On each poll cycle (default 5 minutes, configurable via `WATCHTOWER_POLL_INTERVAL`) it:
1. Pulls the latest digest for each enabled service's image.
2. Compares to the running container's digest.
3. If different, stops the container, removes it, and starts a new one from the new image, preserving all named volumes.
Configuration knobs typically set on the host:
```bash
docker run -d --name watchtower \
--restart unless-stopped \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /root/.docker/config.json:/config.json \ # so it can pull from the private Gitea registry
-e WATCHTOWER_POLL_INTERVAL=300 \
-e WATCHTOWER_LABEL_ENABLE=true \
-e WATCHTOWER_CLEANUP=true \
-e WATCHTOWER_INCLUDE_RESTARTING=true \
containrrr/watchtower
```
The `~/.docker/config.json` must have a valid login for `git.manko.yoga` (created via `docker login git.manko.yoga -u manawenuz`).
---
## 5. First-time deploy (cold start)
> [!warning] Run these steps on a fresh production host. They are destructive on an existing one. See [[Backup & Recovery]] before touching live data.
### Prerequisites on the host
- Ubuntu 22.04+ (or any systemd Linux), Docker Engine 24+, `docker compose` plugin
- `git` installed
- DNS `amn.gg` + `dev.amn.gg` already pointing here
- An SSL terminator (Caddy / Nginx / Cloudflare) reverse-proxying to host port `8083`
- Registry login: `docker login git.manko.yoga -u manawenuz`
### Steps
```bash
# 1. Clone both repos as siblings (compose references ../frontend)
cd /opt
git clone ssh://git@git.manko.yoga:222/nick/backend.git
git clone ssh://git@git.manko.yoga:222/nick/frontend.git
cd backend
git checkout main
# 2. Create the production .env
sudo nano .env # fill from Environment Variables doc; production values, real secrets
# 3. Provision the nginx config + uploads dir
mkdir -p nginx/logs uploads mongo-init
sudo cp /path/to/nginx.conf nginx/nginx.conf
# (the nginx.conf forwards /api/* and /socket.io/* to nickapp-backend:5001,
# forwards /uploads/* to /uploads (volume), and everything else to nickapp-frontend:8083)
# 4. Build & start the stack
docker compose -f docker-compose.production.yml up --build -d
# 5. Verify
docker compose -f docker-compose.production.yml ps
docker compose -f docker-compose.production.yml logs -f --tail=200
curl -fsS http://localhost:8083/api/health | jq .
# 6. Seed initial data (optional — if AUTO_SEED_ON_START=true is set, it's already done)
docker compose -f docker-compose.production.yml exec nickapp-backend node dist/scripts/seedCategories.js
# 7. Start Watchtower (one-time)
docker run -d --name watchtower --restart unless-stopped \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /root/.docker/config.json:/config.json \
-e WATCHTOWER_POLL_INTERVAL=300 \
-e WATCHTOWER_LABEL_ENABLE=true \
-e WATCHTOWER_CLEANUP=true \
containrrr/watchtower
```
### SSL / TLS
Termination happens at the edge — outside the compose stack. The two common setups:
- **Caddy on the host** forwarding `amn.gg` and `dev.amn.gg` to `127.0.0.1:8083`. Caddy handles Let's Encrypt automatically.
- **Cloudflare Full (strict)** in front of the host. Use Cloudflare Origin certificates on the host's Caddy/Nginx.
Either way, the compose stack itself sees only HTTP on port 80 inside the nginx container. The `nginx.conf` should set `proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto` and the backend already trusts the proxy when `NODE_ENV=production` (see `trust proxy` block in `src/app.ts`).
---
## 6. Routine deploy (after first deploy)
The normal flow is **fully automatic**:
1. Developer merges PR to `main` (see [[Git Workflow]]).
2. Gitea Actions runs `.gitea/workflows/docker-build-no-cache.yml` (backend) or `deploy.yml` (frontend). The workflow builds the production image and pushes `:latest` + `:<version>` to the registry. See [[CI-CD Pipeline]].
3. Watchtower polls the registry, sees a new digest, restarts the container.
4. Healthcheck on the new container passes after `start_period=40s`, traffic resumes.
Total time from merge to live: **5–10 minutes** depending on Watchtower poll interval and image size.
### Force an immediate deploy
If you don't want to wait for the poll:
```bash
# On the production host:
cd /opt/backend
docker login git.manko.yoga -u manawenuz # if creds expired
docker compose -f docker-compose.production.yml pull nickapp-backend nickapp-frontend
docker compose -f docker-compose.production.yml up -d nickapp-backend nickapp-frontend
```
The `up -d` will detect changed images and restart only the affected containers.
### Roll back
```bash
# Find available versions
docker images git.manko.yoga/manawenuz/escrow-backend
# Pin to the previous tag in the compose file
sed -i 's|escrow-backend:latest|escrow-backend:2.6.2|' docker-compose.production.yml
# Re-up
docker compose -f docker-compose.production.yml up -d nickapp-backend
# Disable Watchtower for the affected service until you're ready to resume
docker compose ... restart # no-op if you removed the watchtower label
```
> [!warning] Watchtower will undo a pin to a non-`latest` tag on its next poll if the container still has the `watchtower.enable=true` label. Either remove the label temporarily or pause Watchtower (`docker stop watchtower`).
---
## 7. Logs
```bash
# All services
docker compose -f docker-compose.production.yml logs -f --tail=300
# Single service
docker compose -f docker-compose.production.yml logs -f nickapp-backend
# Nginx access log
tail -f /opt/backend/nginx/logs/access.log
```
Backend logs are also captured by Sentry breadcrumbs when an error occurs — see [[Monitoring]].
---
## 8. Maintenance window
Plan a 5-minute window when bumping major versions or running migrations:
```bash
# Announce + drain
# (set a maintenance banner in the frontend if possible)
# Take a backup first
./scripts/backup-mongo.sh # or per Backup & Recovery
# Pull new images, restart
docker compose -f docker-compose.production.yml pull
docker compose -f docker-compose.production.yml up -d
# Verify
curl -fsS https://amn.gg/api/health
```
If anything goes sideways, follow [[Incident Response]].