nick-doc/08 - Operations/Monitoring.md

---
title: Monitoring
tags: [operations]
---

# Monitoring

What's instrumented today and what to watch. Today's stack is intentionally lean — health endpoints, Docker healthchecks, Sentry, and access logs. Bigger metric pipelines (Prometheus, Grafana, OpenSearch) are a future addition.

---

## 1. Health endpoint

Two paths are registered (both are public, rate-limited, not auth-gated):

- `GET /health` — simple ping used by Docker healthchecks. Returns `200 { success, message, timestamp, environment, version }`. Does **not** probe MongoDB or Redis.
- `GET /api/health` — deep health check added in commit `44579d6` (backend v2.6.49). Calls `runHealthChecks` from `backend/src/services/health/healthCheckService.ts`. Probes MongoDB, Postgres, Redis, Request Network registry data, and Request Network API reachability. Returns `503` only when `report.status === 'down'`. As of backend `2.8.79`, Postgres is a hard dependency only when at least one `*_STORE=postgres` flag is enabled; otherwise an unconfigured Postgres check is reported as skipped. The Postgres check also reports active store modes so monitoring can distinguish "PG is reachable" from "this runtime is actually using PG-backed stores". As of deployment `38cb75b`, dev Gatus requires all seven PG-capable store modes to be `postgres` and `enabledStoreCount >= 7`.

`GET /api/health` response shape (from `healthCheckService`):
```json
{
  "status": "ok",
  "version": "2.8.79",
  "uptimeSec": 662,
  "checks": {
    "db": { "ok": true, "latencyMs": 4 },
    "postgres": {
      "ok": true,
      "latencyMs": 5,
      "configured": true,
      "required": true,
      "storeModes": {
        "auth": "postgres",
        "config": "postgres",
        "address": "postgres",
        "category": "postgres",
        "levelConfig": "postgres",
        "shopSettings": "postgres",
        "review": "postgres"
      },
      "enabledStores": [
        "auth",
        "config",
        "address",
        "category",
        "levelConfig",
        "shopSettings",
        "review"
      ],
      "enabledStoreCount": 7,
      "database": "amanat_dev",
      "user": "amanat"
    },
    "redis": { "ok": true, "latencyMs": 1 },
    "rnChainRegistry": { "ok": true, "latencyMs": 0, "chainCount": 7 },
    "rnTokenRegistry": { "ok": true, "latencyMs": 0, "tokenCount": 12 },
    "rnApi": { "ok": true, "latencyMs": 134, "status": 401 }
  }
}
```

Public URL behind Nginx: `https://amn.gg/api/health`.

---

## 2. Docker healthchecks

Each long-lived container has a `HEALTHCHECK` baked in or declared in compose.

| Container | Probe | Interval | Failure threshold |
|-----------|-------|----------|-------------------|
| `nickapp-backend` | `node healthcheck.js` (HTTP GET `/health`) | 30s | 3 retries |
| `nickapp-frontend` | `curl -f http://localhost:8083/` | 30s | 3 retries |
| `mongodb` | `mongosh --eval "db.adminCommand('ping')"` | 30s | 3 retries |
| `redis` | `redis-cli -a $REDIS_PASSWORD ping` | 30s | 3 retries |

`healthcheck.js` (backend) is a tiny Node script that does a local HTTP GET to `/health` and exits 0 / 1.

Inspect health:

```bash
docker ps --format "table {{.Names}}\t{{.Status}}"

# Detailed
docker inspect --format='{{json .State.Health}}' nickapp-backend | jq
```

If a container is `unhealthy`, Watchtower will **not** roll it (it expects the new container to pass healthcheck). Investigate with `docker logs <container>`.

---

## 3. Sentry — error tracking

### Frontend

`@sentry/nextjs ^10.22.0` is wired in via three config files at the repo root:

- `sentry.client.config.ts` — browser SDK (with Session Replay enabled at 10% session / 100% error rate).
- `sentry.server.config.ts` — server-rendered components (no Replay).
- `sentry.edge.config.ts` — edge runtime (not currently used heavily).

Common settings:

```ts
Sentry.init({
  dsn: process.env.NEXT_PUBLIC_SENTRY_DSN,
  tracesSampleRate: process.env.NODE_ENV === 'production' ? 0.1 : 1.0,
  environment: process.env.NODE_ENV || 'development',
  enabled: process.env.NODE_ENV === 'production',
  ignoreErrors: ['ResizeObserver loop limit exceeded', 'ChunkLoadError', ...],
});
```

Errors from `localhost` are filtered out — only prod errors land in the dashboard.

### Backend

`@sentry/node ^10.22.0` + `@sentry/profiling-node ^10.22.0` are initialised **first** in `src/app.ts` (before any other import) via `src/config/sentry.ts`. DSN comes from `SENTRY_DSN` env var (see [[Environment Variables#sentry]]).

What's captured:

- Uncaught exceptions in route handlers
- Promise rejections inside `asyncHandler`-wrapped routes
- Manually-captured errors via `Sentry.captureException(err)`
- Performance traces (10% sample rate in prod)
- Profiling samples via `@sentry/profiling-node`

### Source maps

Frontend uploads source maps to Sentry at build time when `SENTRY_AUTH_TOKEN`, `SENTRY_ORG`, and `SENTRY_PROJECT` are set in the CI env. Without them the build still succeeds but Sentry traces will show minified frames.

### Alerts

Configure in the Sentry dashboard (Issues → Alerts) — common alerts:

- Any new issue in production → Slack
- Error frequency > 50/minute → page on-call
- Performance regression on `/api/payments/*` traces → email

---

## 4. Logs

### Backend application logs

Routed through `src/utils/logger.ts` — currently a thin `console.log` wrapper with emoji prefixes. Output goes to stdout, captured by Docker:

```bash
# Live tail
docker compose -f docker-compose.production.yml logs -f --tail=200 nickapp-backend

# Search for a request
docker logs nickapp-backend 2>&1 | grep "POST /api/payments"

# Pre-filter by date
docker logs --since 1h nickapp-backend
```

Notable log lines to look for:

| Prefix | Meaning |
|--------|---------|
| `✅ Connected to MongoDB` | DB connection established |
| `🚀 Server running on port 5001` | App fully started |
| `🔌 User connected: <id>` | Socket.IO connection |
| `📥` | Inbound HTTP request log |
| `💳 Request Network` | Request Network webhook / API call |
| `🔐 Webhook verification` | Webhook signature check result |
| `❌ Error` | Manual error log (also captured by Sentry) |

### Nginx access + error logs

Bind-mounted to `./nginx/logs/` on the host:

```bash
tail -f /opt/backend/nginx/logs/access.log
tail -f /opt/backend/nginx/logs/error.log
```

Rotate these via host `logrotate` to avoid disk fill.

### Frontend logs

Next.js logs go to the container stdout:

```bash
docker logs -f nickapp-frontend
```

Browser-side logs that need attention go through Sentry (above) — `src/utils/logger.ts` in the frontend forwards via Sentry breadcrumbs.

---

## 5. Key metrics to watch

Today these are read manually from logs / Sentry. As Prometheus is added, encode them as alerting rules.

### Application

| Metric | Where to check | Healthy | Alert |
|--------|---------------|---------|-------|
| 5xx rate | Sentry, Nginx access.log | < 0.5 % | > 2 % over 5 min |
| `/health` p95 latency | curl + timer | < 100 ms | > 1 s |
| Login success rate | Sentry custom event | > 95 % | < 90 % |
| Socket disconnect storm | `🔌 User disconnected` log frequency | < 1/s sustained | > 10/s sustained |
| OpenAI 429s | Backend log `OpenAI ... 429` | 0 | any |

### Payments

| Metric | Where | Healthy | Alert |
|--------|-------|---------|-------|
| Payment success rate | `db.payments.aggregate([{$group:{_id:"$status",n:{$sum:1}}}])` | > 95 % completed of 24h-old payments | < 90 % |
| Webhook signature failures | log `Webhook verification failed` | 0 | > 0 |
| Request Network webhook 4xx | nginx access log `/api/payment/request-network/webhook` | 0 | any real provider delivery returning 4xx |
| Request Network safety-pending payments | `db.payments.find({"metadata.transactionSafety.status":"pending"})` | explained/short-lived | pending > 10 min without operator note |
| Request Network API errors (5xx) | log + Sentry | 0 | > 5/min sustained |
| Payouts stuck in `pending` > 30 min | `db.payments.find({type:'payout',status:'pending',createdAt:{$lt:ISODate(30 min ago)}})` | empty | non-empty |
| Missing `transactionHash` after `completed` | the same query that drives `fix-transaction-hashes.js` | empty | non-empty |

### MongoDB

```js
db.serverStatus().connections           // active connections; alert if >1000
db.serverStatus().opcounters            // ops/sec
db.serverStatus().wiredTiger.cache      // cache hit ratio; aim > 95 %
db.currentOp({ secs_running: { $gte: 5 } })  // long-running queries
```

### Redis

```bash
docker exec nickapp-redis redis-cli -a "$REDIS_PASSWORD" INFO stats
# Watch: instantaneous_ops_per_sec, keyspace_hits/misses, rejected_connections, evicted_keys
```

Alert thresholds: `rejected_connections > 0`, `evicted_keys` rising while you don't expect cache pressure, `latency_ms` p99 > 5ms.

### Host

| Metric | Tool | Healthy | Alert |
|--------|------|---------|-------|
| Disk usage on `/var/lib/docker` | `df -h` | < 80 % | > 90 % |
| `/opt/backend/uploads` size | `du -sh` | watch trend | bursty growth (>5 GB/day) |
| Memory pressure | `free -h`, `docker stats` | < 80 % | swap actively used |
| Open file descriptors | `cat /proc/<pid>/limits` | well under hard limit | nearing limit |

---

## 6. Smoke tests after a deploy

Drop these in a runbook for the on-call:

```bash
# 1. API health
curl -fsS https://amn.gg/api/health | jq '.success,.version,.environment'

# 2. Login
curl -fsS -X POST https://amn.gg/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@marketplace.com","password":"<prod-admin-pwd>"}' \
  | jq '.success,.data.user.email'

# 3. Frontend HTML loads
curl -fsS https://amn.gg/ -I | head -1   # expect 200

# 4. Socket.IO handshake
curl -fsS "https://amn.gg/socket.io/?EIO=4&transport=polling" -I | head -1

# 5. Containers healthy
docker ps --filter "name=nickapp-" --format "table {{.Names}}\t{{.Status}}"
```

Any non-OK → see [[Incident Response]].

---

## 7. Future work

- **Prometheus + Grafana** with Node exporter + Mongo exporter + Redis exporter — for proper time-series.
- **OpenTelemetry** spans from backend → Sentry / Jaeger.
- **Healthcheck endpoint** that probes Mongo + Redis and returns `503` when degraded.
- **PagerDuty / OpsGenie** wiring from Sentry alerts.
- **Synthetic checks** (Pingdom / UptimeRobot) hitting `/health` from multiple regions.

For now, Sentry + Docker healthchecks + manual log checks cover the basics. See [[Incident Response]] for what to do when something fires.