--- title: Database Operations tags: [operations] --- # Database Operations Day-to-day operations for stateful services: **MongoDB 8.x** (primary runtime data store), **PostgreSQL 18** (migration target and conditional oracle quote store), and **Redis 8** (cache, rate-limit counters, ephemeral session data). For schema details see [[Data Models]]. For backup procedures and disaster recovery see [[Backup & Recovery]]. --- ## 1. MongoDB ### 1.1 Connection | Env | URI in compose | Auth | |-----|---------------|------| | Dev | `mongodb://mongodb:27017` | none | | Prod | `mongodb://mongodb:27017` (private network) or with creds via `.env` | typically none on the private network, but enable `--auth` if exposed | The DB name comes from `DB_NAME` (e.g. `marketplace`). See [[Environment Variables#database]]. Connect from a shell inside the host: ```bash # Dev docker exec -it nickdev-mongodb mongosh # Prod docker exec -it nickapp-mongodb mongosh > use marketplace > show collections ``` If auth is enabled: ```bash docker exec -it nickapp-mongodb mongosh \ -u "$MONGO_INITDB_ROOT_USERNAME" -p "$MONGO_INITDB_ROOT_PASSWORD" \ --authenticationDatabase admin ``` ### 1.2 Init scripts (`mongo-init/`) The production compose bind-mounts `./mongo-init` into `/docker-entrypoint-initdb.d`. Mongo runs `*.js` and `*.sh` from this folder **only on a fresh datadir** (first boot of a new volume). Use this to: - Create application users (`db.createUser({...})`) - Bootstrap collections + indexes that must exist before the app starts Example `mongo-init/01-create-user.js`: ```js db = db.getSiblingDB('marketplace'); db.createUser({ user: 'marketplace_app', pwd: process.env.MARKETPLACE_APP_PWD, roles: [{ role: 'readWrite', db: 'marketplace' }], }); ``` > [!warning] These scripts do **not** run when you restart an existing container. To force re-init, drop the `mongodb_data` volume — which destroys all data. Plan accordingly. ### 1.3 Indexes Indexes are declared in Mongoose schemas under `backend/src/models/`. The app calls `Model.createIndexes()` on connection (via the model's `syncIndexes`/`ensureIndexes` lifecycle). Highlights: | Collection | Key indexes | |------------|-------------| | `users` | `email` (unique), `googleId` (sparse), `role`, `createdAt` | | `addresses` | `userId` + compound for primary lookup | | `purchaserequests` | `buyerId`, `status`, `createdAt`, text index on `title`+`description` | | `selleroffers` | `requestId`, `sellerId`, `status` | | `payments` | `providerPaymentId` (unique sparse), `userId`, `status`, `createdAt`, `transactionHash` | | `chats` | `participants` (array), `updatedAt` | | `notifications` | `userId` + `read`, `createdAt` | | `tempverifications` | TTL on `expiresAt` (auto-deletes expired OTPs) | To verify a specific collection: ```js db.payments.getIndexes() ``` To add a new index without code-gen — preferred path is to declare it in the Mongoose schema and ship a deploy. For emergency hotfixes: ```js db.payments.createIndex({ providerPaymentId: 1 }, { unique: true, sparse: true }); ``` ### 1.4 TTL indexes Currently used on `tempverifications.expiresAt` (5-minute auto-purge of email OTPs / passkey challenges). Mongo's TTL monitor runs every 60 seconds — purge isn't immediate. If you add more TTL indexes: ```js db.notifications.createIndex({ createdAt: 1 }, { expireAfterSeconds: 60 * 60 * 24 * 90 }); // 90 days ``` ### 1.5 Backup with `mongodump` ```bash # Connect into the container, dump locally, copy out docker exec nickapp-mongodb sh -c \ "mongodump --db=marketplace --archive=/tmp/marketplace-$(date +%F).archive --gzip" docker cp nickapp-mongodb:/tmp/marketplace-$(date +%F).archive ./backups/ # Or stream directly to host docker exec nickapp-mongodb \ mongodump --db=marketplace --archive --gzip \ > ./backups/marketplace-$(date +%F).gz ``` For full details (retention, RTO/RPO, offsite copies) see [[Backup & Recovery]]. ### 1.6 Restore ```bash # Restore an archive to an empty database docker exec -i nickapp-mongodb \ mongorestore --archive --gzip --drop \ < ./backups/marketplace-2026-05-20.gz ``` `--drop` drops each collection before restoring. Omit it to merge. > [!warning] Restoring is **destructive** to current data. Always practise on a staging clone first. ### 1.7 Migrations There is no formal migration framework. Two patterns are used: - **Mongoose schema changes** are forward-compatible (new optional fields default to `undefined`). Older documents will still load. - **Data backfills** are one-shot scripts in `backend/src/scripts/` (e.g. `migrateUserPoints.ts`, `fix-transaction-hashes.js`, `fix-dispute-sellers.js`). Pattern for a new migration: 1. Add a `src/seeds/migrate.ts` script that is idempotent (use `$exists: false` guards). 2. Run on staging, confirm. 3. Take a backup ([[Backup & Recovery]]). 4. Run in production: `docker exec -it nickapp-backend node dist/seeds/migrate.js`. 5. Commit the script (it serves as a record of what changed). ### 1.8 Common admin queries ```js // Count by collection db.users.countDocuments({ role: 'buyer' }) // Disk usage per collection db.runCommand({ collStats: 'payments', scale: 1024*1024 }).size // Slow queries db.setProfilingLevel(1, { slowms: 200 }) // log queries > 200ms db.system.profile.find().sort({ ts: -1 }).limit(10) // Lock contention db.serverStatus().locks ``` ### 1.9 Seeding production safely Seed scripts are designed to be idempotent for **categories** but **destructive** for users/addresses. Don't run `seed:all` in production. Safe in production: ```bash docker exec -it nickapp-backend node dist/seeds/seedCategories.js docker exec -it nickapp-backend node dist/seeds/seedLevels.js ``` Optional auto-seed on startup: set `AUTO_SEED_ON_START=true` in `.env`. The bootstrap code only seeds when no non-admin users exist — safe to leave on. > [!warning] **Never** run `seed:all` or `seed:users` against production. They drop the existing `users` and `addresses` collections. --- ## 2. PostgreSQL 18 ### 2.1 Runtime role Postgres is present in the current dev/integration stack, but MongoDB remains the primary runtime store. Use Postgres for: - Drizzle migrations and schema verification. - Mongo → Postgres backfill and reconciliation work. - `payment_quotes` when `ORACLE_QUOTING_ENABLED=true` and a PG parent payment row exists. Do **not** treat Postgres as the authoritative app database until the relevant domain has been wired through repository interfaces, backfilled, shadow-read, and cut over. See [[Postgres Runtime Cutover Status]]. ### 2.2 Docker volume layout for Postgres 18 Postgres 18 Docker images expect the mount at `/var/lib/postgresql`, not directly at `/var/lib/postgresql/data`, because the image stores data under a major-version-specific directory such as `/var/lib/postgresql/18/docker`. ```yaml postgres: image: postgres:18-alpine environment: POSTGRES_DB: amanat_dev POSTGRES_USER: amanat POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} volumes: - /var/data/escrowDev/postgres_data:/var/lib/postgresql ``` For a disposable dev reset: ```bash docker rm -f amanat-postgres 2>/dev/null || true rm -rf /var/data/escrowDev/postgres_data mkdir -p /var/data/escrowDev/postgres_data ``` ### 2.3 Apply migrations Run migrations only after the database is healthy and the DSN points at the intended non-production target: ```bash PG_URL=postgres://amanat:...@postgres:5432/amanat_dev npx drizzle-kit migrate ``` The backend image contains migrations through `0008`. Application startup does not apply them automatically. ### 2.4 Backfill and verification Backfills use `MIGRATION_PG_URL`, not `PG_URL`, and the scripts enforce a host allowlist. Run dry-run and verification before any dual-write/PG read flip: ```bash MIGRATION_MONGO_URL=mongodb://mongodb:27017/marketplace \ MIGRATION_PG_URL=postgres://amanat:...@postgres:5432/amanat_dev \ node dist/db/backfill/run-backfill.js --dry-run ``` Verify row counts/checksums and inspect `pg_dualwrite_gaps` before enabling any cutover flag. ### 2.5 Backup For dev/staging: ```bash docker exec amanat-postgres pg_dump -U amanat -d amanat_dev --format=custom \ > backups/amanat_dev_pg_$(date +%F).dump ``` Before production cutover, use managed backups or self-hosted WAL archiving/PITR. A plain dev bind mount is not a production backup strategy. --- ## 3. Redis ### 3.1 Connection Dev: `redis://redis:6379` (no password). Prod: `redis://:@redis:6379`. The compose command line is `redis-server --requirepass "$REDIS_PASSWORD"`. Inspect: ```bash docker exec -it nickapp-redis redis-cli -a "$REDIS_PASSWORD" > INFO server > DBSIZE > KEYS * # prod-unsafe on large datasets, use SCAN ``` ### 3.2 What we store - **Rate-limit counters** for `express-rate-limit` - **Session data** for refresh-token tracking and revocation lists - **Socket.IO adapter state** (when scaled horizontally — currently single-node) - **Application caches** (TTL'd keys for expensive aggregates) - **Idempotency keys** for webhook deduplication Key prefixes follow `::`. E.g. `payment:idem:`, `auth:refresh:`. ### 3.3 Persistence Redis 8 defaults to **RDB snapshots** + optional **AOF**. Our compose uses the default config: - RDB snapshot triggers: `save 3600 1`, `save 300 100`, `save 60 10000`. - AOF is **disabled** by default. - RDB file lives at `/data/dump.rdb` inside the `redis_data` volume. **To enable AOF** for stronger durability, override the command in `docker-compose.production.yml`: ```yaml redis: command: ["sh","-lc","redis-server --requirepass \"$${REDIS_PASSWORD}\" --appendonly yes --appendfsync everysec"] ``` `appendfsync everysec` is the common compromise: at most 1 second of writes lost on crash, with negligible perf impact. ### 3.4 Eviction policy Default is `noeviction` — Redis refuses writes when memory is full. For our use (caches that can be regenerated), set: ```bash docker exec nickapp-redis redis-cli -a "$REDIS_PASSWORD" \ CONFIG SET maxmemory 256mb docker exec nickapp-redis redis-cli -a "$REDIS_PASSWORD" \ CONFIG SET maxmemory-policy allkeys-lru ``` Persist by adding to a custom `redis.conf` mounted at `/usr/local/etc/redis/redis.conf` (then change the compose `command:` to `["redis-server","/usr/local/etc/redis/redis.conf","--requirepass",...]`). ### 3.5 Backup Redis backups are usually unnecessary (the data is regeneratable) but still cheap: ```bash # Snapshot now docker exec nickapp-redis redis-cli -a "$REDIS_PASSWORD" BGSAVE docker cp nickapp-redis:/data/dump.rdb ./backups/redis-$(date +%F).rdb ``` `BGSAVE` is non-blocking (forks). For AOF, copy `/data/appendonly.aof` too. ### 3.6 Cache flush When deploying breaking changes to cached schemas: ```bash # Flush everything (DEV ONLY) docker exec nickapp-redis redis-cli -a "$REDIS_PASSWORD" FLUSHALL # Targeted (safer) docker exec nickapp-redis redis-cli -a "$REDIS_PASSWORD" \ --scan --pattern 'payment:idem:*' | \ xargs -L 1 docker exec nickapp-redis redis-cli -a "$REDIS_PASSWORD" DEL ``` > [!warning] `FLUSHALL` will sign out every user with an active refresh token and reset every rate-limit counter. Avoid in production unless that is what you want. ### 3.7 Monitoring ```bash docker exec nickapp-redis redis-cli -a "$REDIS_PASSWORD" INFO stats docker exec nickapp-redis redis-cli -a "$REDIS_PASSWORD" INFO memory docker exec nickapp-redis redis-cli -a "$REDIS_PASSWORD" SLOWLOG GET 10 ``` Watch `evicted_keys`, `keyspace_misses`, `rejected_connections` — see [[Monitoring]] for thresholds. --- ## 4. Maintenance windows For both DBs, schedule a window when: - Bumping major version (Mongo 8 → 9, Redis 8 → 9) - Restoring from backup - Running a destructive migration Suggested checklist: 1. Announce in #ops Slack / status page. 2. Trigger `mongodump` (see [[Backup & Recovery]]). 3. Stop the backend container so writes stop: `docker compose stop nickapp-backend`. 4. Perform the operation. 5. Restart backend: `docker compose start nickapp-backend`. 6. Verify health: `curl https://amn.gg/api/health`. 7. Close window. --- ## 5. Cross-links - [[Backup & Recovery]] — formal backup/restore procedures, RTO/RPO targets, offsite storage. - [[Monitoring]] — what metrics to watch (slow queries, evictions, replication lag). - [[Incident Response]] — runbooks for "MongoDB unreachable" and "Redis unreachable". - [[Data Models]] — schema details for every collection.