nick/nick-doc
3
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs: sync from backend 8fc2309 — M43/M44 missing FKs + H37 dispute enums

Browse Source
This commit is contained in:
Siavash Sameni
2026-06-07 07:16:02 +04:00
parent a2967ec594
commit 0bb60dbc98
24 changed files with 3428 additions and 906 deletions
Download Patch File Download Diff File Expand all files Collapse all files

251
08 - Operations/Database Operations.md
View File

@@ -5,16 +5,143 @@ tags: [operations]
# Database Operations
Day-to-day operations for stateful services: **MongoDB 8.x** (primary runtime data store), **PostgreSQL 18** (migration target and conditional oracle quote store), and **Redis 8** (cache, rate-limit counters, ephemeral session data).
> [!important] MongoDB Removed (2026-06-06 / v2.9.12) — PostgreSQL is the sole database. MongoDB operational procedures below are retained as historical reference.
Day-to-day operations for stateful services: **PostgreSQL** (sole runtime data store as of v2.9.12), and **Redis 8** (cache, rate-limit counters, ephemeral session data).
For schema details see [[Data Models]]. For backup procedures and disaster recovery see [[Backup & Recovery]].
---
## PostgreSQL Operations
### Connection
`PG_URL` env var is **required**. MongoDB env vars (`MONGO_URI`, `MONGODB_URI`, `MONGO_CONNECT_MODE`) are obsolete and ignored.
| Env | Example DSN |
|-----|-------------|
| Dev | `postgres://amanat:<password>@postgres:5432/amanat_dev` |
| Prod | `postgres://amanat:<password>@postgres:5432/amanat` |
Connect from a shell:
```bash
docker exec -it amanat-postgres psql -U amanat -d amanat_dev
```
### Run migrations
```bash
cd backend && npx drizzle-kit migrate
```
19 migrations have landed (0000–0019), covering 32 tables. Application startup does **not** apply migrations automatically — run them explicitly before starting the backend after a version upgrade.
### Schema files
```
backend/src/db/schema/*.ts
```
Each file declares one or more Drizzle table definitions. Migrations in `backend/drizzle/` are generated from these schema files via `npx drizzle-kit generate`.
### Repositories
```
backend/src/db/repositories/drizzle/Drizzle*.ts
```
All domain repositories are Drizzle-backed. The repository factory returns Drizzle repos exclusively; there is no runtime fallback to MongoDB.
Key facts:
- IDs are PostgreSQL UUIDs (`.id` string field), not MongoDB ObjectIds
- `User._id` is kept as `legacy_object_id` column for backwards-compat; marketplace FKs use `user.pgId` (UUID)
- Chat is stored in the `chats` table with `messages`/`participants` as JSONB arrays
- `PaymentDTO.amount` is a decimal string
- `PurchaseRequest` does **not** have a top-level `paymentId` field
### Docker volume layout
```yaml
postgres:
image: postgres:18-alpine
environment:
POSTGRES_DB: amanat_dev
POSTGRES_USER: amanat
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
volumes:
- /var/data/escrowDev/postgres_data:/var/lib/postgresql
```
Mount at `/var/lib/postgresql` (not `/var/lib/postgresql/data`) — Postgres 18 stores data under a version-specific subdirectory.
For a disposable dev reset:
```bash
docker rm -f amanat-postgres 2>/dev/null || true
rm -rf /var/data/escrowDev/postgres_data
mkdir -p /var/data/escrowDev/postgres_data
```
### Backup
Standard PostgreSQL tooling:
```bash
docker exec amanat-postgres pg_dump -U amanat -d amanat_dev --format=custom \
> backups/amanat_dev_pg_$(date +%F).dump
```
Restore:
```bash
docker exec -i amanat-postgres pg_restore -U amanat -d amanat_dev --clean \
< backups/amanat_dev_pg_2026-06-06.dump
```
For production use managed backups or WAL archiving/PITR. See [[Backup & Recovery]].
### Seeding
Seeds are Postgres-only, store-aware, and idempotent. Run against a running backend container:
```bash
docker exec -it nickapp-backend node -e "require('./dist/seeds/seedCategories.js')"
docker exec -it nickapp-backend node -e "require('./dist/seeds/seedLevels.js')"
```
> [!warning] **Never** run `seed:all` or `seed:users` against production. These are destructive.
### Common admin queries
```sql
-- Row counts
SELECT schemaname, relname, n_live_tup
FROM pg_stat_user_tables ORDER BY n_live_tup DESC;
-- Active connections
SELECT count(*), state FROM pg_stat_activity GROUP BY state;
-- Slow queries (requires pg_stat_statements)
SELECT query, mean_exec_time, calls
FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;
-- Table sizes
SELECT relname, pg_size_pretty(pg_total_relation_size(relid))
FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC;
```
---
## 1. MongoDB
> [!note] Historical — MongoDB has been removed. The content below is retained as a reference for data archaeology, incident retrospectives, or backfill tooling. Do not use these procedures against the live application.
### 1.1 Connection
> [!note] Historical — MongoDB has been removed.
| Env | URI in compose | Auth |
|-----|---------------|------|
| Dev | `mongodb://mongodb:27017` | none |
@@ -44,6 +171,8 @@ docker exec -it nickapp-mongodb mongosh \
### 1.2 Init scripts (`mongo-init/`)
> [!note] Historical — MongoDB has been removed.
The production compose bind-mounts `./mongo-init` into `/docker-entrypoint-initdb.d`. Mongo runs `*.js` and `*.sh` from this folder **only on a fresh datadir** (first boot of a new volume). Use this to:
- Create application users (`db.createUser({...})`)
@@ -64,7 +193,9 @@ db.createUser({
### 1.3 Indexes
Indexes are declared in Mongoose schemas under `backend/src/models/`. The app calls `Model.createIndexes()` on connection (via the model's `syncIndexes`/`ensureIndexes` lifecycle). Highlights:
> [!note] Historical — MongoDB has been removed. Indexes are now declared in Drizzle schema files under `backend/src/db/schema/`.
Indexes were declared in Mongoose schemas under `backend/src/models/`. The app called `Model.createIndexes()` on connection. Highlights:
| Collection | Key indexes |
|------------|-------------|
@@ -77,30 +208,16 @@ Indexes are declared in Mongoose schemas under `backend/src/models/`. The app ca
| `notifications` | `userId` + `read`, `createdAt` |
| `tempverifications` | TTL on `expiresAt` (auto-deletes expired OTPs) |
To verify a specific collection:
```js
db.payments.getIndexes()
```
To add a new index without code-gen — preferred path is to declare it in the Mongoose schema and ship a deploy. For emergency hotfixes:
```js
db.payments.createIndex({ providerPaymentId: 1 }, { unique: true, sparse: true });
```
### 1.4 TTL indexes
Currently used on `tempverifications.expiresAt` (5-minute auto-purge of email OTPs / passkey challenges). Mongo's TTL monitor runs every 60 seconds — purge isn't immediate.
> [!note] Historical — MongoDB has been removed.
If you add more TTL indexes:
```js
db.notifications.createIndex({ createdAt: 1 }, { expireAfterSeconds: 60 * 60 * 24 * 90 }); // 90 days
```
Used on `tempverifications.expiresAt` (5-minute auto-purge of email OTPs / passkey challenges). Mongo's TTL monitor ran every 60 seconds.
### 1.5 Backup with `mongodump`
> [!note] Historical — MongoDB has been removed.
```bash
# Connect into the container, dump locally, copy out
docker exec nickapp-mongodb sh -c \
@@ -117,6 +234,8 @@ For full details (retention, RTO/RPO, offsite copies) see [[Backup & Recovery]].
### 1.6 Restore
> [!note] Historical — MongoDB has been removed.
```bash
# Restore an archive to an empty database
docker exec -i nickapp-mongodb \
@@ -130,21 +249,17 @@ docker exec -i nickapp-mongodb \
### 1.7 Migrations
There is no formal migration framework. Two patterns are used:
> [!note] Historical — MongoDB has been removed. Drizzle migrations are now used exclusively (`npx drizzle-kit migrate`).
- **Mongoose schema changes** are forward-compatible (new optional fields default to `undefined`). Older documents will still load.
- **Data backfills** are one-shot scripts in `backend/src/scripts/` (e.g. `migrateUserPoints.ts`, `fix-transaction-hashes.js`, `fix-dispute-sellers.js`).
There was no formal migration framework. Two patterns were used:
Pattern for a new migration:
1. Add a `src/seeds/migrate<Thing>.ts` script that is idempotent (use `$exists: false` guards).
2. Run on staging, confirm.
3. Take a backup ([[Backup & Recovery]]).
4. Run in production: `docker exec -it nickapp-backend node dist/seeds/migrate<Thing>.js`.
5. Commit the script (it serves as a record of what changed).
- **Mongoose schema changes** were forward-compatible (new optional fields default to `undefined`). Older documents would still load.
- **Data backfills** were one-shot scripts in `backend/src/scripts/` (e.g. `migrateUserPoints.ts`, `fix-transaction-hashes.js`, `fix-dispute-sellers.js`).
### 1.8 Common admin queries
> [!note] Historical — MongoDB has been removed.
```js
// Count by collection
db.users.countDocuments({ role: 'buyer' })
@@ -162,69 +277,41 @@ db.serverStatus().locks
### 1.9 Seeding production safely
Seed scripts are designed to be idempotent for **categories** but **destructive** for users/addresses. Don't run `seed:all` in production.
> [!note] Historical — MongoDB has been removed. Seeds are now Postgres-only and idempotent; see the PostgreSQL Operations section above.
Safe in production:
```bash
docker exec -it nickapp-backend node dist/seeds/seedCategories.js
docker exec -it nickapp-backend node dist/seeds/seedLevels.js
```
Optional auto-seed on startup: set `AUTO_SEED_ON_START=true` in `.env`. The bootstrap code only seeds when no non-admin users exist — safe to leave on.
Seed scripts were designed to be idempotent for **categories** but **destructive** for users/addresses.
> [!warning] **Never** run `seed:all` or `seed:users` against production. They drop the existing `users` and `addresses` collections.
---
## 2. PostgreSQL 18
## 2. PostgreSQL 18 (legacy section — superseded by PostgreSQL Operations above)
> [!note] Historical — This section documented the partial migration era. PostgreSQL is now the sole database; see the PostgreSQL Operations section at the top of this document.
### 2.1 Runtime role
Postgres is present in the current dev/integration stack, but MongoDB remains the primary runtime store. Use Postgres for:
~~Postgres is present in the current dev/integration stack, but MongoDB remains the primary runtime store.~~
- Drizzle migrations and schema verification.
- Mongo → Postgres backfill and reconciliation work.
- `payment_quotes` when `ORACLE_QUOTING_ENABLED=true` and a PG parent payment row exists.
Do **not** treat Postgres as the authoritative app database until the relevant domain has been wired through repository interfaces, backfilled, shadow-read, and cut over. See [[Postgres Runtime Cutover Status]].
As of v2.9.12, PostgreSQL is the **only** runtime store. All domain repositories use Drizzle. There is no dual-write mode.
### 2.2 Docker volume layout for Postgres 18
Postgres 18 Docker images expect the mount at `/var/lib/postgresql`, not directly at `/var/lib/postgresql/data`, because the image stores data under a major-version-specific directory such as `/var/lib/postgresql/18/docker`.
```yaml
postgres:
image: postgres:18-alpine
environment:
POSTGRES_DB: amanat_dev
POSTGRES_USER: amanat
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
volumes:
- /var/data/escrowDev/postgres_data:/var/lib/postgresql
```
For a disposable dev reset:
```bash
docker rm -f amanat-postgres 2>/dev/null || true
rm -rf /var/data/escrowDev/postgres_data
mkdir -p /var/data/escrowDev/postgres_data
```
See the Docker volume layout subsection in PostgreSQL Operations above.
### 2.3 Apply migrations
Run migrations only after the database is healthy and the DSN points at the intended non-production target:
```bash
PG_URL=postgres://amanat:...@postgres:5432/amanat_dev npx drizzle-kit migrate
cd backend && npx drizzle-kit migrate
```
The backend image contains migrations through `0008`. Application startup does not apply them automatically.
19 migrations (0000–0019) covering 32 tables. See PostgreSQL Operations above.
### 2.4 Backfill and verification
Backfills use `MIGRATION_PG_URL`, not `PG_URL`, and the scripts enforce a host allowlist. Run dry-run and verification before any dual-write/PG read flip:
> [!note] Historical — Mongo→Postgres backfill tooling is no longer needed. The migration is complete.
Backfills used `MIGRATION_PG_URL` (not `PG_URL`) and enforced a host allowlist:
```bash
MIGRATION_MONGO_URL=mongodb://mongodb:27017/marketplace \
@@ -232,18 +319,9 @@ MIGRATION_PG_URL=postgres://amanat:...@postgres:5432/amanat_dev \
node dist/db/backfill/run-backfill.js --dry-run
```
Verify row counts/checksums and inspect `pg_dualwrite_gaps` before enabling any cutover flag.
### 2.5 Backup
For dev/staging:
```bash
docker exec amanat-postgres pg_dump -U amanat -d amanat_dev --format=custom \
> backups/amanat_dev_pg_$(date +%F).dump
```
Before production cutover, use managed backups or self-hosted WAL archiving/PITR. A plain dev bind mount is not a production backup strategy.
See the Backup subsection in PostgreSQL Operations above.
---
@@ -345,16 +423,16 @@ Watch `evicted_keys`, `keyspace_misses`, `rejected_connections` — see [[Monito
## 4. Maintenance windows
For both DBs, schedule a window when:
Schedule a window when:
- Bumping major version (Mongo 8 → 9, Redis 8 → 9)
- Bumping major version (PostgreSQL, Redis 8 → 9)
- Restoring from backup
- Running a destructive migration
Suggested checklist:
1. Announce in #ops Slack / status page.
2. Trigger `mongodump` (see [[Backup & Recovery]]).
2. Trigger `pg_dump` backup (see [[Backup & Recovery]]).
3. Stop the backend container so writes stop: `docker compose stop nickapp-backend`.
4. Perform the operation.
5. Restart backend: `docker compose start nickapp-backend`.
@@ -367,5 +445,6 @@ Suggested checklist:
- [[Backup & Recovery]] — formal backup/restore procedures, RTO/RPO targets, offsite storage.
- [[Monitoring]] — what metrics to watch (slow queries, evictions, replication lag).
- [[Incident Response]] — runbooks for "MongoDB unreachable" and "Redis unreachable".
- [[Data Models]] — schema details for every collection.
- [[Incident Response]] — runbooks for database unreachable scenarios.
- [[Data Models]] — schema details for every table.
- [[Postgres Runtime Cutover Status]] — migration history and current state.