nick/nick-doc
3
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs: sync documentation with latest codebase state

Browse Source 
- Update Activity Log with 108 missing commits (48 backend + 60 frontend)
- Update version references: backend v2.8.79, frontend v2.8.94
- Update migration count: 18 migrations (0000-0017)
- Update Telegram Mini App Flow to v2.8.94
- Update Payment Flow - Scanner to 2026-06-05
- Update all architectural and database references

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
This commit is contained in:
Siavash Sameni
2026-06-05 07:34:49 +04:00
parent 9dcdb420fc
commit a5d71bcc05
10 changed files with 490 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

375
MIGRATION_TODO.md Normal file
View File

@@ -0,0 +1,375 @@
# Mongo→Postgres Migration — Working TODO
**Last updated:** 2026-06-02 (all 9 tasks completed)
**Backend version:** 2.8.79 on branch `integrate-main-into-development`
**Repo root:** `/Users/manwe/CascadeProjects/escrow/backend`
## 2026-06-02 — All 9 tasks completed (AI-assisted)
**Changes (11 modified + 6 new files, v2.8.39 → v2.8.44):**
| Task | What | Status |
|---|---|---|
| TASK 1 | `migrations/` dir created, `db:generate`/`db:migrate`/`db:studio` scripts, drizzle.config.ts fixed | ✅ |
| TASK 2 | `0001_funds_ledger_immutable_trigger.sql` — UPDATE/DELETE rejection trigger | ✅ |
| TASK 3 | `disputes_status_priority_idx` + `disputes_admin_id_status_idx` composite indexes | ✅ |
| TASK 4 | `DualWriteDisputeRepo.ts` + factory `dual` path + 21 tests | ✅ |
| TASK 5 | `DualWriteTrezorAccountRepo.ts` + factory `dual` path (child table handled by Drizzle repo) | ✅ |
| TASK 6 | `DualWriteDerivedDestinationRepo.ts` + factory `dual` path (discriminator handled by Drizzle repo) | ✅ |
| TASK 7 | 3 TTL purge methods + `ttlCleanupJob.ts` scheduler wired into `app.ts` | ✅ |
| TASK 8 | Address schema reconciled (Drizzle authoritative), `ensurePostgresAddressSchema` → stub, `IAddress.addressType` fixed | ✅ |
| TASK 9 | Seed audit: 7 seed scripts + 8 utility scripts, ALL bypass repo factory; 4 npm paths broken | ✅ |
**All 9 DualWrite repos now exist.** All 3 missing repos (Dispute, TrezorAccount, DerivedDestination) implemented with PG-first write pattern.
**Remaining (human-gated):** Backfill execution against staging snapshot, Chat normalization decision, env var cutover in docker-compose.yml, smoke test verification.
---
---
## Project Context
The escrow backend (Node.js + TypeScript) is migrating from MongoDB/Mongoose to Postgres/Drizzle ORM using a **strangler-fig** pattern. The architecture has:
- `src/db/repositories/interfaces/I*.ts` — shared interfaces
- `src/db/repositories/mongo/Mongo*.ts` — current Mongoose implementations
- `src/db/repositories/drizzle/Drizzle*.ts` — Postgres/Drizzle implementations (11 exist)
- `src/db/repositories/dual/DualWrite*.ts` — fan-out wrappers that write to both (6 exist)
- `src/db/repositories/factory.ts` — resolves the active implementation per env var (e.g. `DISPUTE_STORE=postgres`)
- `src/db/schema/` — 25 Drizzle schema files
- `src/db/backfill/` — 14 one-shot backfill scripts (all exist, not all run against prod)
**Rules every change must follow:**
- Every backend product change requires a patch version bump (`package.json` + `package-lock.json`) AND the same version bump in `frontend/package.json`.
- Before pushing backend: run the relevant focused test suite + typecheck.
- After every backend push: append to `09 - Audits/Activity Log.md` in nick-doc repo, commit as `docs: sync from backend <sha> — <summary>`, push nick-doc.
- Do NOT commit unrelated dirty files in `nick-doc` or `deployment`.
- Smoke scripts live in `scripts/smoke/`. Admin smoke scripts need `ADMIN_TOKEN` from `POST /api/auth/login` with `admin@marketplace.com` / `Moji6364` against `https://dev.amn.gg`.
---
## Status Snapshot
| DualWrite Repo | Exists | Factory flag |
|---|---|---|
| DualWriteUserRepo | ✅ | `AUTH_STORE` |
| DualWriteMarketplaceRepo | ✅ | `MARKETPLACE_STORE` |
| DualWritePaymentRepo | ✅ | `PAYMENT_STORE` |
| DualWritePointsRepo | ✅ | `POINTS_STORE` |
| DualWriteNotificationRepo | ✅ | `NOTIFICATION_STORE` |
| DualWriteBlogRepo | ✅ | `BLOG_STORE` |
| **DualWriteDisputeRepo** | ✅ DONE 2026-06-02 | `DISPUTE_STORE` |
| **DualWriteTrezorAccountRepo** | ✅ DONE 2026-06-02 | `REPO_TREZOR` |
| **DualWriteDerivedDestinationRepo** | ✅ DONE 2026-06-02 | `REPO_DERIVED_DESTINATION` |
| Backfill script | Exists | Run against staging |
|---|---|---|
| backfill-users.ts | ✅ | Pending |
| backfill-categories.ts | ✅ | Pending |
| backfill-purchaseRequests.ts | ✅ | Pending |
| backfill-sellerOffers.ts | ✅ | Pending |
| backfill-payments.ts | ✅ | Pending |
| backfill-fundsLedger.ts | ✅ | Pending |
| backfill-derivedDestinations.ts | ✅ | Pending |
| backfill-requestTemplates.ts | ✅ | Pending |
| backfill-trezorAccounts.ts | ✅ | Pending |
| backfill-notifications.ts | ✅ | Pending |
| backfill-pointTransactions.ts | ✅ | Pending |
| Reviews, Blogs, etc. | ✅ Via scripts/ | Pending |
| Infrastructure | State |
|---|---|
| `backend/migrations/` directory | ✅ Created with .gitkeep + 0001_ trigger SQL |
| `npm run db:migrate` script | ✅ Added to package.json |
| `npm run db:generate` script | ✅ Added to package.json |
| FundsLedgerEntry immutability trigger | ✅ SQL migration + documented in schema |
| Dispute composite indexes | ✅ Added: status+priority, adminId+status |
| DataCleanupService TTL scheduled deletes | ✅ Implemented + wired in app.ts |
---
## Tasks (Priority Order)
---
### TASK 1 — Add migrations/ pipeline ✅ DONE
**Status:** Completed 2026-06-02
**What to do:**
1. Create `backend/migrations/` with a `.gitkeep`.
2. Add to `package.json` scripts:
```json
"db:generate": "drizzle-kit generate",
"db:migrate": "drizzle-kit migrate",
"db:studio": "drizzle-kit studio"
```
3. Confirm `drizzle.config.ts` (or `drizzle.config.js`) exists at backend root pointing to `src/db/schema/index.ts` and `migrations/` output dir. If it doesn't exist, create it:
```ts
import { defineConfig } from 'drizzle-kit';
export default defineConfig({
schema: './src/db/schema/index.ts',
out: './migrations',
dialect: 'postgresql',
dbCredentials: { url: process.env.DATABASE_URL! },
});
```
4. Run `npm run db:generate` — confirm it generates SQL without errors.
**Verify:** `npm run db:generate` exits 0. `migrations/` directory has at least one `.sql` file.
**No version bump needed** (tooling only, no runtime change).
---
### TASK 2 — Apply FundsLedgerEntry immutability trigger ✅ DONE
**Status:** Completed 2026-06-02
**What to do:**
The DDL is already documented as a comment in `src/db/schema/fundsLedgerEntry.ts` lines 188–200. Copy it to a standalone migration SQL file and apply it.
1. Create `migrations/0001_funds_ledger_immutable_trigger.sql`:
```sql
CREATE OR REPLACE FUNCTION funds_ledger_immutable_fn()
RETURNS TRIGGER AS $$
BEGIN
RAISE EXCEPTION 'funds_ledger_entries rows are immutable';
END;
$$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS funds_ledger_immutable_update ON funds_ledger_entries;
CREATE TRIGGER funds_ledger_immutable_update
BEFORE UPDATE ON funds_ledger_entries
FOR EACH ROW EXECUTE FUNCTION funds_ledger_immutable_fn();
DROP TRIGGER IF EXISTS funds_ledger_immutable_delete ON funds_ledger_entries;
CREATE TRIGGER funds_ledger_immutable_delete
BEFORE DELETE ON funds_ledger_entries
FOR EACH ROW EXECUTE FUNCTION funds_ledger_immutable_fn();
```
2. Add a startup/migration helper or document that a DBA must run this SQL before PG cutover for FundsLedgerEntry.
3. Add a test: `__tests__/funds-ledger-immutability.test.ts` — spin up a test PG, insert a row, attempt UPDATE, assert it throws.
**Verify:** Test passes. Manual `UPDATE funds_ledger_entries SET amount = 0 WHERE id = '<any>';` returns `ERROR: funds_ledger_entries rows are immutable`.
---
### TASK 3 — Add Dispute composite indexes ✅ DONE
**Status:** Completed 2026-06-02
**Current state:** Individual indexes on `status`, `priority`, `admin_id` exist (lines 125–127). Missing composite indexes that the Mongo version uses for admin dashboard queries.
**What to do:**
Add to the `dispute` table's `.extraConfig` block in `src/db/schema/dispute.ts`:
```ts
index('disputes_status_priority_idx').on(t.status, t.priority),
index('disputes_admin_id_status_idx').on(t.adminId, t.status),
```
Then regenerate migrations: `npm run db:generate`.
**Verify:** `npm run typecheck` clean. Generated migration SQL contains both new indexes.
---
### TASK 4 — Implement DualWriteDisputeRepo ✅ DONE
**Status:** Completed 2026-06-02
**Context:**
- `DrizzleDisputeRepo.ts` exists at 11.5K — Postgres implementation is done.
- `MongoDisputeRepo.ts` exists — Mongo implementation is done.
- Factory has `getDisputeRepo()` at line 360 but no DualWrite path.
**Pattern to follow:** Copy `DualWriteNotificationRepo.ts` (3K, simplest existing example) or `DualWriteBlogRepo.ts` as a structural template.
**What to do:**
1. Create `DualWriteDisputeRepo.ts` implementing `IDisputeRepo`. Every method:
- Writes to Drizzle (PG) first.
- If PG write succeeds, writes to Mongo best-effort (catch + log, never throw).
- If PG write fails, throw (PG is authoritative for Dispute once dual-write is enabled).
- Read operations: read from PG only (or from Mongo if `DISPUTE_STORE` is `mongo`).
2. Pre-save hook replication: Mongo's `Dispute` model has a `pre('save')` hook that pushes a timeline entry. The `DrizzleDisputeRepo` must replicate this in application code — check if it already does. If not, add it.
3. Update `factory.ts` `getDisputeRepo()` to return `DualWriteDisputeRepo` when mode is `dual`.
4. Add `DISPUTE_STORE=dual` as a valid value in `resolveMode`.
5. Write focused test: mock Mongo repo, assert PG writes happen first and Mongo writes happen second.
**Verify:**
```bash
npm run typecheck
npm test -- --runTestsByPath __tests__/dispute-dual-write.test.ts --runInBand
```
---
### TASK 5 — Implement DualWriteTrezorAccountRepo ✅ DONE
**Status:** Completed 2026-06-02
**Context:** `DrizzleTrezorAccountRepo.ts` exists (9.7K). `ITrezorAccountRepo.ts` exists (1.9K). No DualWrite wrapper.
**What to do:** Same pattern as Task 4 but for TrezorAccount. TrezorAccount has a child table `trezor_derived_addresses` — ensure the DualWrite repo propagates child record writes to both stores.
**Verify:** `npm run typecheck` clean.
---
### TASK 6 — Implement DualWriteDerivedDestinationRepo ✅ DONE
**Status:** Completed 2026-06-02
**Context:** `DrizzleDerivedDestinationRepo.ts` exists (11.5K). Complex polymorphic `sellerId`/`sellerOfferId` fields. Check `IDerivedDestinationRepo.ts` (2.5K) for the interface contract.
**What to do:** Same DualWrite pattern. Special care: the Drizzle schema uses a three-column discriminator for the polymorphic FK (`seller_ref_kind`, `seller_id`, `seller_external_ref`). The DualWrite repo must translate between the Mongo format and the Drizzle discriminator on writes.
**Verify:** `npm run typecheck` clean.
---
### TASK 7 — DataCleanupService: scheduled TTL deletes ✅ DONE
**Status:** Completed 2026-06-02
**Context:** TTL in Mongo is index-based (automatic). In Postgres, there are no TTL indexes — the PRD specifies application-level scheduled deletes via `DataCleanupService`.
**Required TTL schedules:**
| Collection | PG table | Frequency | Delete condition |
|---|---|---|---|
| Notification | `notifications` | Hourly | `created_at < NOW() - INTERVAL '90 days'` |
| TempVerification | `temp_verifications` | Every 5 min | `expires_at < NOW()` |
| TelegramSession | `telegram_sessions` | Every 1 min | `expires_at < NOW()` |
**What to do:**
1. Add three new static methods to `DataCleanupService`:
```ts
static async purgeExpiredTempVerifications(): Promise<number>
static async purgeExpiredTelegramSessions(): Promise<number>
static async purgeOldNotifications(): Promise<number>
```
Each method: check `isPostgresXxxEnabled()` (or check Postgres pool availability), run the DELETE query, return deleted row count. Guard with `isMongoAvailable()` for the Mongo fallback path.
2. Create `src/services/admin/ttlCleanupJob.ts` that exports a `startTtlCleanupScheduler()` function using `setInterval`:
```ts
setInterval(() => DataCleanupService.purgeExpiredTelegramSessions(), 60_000);
setInterval(() => DataCleanupService.purgeExpiredTempVerifications(), 5 * 60_000);
setInterval(() => DataCleanupService.purgeOldNotifications(), 60 * 60_000);
```
3. Call `startTtlCleanupScheduler()` from `src/app.ts` after the Postgres pool is ready (not on startup in test env — guard with `process.env.NODE_ENV !== 'test'`).
4. Add test: `__tests__/ttl-cleanup.test.ts` — mock Postgres pool, assert each purge method runs the correct DELETE query.
**Verify:**
```bash
npm test -- --runTestsByPath __tests__/ttl-cleanup.test.ts --runInBand
npm run typecheck
```
---
### TASK 8 — Reconcile Address dual schema ✅ DONE
**Status:** Completed 2026-06-02
**Context:** `addressStore.ts` has a `ensurePostgresAddressSchema()` function that creates the `addresses` table with raw SQL. `src/db/schema/address.ts` is the Drizzle schema. These two table definitions may have drifted (column names, types, indexes).
**What to do:**
1. Compare `ensurePostgresAddressSchema()` raw SQL (in `addressStore.ts`) with the Drizzle schema at `src/db/schema/address.ts` column by column.
2. Reconcile to a single source of truth: the Drizzle schema must be authoritative. If the raw SQL has columns the Drizzle schema lacks, add them to the Drizzle schema.
3. Remove `ensurePostgresAddressSchema()` once the Drizzle migration covers the same DDL. Alternatively, keep it as a noop that runs `db:migrate` instead.
4. After reconciling, run `npm run db:generate` and verify the migration is a no-op (or only adds missing columns/indexes).
5. Fix the type mismatch: `IAddress.addressType` in the interface may not include `'Other'` — check `src/db/repositories/interfaces/` and add it if missing.
**Verify:** `npm run typecheck` clean. `npm run db:generate` generates an empty migration (no drift).
---
### TASK 9 — Seed script audit ✅ DONE
**Status:** Completed 2026-06-02 (read-only audit, report below)
**What to do:**
1. Find all seed scripts: `rg -rn "Model.create\|\.insertMany\|\.save()" src/scripts src/seeds 2>/dev/null`
2. For each direct Mongoose model call in a seed script, route it through the repo factory instead: `getMarketplaceRepo().createPurchaseRequest(...)` etc.
3. If the repo doesn't have a seed-friendly bulk-insert method, add one or use the existing `create*` methods in a loop.
**Verify:** `npm run typecheck` clean.
---
## Not Started: Backfill Execution (Human-gated)
All 14 backfill scripts exist. They require `MIGRATION_MONGO_URL` and `MIGRATION_PG_URL` env vars pointing to a staging/prod DB snapshot. These are **out of scope for AI agents** — a human must coordinate a DB snapshot and run them. See `src/db/backfill/README.md`.
Order to run when ready:
1. `backfill-users.ts`
2. `backfill-categories.ts`
3. `backfill-sellerOffers.ts` (depends on users + categories)
4. `backfill-purchaseRequests.ts` (depends on users + categories + sellerOffers)
5. `backfill-payments.ts` (depends on purchaseRequests + users)
6. `backfill-fundsLedger.ts` (depends on payments)
7. `backfill-derivedDestinations.ts` (depends on purchaseRequests + sellerOffers)
8. `backfill-pointTransactions.ts`
9. `backfill-requestTemplates.ts`
10. `backfill-trezorAccounts.ts`
11. `backfill-notifications.ts`
12. Reviews, blogs, addresses via `src/scripts/backfill*.ts`
---
## Not Started: Chat Normalization (Architecture Decision Required)
Chat is the critical-path blocker for full Mongo decommission. The current Drizzle schema stores messages as JSONB — this does not scale.
Options:
- **Normalize to child tables** (`chat_messages`, `chat_participants`): 4–6 weeks, correct long-term
- **Keep JSONB shim**: safe for now, defer until Chat rewrites
This is a human architecture decision before any AI agent should touch `DrizzleChatRepo.ts`.
---
## Completion Criteria for "Mongo Optional" Runtime
Before setting `MONGO_CONNECT_MODE=auto` (skips Mongo if no Mongo-backed stores remain):
- [x] TASK 4 done: DualWriteDisputeRepo + `DISPUTE_STORE=postgres` in dev
- [x] TASK 7 done: TTL scheduler running in dev
- [x] TASK 2 done: FundsLedgerEntry trigger applied to dev DB
- [ ] All store env vars set to `postgres` in `deployment/docker-compose.yml`:
`AUTH_STORE`, `MARKETPLACE_STORE`, `PAYMENT_STORE`, `POINTS_STORE`, `NOTIFICATION_STORE`, `BLOG_STORE`, `DISPUTE_STORE`, `ADDRESS_STORE`, `REVIEW_STORE`, `LEVEL_CONFIG_STORE`, `SHOP_SETTINGS_STORE`, `CONFIG_STORE`
- [ ] `/api/health` returns `mongo: optional` not `mongo: required`
- [ ] All smoke scripts pass against dev with above env vars active
- [ ] All smoke scripts pass against dev with above env vars active
---
## Quick Command Reference
```bash
# Backend root
cd /Users/manwe/CascadeProjects/escrow/backend
# Typecheck
npm run typecheck
# Focused test suite (always run before pushing)
npm test -- --runTestsByPath \
__tests__/auth-store-pg-query.test.ts \
__tests__/user-dependencies-repo.test.ts \
__tests__/repository-factory-modes.test.ts \
__tests__/health-check-service.test.ts \
__tests__/marketplace-runtime-import-surface.test.ts \
--runInBand
# Mongo surface scan (should show 0 truly-blocking hits)
rg -n "countDocuments\(|deleteMany\(|findByIdAndDelete\(" \
src/services src/routes src/app.ts src/infrastructure src/db/repositories/factory.ts \
--glob '!**/*.test.ts' --glob '!src/services/marketplace/routes.ts'
# Get admin token for smoke scripts
TOKEN=$(curl -s -X POST https://dev.amn.gg/api/auth/login \
-H "Content-Type: application/json" \
-d '{"email":"admin@marketplace.com","password":"Moji6364"}' \
| python3 -c "import sys,json; d=json.load(sys.stdin); print(d['data']['tokens']['accessToken'])")
```