Files
nick-doc/mongo-to-pg-migration-prd.md
Siavash Sameni a5d71bcc05 docs: sync documentation with latest codebase state
- Update Activity Log with 108 missing commits (48 backend + 60 frontend)
- Update version references: backend v2.8.79, frontend v2.8.94
- Update migration count: 18 migrations (0000-0017)
- Update Telegram Mini App Flow to v2.8.94
- Update Payment Flow - Scanner to 2026-06-05
- Update all architectural and database references

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
2026-06-05 07:34:49 +04:00

26 KiB
Raw Blame History

Status Update — 2026-06-03

Backend version: v2.8.79 Updated: 2026-06-03

Infrastructure Milestones Reached

Milestone Status
Drizzle migrations landed 17 migrations (00000017)
Drizzle schema coverage All 25+ entities have schemas
Dual-write repositories All entities covered
PG-only boot (MONGO_CONNECT_MODE=never) Confirmed working, seeds pass
Fresh DB path (drizzle-kit migrate + seed) Verified end-to-end
Read cutover Not yet — most domains still read from Mongo
Chat normalization Blocker for Chat read cutover (JSONB shim remains)
Production backfill Not yet executed — runbook at src/db/BACKFILL_RUNBOOK.md

Current State Summary

The migration has reached write-parity: every entity is dual-written to Postgres, the schema is complete, and the system can boot and seed against a fresh PG-only database. The remaining work is on the read side: flipping each domain's reads from Mongo to Postgres, gated by backfill execution (human-gated) and Chat normalization (engineering blocker).

Critical path remains unchanged: Chat normalization (46 wks) must precede any Chat read cutover. All other Phase 1/2 domains can proceed to read cutover independently once their backfill is confirmed correct.

Next Steps (in order)

  1. Execute production backfill per src/db/BACKFILL_RUNBOOK.md (human-gated, per-domain)
  2. Enable shadow-reads per domain, validate row-count and field parity
  3. Flip read flags domain by domain (User → Category/Shop/Level → Payment → Dispute → …)
  4. Begin Chat normalization (child tables replacing JSONB shim)

PRD: Mongo→Postgres Migration — Escrow Backend

Status: In Progress — Foundation tasks complete, dual-write coverage 100% Date: 2026-06-02 (audit) / updated 2026-06-02 (9 tasks completed) Backend version: 2.8.79 on integrate-main-into-development Source: Automated audit via mongo-to-pg-migration-audit workflow (49 agents, 528 DB operations catalogued) Target: Full migration — 23 Mongoose models → Postgres + Drizzle ORM


Executive Summary

The escrow backend currently runs a hybrid Mongo + Postgres architecture with an in-progress migration. The audit found 23 Mongoose models, 528 catalogued DB operations, 96 relationships, 110+ .populate() sites, 14 aggregation pipelines, and 6 transaction sites that must be migrated. Migration infrastructure was ~80% scaffolded at audit time.

As of 2026-06-02, all 9 MIGRATION_TODO foundation tasks are complete:

  • All 9 DualWrite repos exist (3 missing repos implemented)
  • Migration pipeline established (migrations/ dir, drizzle-kit scripts)
  • FundsLedgerEntry immutability trigger SQL migration created
  • Dispute composite indexes added to Drizzle schema
  • TTL scheduled cleanup implemented and wired into app startup
  • Address dual schema reconciled (Drizzle authoritative)
  • Seed script audit complete
  • 21 new tests for DualWriteDisputeRepo

Remaining: Backfill execution (human-gated, 14 scripts exist), Chat normalization decision, env var cutover.

Recommendation: Incremental strangler pattern, phased over 3 stages with dual-write + shadow-read verification at each cutover. Total realistic estimate: 28 engineer-weeks for full migration (2433 range), or 17 engineer-weeks for money/relational core only.

Critical path blocker: The Chat domain's current JSONB document shim is non-scalable and must be normalized to child tables before production cutover (46 wks alone).


1. Scope & Inventory

1.1 Collections (23)

Collection Complexity Migration Status Phase
User High (30+ fields, embedded arrays) AuthStore dual-write behind flag 1
PurchaseRequest Critical (7 child tables, 40+ fields, 13-value enum) Drizzle schema exists, no backfill 1
SellerOffer Medium (2 embedded subdocs) Drizzle schema exists, no backfill 1
Payment Critical (3 polymorphic Mixed FKs) Drizzle + DualWrite repos done 1
FundsLedgerEntry Critical (append-only, 3 polymorphic FKs) In DrizzlePaymentRepo 1
Dispute High (4 embedded arrays, pre-save hook) DualWrite done, composite indexes added, 21 tests 1
Category Low (self-referential FK) Dual-wired in production 1
ShopSettings Low Dual-wired in production 1
LevelConfig Low Dual-wired in production 1
ConfigSetting Low (key-value) Via configStore adapter 1
Address Low Schema reconciled, ensurePostgresAddressSchema → stub, IAddress fixed 2
Review Low Dual-write + backfill done 2
BlogPost Low Full dual-write done 2
PointTransaction Low (append-only) Dual-write done 2
Notification Medium (TTL index) Dual-write done, TTL DataCleanupService implemented 2
TrezorAccount Medium (child table) DualWrite repo done, factory wired 2
RequestTemplate Medium (6 embedded arrays) Drizzle schema exists, no backfill 2
DerivedDestination Medium (polymorphic sellerId/sellerOfferId) DualWrite repo done, factory wired 2
Chat Highest (4-level nesting, JSONB shim) Drizzle shim exists, needs normalization 3
TelegramSession Low (TTL) Dual-write in authStore, TTL DataCleanupService implemented 3
TelegramLink Low Dual-write deployed in production 3
TempVerification Low (TTL) PgTempVerification behind flag, TTL DataCleanupService implemented 3
ConfigSettingHistory Low (audit trail) Raw SQL table exists, Drizzle schema exists 1

1.2 Service Domains (19)

Domain Ops Models Touched Migration State
auth 47+ User, TempVerification, TelegramLink, TelegramSession Behind AUTH_STORE flag
marketplace 60+ PurchaseRequest, SellerOffer, Category, RequestTemplate, ShopSettings Partial Drizzle ORM paths
payment 30 Payment, FundsLedgerEntry Full dual-write
points 48 User, PointTransaction, LevelConfig Triple parallel repos
chat 10 Chat JSONB shim only
dispute 22 Dispute, Chat DualWrite missing
notification 10 Notification Dual-write done
delivery 7 PurchaseRequest (via marketplace repo) Via repo seam
blog 12 BlogPost Dual-write done
address 11 Address Dual-write partially wired
trezor 3+ TrezorAccount DualWrite missing
telegram 19 6 collections Mixed read paths
user 47+ User Via authStore
admin 5+ User, various Partial migration
blockchain 3 Payment (via repo) Via repo abstraction
email 0 None Pure side-effect layer
file 1 User (avatar URL) Filesystem + single write
health 0 None Connectivity probe only
redis 0 None Redis-only, no migration needed

1.3 Cross-Cutting Inventory

Concern Count Detail
.populate() call sites 110+ Across 7 files, User is dominant target (20+ FK paths)
.aggregate() pipelines 14 13 are simple $match+$group; 1 complex with 2× $lookup
Transaction sites 6 4 Mongoose + 2 PG SERIALIZABLE
$inc atomic update sites 6 Points and referral flows
Mixed/polymorphic ID fields 11 Across Payment, FundsLedgerEntry, DerivedDestination
TTL indexes 3 Notification (90d), TempVerification, TelegramSession
Mongoose pre/post hooks 4 models Address, Dispute, BlogPost, PurchaseRequest
Backfill scripts written 8 Per-store, variable maturity
DualWrite repos implemented 6 Payment, Marketplace, Points, User, Blog, Notification
Drizzle repos implemented 11 Covering all Phase 1 models + some Phase 2

2. Strategy Recommendation

Option A — Big-Bang

Cut all 23 collections in a single deployment. Rejected — unacceptable risk, no rollback plan, Chat shim would go live unscalable, TTL replacements not in place.

Per-domain dual-write → backfill → shadow-read → PG cutover → retire Mongo mirror. The codebase is already architected for this with 6 DualWrite repos, repository factory (425-line factory.ts), and AUTH_STORE=postgres proving the pattern works.

Option C — Hybrid (Mongo for documents, PG for relational) ⚠️

Keep Chat/Notification/BlogPost/TelegramSession/TempVerification in Mongo permanently. Pragmatic stepping stone but strands the Drizzle investment already made for those collections.

Decision

Go with Option B (strangler), sequenced as Option C's subset first. Phase 1 migrates the money/relational core where PG adds the most value (atomicity, FK constraints, money safety). Phase 2 migrates content/engagement. Phase 3 tackles document-shaped collections, with Chat normalization as the gating deliverable.


3. Phased Migration Plan

Phase 0: Foundation — 23 eng-wks

# Deliverable Detail
P0.1 Migration pipeline npm run db:migrate + npm run db:generate via drizzle-kit
P0.2 id_map utility Cached (LRU) resolveLegacyId/resolveUuid, warm on startup
P0.3 Gap reconciliation Script reads pg_dualwrite_gaps, retries PG writes, alerts on persistent failure
P0.4 Monitoring Row-count diff metrics (Mongo vs PG per collection), shadow-read mismatch alerts
P0.5 CI verify step db:verify connects both DBs, confirms dual-write mirrors alive, detects schema drift
P0.6 Rollback drill Documented procedure: flip feature flag → Mongo, confirm reads/writes, reconcile gaps

Phase 1: Money + Marketplace Core — 1416 eng-wks

Order: User → Category → ShopSettings + LevelConfig (parallel) → PurchaseRequest → SellerOffer → Payment + FundsLedgerEntry (atomic pair) → Dispute

# Collection Schema Repo DualWrite Backfill Action Needed
1.1 User Done Done Done Done Most mature — cutover first
1.2 Category Done Done Done Done Self-referential FK, already dual-wired
1.3 ShopSettings Done Done Done Done Clean model, already dual-wired
1.4 LevelConfig Done Done Done Done Already dual-wired
1.5 ConfigSetting Done Via adapter Via adapter Done Adapter pattern, no formal repo
1.6 PurchaseRequest Done Done Done NEEDED 7 child tables, 40+ fields, highest effort
1.7 SellerOffer Done Done Done NEEDED Simple model, many FKs
1.8 Dispute Done Done DONE NEEDED Composite indexes added, pre-save hook confirmed in Drizzle repo
1.9 Payment Done Done Done NEEDED Most money-critical, SERIALIZABLE transactions
1.10 FundsLedgerEntry Done Done Done NEEDED Immutability trigger SQL migration created

Phase 2: Content + Engagement — 45 eng-wks

# Collection Effort Key Action
2.1 Address Low Reconciled: Drizzle authoritative, ensurePostgresAddressSchema → stub, IAddress type fixed
2.2 Review Low Dual-write + backfill done; polymorphic subjectId handled without FK
2.3 BlogPost Low Full dual-write done; replicate slug generation hook
2.4 PointTransaction Low Dual-write done; sparse index → partial index
2.5 Notification Medium Dual-write done, TTL DataCleanupService implemented
2.6 TrezorAccount Medium DualWrite repo done, factory wired; child table handled by Drizzle repo
2.7 RequestTemplate Medium Decide child tables vs JSONB for 6 embedded arrays
2.8 DerivedDestination Medium DualWrite repo done, factory wired; discriminator handled by Drizzle repo

Phase 3: Document-Shaped Collections — 58 eng-wks

# Collection Effort Key Action
3.1 Chat High (46 wks) Normalize JSONB shim → chat_messages + chat_participants child tables; rewrite all ChatService methods; migrate existing data; perf test with production volumes
3.2 TelegramSession Low integerbigint risk flagged, TTL DataCleanupService implemented
3.3 TelegramLink Low Already dual-wired in production
3.4 TempVerification Low Behind feature flag, TTL DataCleanupService implemented

End state: MONGO_CONNECT_MODE=auto skips Mongo entirely. Mongoose driver removed from package.json.


4. Data Modeling Decisions

4.1 Polymorphic Mixed IDs

Rule: Three-column discriminator pattern for ALL Schema.Types.Mixed fields that hold ObjectId | string.

field_ref_kind     ref_kind ENUM('entity','template')  NOT NULL
field_id           UUID                                 NULL (FK when kind='entity')
field_external_ref TEXT                                 NULL (when kind='template')
CHECK constraint: (kind='entity' AND id IS NOT NULL AND external_ref IS NULL)
                OR (kind='template' AND id IS NULL AND external_ref IS NOT NULL)

Applicable to: Payment (purchaseRequestId, sellerOfferId, sellerId), FundsLedgerEntry (purchaseRequestId, paymentId, createdBy), DerivedDestination (sellerId, sellerOfferId), Dispute (chatId — schema gap, must add).

4.2 Interface-Level ObjectId | string

Rule: Treat as type-system bugs, NOT schema requirements. Resolve the string to UUID in the service layer before writing to PG. Found in ~10 interface fields (PurchaseRequest.categoryId, SellerOffer.purchaseRequestId/sellerId, RequestTemplate.sellerId/categoryId).

4.3 Embedded Arrays: Child Tables vs JSONB

Child tables (preferred):

  • PurchaseRequest.specifications[], deliveryAttempts[], preferredSellers[]
  • Chat.messages[] (must be child table), Chat.participants[]
  • TrezorAccount.addresses[] (already a child table)

JSONB (acceptable):

  • Dispute.evidence[], messages[], timeline[], resolution
  • Payment.blockchain, metadata
  • BlogPost.author, videos, seo
  • PurchaseRequest.deliveryInfo, serviceInfo (partial: extract flat fields to columns)

4.4 Money Numerics

Rule: numeric(38,18) for all amounts. numeric(78,0) for on-chain uint256 balances.

4.5 TTL Replacement

Rule: Application-level DataCleanupService with scheduled DELETE queries. Do NOT use pg_cron (extension dependency, may not be available on managed Postgres).

Collection Frequency Query
Notification Hourly DELETE WHERE created_at < NOW() - INTERVAL '90 days'
TempVerification Every 5 min DELETE WHERE expires_at < NOW()
TelegramSession Every 1 min DELETE WHERE expires_at < NOW()

5. Risk Register

# Risk L I Mitigation
R1 Data loss during backfill — concurrent writes cause gaps/doubles M H ON CONFLICT DO NOTHING on legacy_object_id; backfill from snapshot; replay gaps
R2 Polymorphic ID ambiguity — string that looks like UUID but is external ref L M Discriminator pattern makes ref_kind explicit; manual review of UUID-format strings
R3 Atomicity gap in dual-write — PG and Mongo diverge H H Gap table reconciliation every 515min; money: PG first, Mongo best-effort
R4 populate() → N+1 — 110+ sites convert to individual SELECTs H M Use Drizzle relations with JOINs; monitor query count per request post-cutover
R5 ObjectId→UUID remapping broken — id_map misses rows M H Startup verification: count Mongo vs PG per table; alert on difference
R6 Downtime during cutover — PG not ready, feature flag flips L M Canary deploy; Mongo mirror keeps writing; cutover during low-traffic
R7 Mongoose hooks not replicated — unchecked data in PG M M Address primary enforcement (reconciled to Drizzle schema + partial unique index), Dispute timeline push (confirmed in DrizzleDisputeRepo.create), BlogPost slug gen still needs PG equivalent
R8 Aggregation SQL mismatch — pipeline → SQL gives different results M H Test all 14 pipelines against identical staging data; 12 are simple GROUP BYs
R9 Chat JSONB scalability collapse — loads entire messages[] into memory H H BLOCKS Phase 3 — normalize to child tables before cutover
R10 FundsLedgerEntry updatable in PG L C MITIGATED — 0001_funds_ledger_immutable_trigger.sql migration created with BEFORE UPDATE + BEFORE DELETE triggers rejecting all modifications
R11 Dual PG schema definitions L L MITIGATED — Address schema reconciled (Drizzle authoritative), ensurePostgresAddressSchema() reduced to no-op stub. BlogPost, PointTransaction, ConfigSettingHistory still have dual definitions to check
R12 Polymorphic id resolution at read time — UUIDs must match id_map M L LRU-cached id_map (10k entries), warmed on startup
R13 Seed scripts bypass repo layer — direct Mongoose calls L M Update seed scripts per phase to go through PG path

6. Verification & Cutover Requirements

Per-Collection Cutover Checklist

  1. Drizzle schema frozen and reviewed
  2. Repository implemented with all hooks replicated
  3. Backfill script run against staging snapshot, row counts verified
  4. Dual-write active, pg_dualwrite_gaps < 10 rows
  5. Shadow-read enabled (7 days money, 3 days content)
  6. Zero checksum mismatches in last 48 hours
  7. PG p95 latency < 1.5× Mongo p95 latency
  8. Rollback drill passed: flip to Mongo, no data loss, gaps reconciled
  9. PG-read cutover → monitor 48h → retire Mongo mirror

Verification Layers

Layer Frequency Method
Row count parity Hourly Mongo.countDocuments vs PG SELECT count per collection
Checksum parity Daily Financial reconciliation by bucket (provider, status, date)
Shadow-read parity Continuous MD5 hash of sorted result set; PagerDuty on mismatch

7. Effort & Timeline

Team: 2 senior backend engineers + 1 staff engineer (50% architecture review)

Scenario Low Realistic High
Money core only (Phase 0+1) 14 wks 17 wks 19 wks
Full migration (Phases 03) 24 wks 28 wks 33 wks
Full with Chat JSONB kept 19 wks 22 wks 27 wks

Uncertainty buffer: +25%. Chat workstream: +50% (unknown production data volume).

Gantt Summary

Phase 0: Foundation          ██░░░░░░░░░░░░░░░░░░░░░░░░░░  2-3 wks
Phase 1a: User+Cat+Config    ░░██████░░░░░░░░░░░░░░░░░░░░  3-4 wks
Phase 1b: PR+SellOffer       ░░░░░░████████░░░░░░░░░░░░░░  4-5 wks
Phase 1c: Dispute+Pay+Ledger ░░░░░░░░░░░░████████░░░░░░░░  4-5 wks
Phase 1 QA + soak            ░░░░░░░░░░░░░░░░░░░░██░░░░░░  1-2 wks
Phase 2: Content+Engagement  ░░░░░░░░░░░░░░░░░░░░░░████░░  4-5 wks
Phase 3a: Chat normalization ░░░░░░░░░░░░░░░░░░░░░░░░░░██  4-6 wks  ← Critical path
Phase 3b: Sessions+Temp      ░░░░░░░░░░░░░░░░░░░░░░░░░░██  1-2 wks  (parallel with 3a)
Phase 3c: Mongo decommission ░░░░░░░░░░░░░░░░░░░░░░░░░░░░  1 wk
E2E migration test           ░░░░░░░░░░░░░░░░░░░░░░░░░░░░  2-3 wks

8. Tooling & Approach

ORM: Drizzle (confirmed)

Already v0.44.1 in package.json. 25 schema files, 11 repos, drizzle-kit v0.31.1. Team is familiar. No switch to Prisma/TypeORM/Kysely is justified.

Migration Pipeline

drizzle-kit generate → review + commit migration SQL → npm run db:migrate → CI verify no drift

Gap: Create backend/migrations/ directory, add npm run db:migrate script.

Dual-Write Mechanics

Money-core (Payment, FundsLedgerEntry): PG authoritative, Mongo best-effort mirror. Gap table catches failures.

Content (BlogPost, Review, Notification): Mongo authoritative, PG best-effort mirror. Failed PG writes retry from gap log.

Auth (User, TempVerification, TelegramLink, TelegramSession): AUTH_STORE=postgres routes cleanly. No dual-write.

Backfill Pattern

Stream Mongo docs in batches (500) → transform (resolve ObjectIds, apply discriminator,
flatten subdocs, serialize arrays) → INSERT ON CONFLICT DO NOTHING → replay gaps

8 scripts exist as templates. 5 new backfill scripts needed (PurchaseRequest, SellerOffer, Payment, FundsLedgerEntry, Dispute).


9. Immediate Action Items — Status

Completed 2026-06-02 (8 agents, 9 tasks)

# Action Status
A1 Reconcile Address dual schemaensurePostgresAddressSchema() → stub, IAddress fixed DONE
A2 Implement DualWriteDisputeRepo — PG-first pattern, 21 tests passing DONE
A3 Add FundsLedgerEntry immutability trigger — SQL migration created DONE
A4 Add missing Dispute indexes — composite status+priority, adminId+status DONE
A5 Create backend/migrations/ directorydb:generate/db:migrate/db:studio scripts DONE
A6 Implement DataCleanupService TTL — 3 purge methods + ttlCleanupJob.ts wired into app.ts DONE
A7 Decision: Chat normalization or JSONB shim PENDING
A8 Seed script audit — 7 seed + 8 utility scripts audited, 4 npm paths broken DONE
A9 Implement DualWriteTrezorAccountRepo — factory dual path wired DONE
A10 Implement DualWriteDerivedDestinationRepo — factory dual path wired DONE

Phase 1 Blockers Status

  1. Dispute DualWrite repo + pre-save hook DONE
  2. PurchaseRequest backfill script with 7-child-table transform — still needed
  3. Payment backfill script with financial reconciliation — still needed
  4. FundsLedgerEntry immutability trigger DONE

Quick Wins Status

  • Reconcile IAddress.addressType to include 'Other'DONE
  • Fix ObjectId | string type bugs at the controller layer (~10 interface fields)
  • Add CI db:verify step that fails on uncommitted schema changes
  • Index audit: verify every Mongo index has a PG equivalent with matching EXPLAIN ANALYZE

Seed Audit Key Findings (TASK 9)

  • All 7 seed scripts in src/seeds/ bypass the repo factory (use direct Mongoose)
  • 4 npm seed paths are BROKENseed:* scripts point to src/scripts/ but files live in src/seeds/
  • init-admin.ts runs on every startup, uses AuthUser with new User()/user.save() — first DB touch on boot
  • seedCategories.ts runs UNCONDITIONALLY on every startup, touches RequestTemplate + PurchaseRequest
  • No DrizzleCategoryRepo or DrizzleAddressRepo exist despite having Drizzle schemas — largest seed migration gaps
  • Money-critical tables (Payment, FundsLedgerEntry) are NOT touched by seed scripts — low risk

10. Appendices

A. Cross-Cutting Detail: populate() Hotspots

File Populate Count Dominant Join Notes
MongoMarketplaceRepo.ts 40+ User, Category, SellerOffer Heaviest file, all marketplace reads
MongoChatRepo.ts 20+ User (participants, messages.author) Nested inside embedded arrays
MongoDisputeRepo.ts 15+ User, Chat Polymorphic chatId populate
MongoPaymentRepo.ts 15+ User, PurchaseRequest, SellerOffer 3 Mixed fields populated without ref
MongoReviewRepo.ts 10+ User Polymorphic subjectId
MongoNotificationRepo.ts 5 User Straightforward 1:1
MongoPointsRepo.ts 5 User One silently omitted in Drizzle migration

B. Cross-Cutting Detail: Aggregation Pipelines

# File Model Complexity Stages
1 MongoMarketplaceRepo.ts:1158 User, ShopSettings High $match + 2× $lookup + $project + $sort
24 MongoMarketplaceRepo.ts PurchaseRequest, SellerOffer Low $match + $group
56 MongoPointsRepo.ts PointTransaction Low $match + $group + $sort
78 MongoPaymentRepo.ts Payment Low $match + $group
910 MongoDisputeRepo.ts Dispute Low $match + $group
1112 MongoNotificationRepo.ts Notification Low $match + $group + $count
13 MongoChatRepo.ts Chat Low $match + $unwind + $group
14 MongoBlogRepo.ts BlogPost Low $match + $group

C. Cross-Cutting Detail: Transaction & Atomicity Sites

Site Model Pattern Risk
MongoPointsRepo User, PointTransaction Mongoose session + transaction Dual-write gap if PG insert fails
MongoMarketplaceRepo (2 sites) PurchaseRequest, SellerOffer Mongoose session + transaction Same dual-write gap
DrizzleUserRepo User PG SERIALIZABLE Reference implementation
DrizzlePaymentRepo Payment, FundsLedgerEntry PG SERIALIZABLE + FOR UPDATE Best practice
PointsService.processReferralReward User, PointTransaction NO TRANSACTION ⚠️ addPoints + updateReferralStats separate
PaymentCoordinator AML fee FundsLedgerEntry NO TRANSACTION ⚠️ Explicit TODO on line 298
MongoMarketplaceRepo.processReferralReward User NO TRANSACTION ⚠️ referralStats outside session

Generated from automated audit: 49 agents, 23 model analyses, 19 service catalogues, 6 cross-cutting inventories. Full raw output: ~/.claude/projects/.../tasks/wbq6syb3q.output