docs: sync from backend 6df113d — marketplace pg backfill
This commit is contained in:
Siavash Sameni
@@ -4,7 +4,7 @@ tags: [data-model, migration, mongodb, postgres, mongoose, helper]
|
||||
aliases: [Mongo to Postgres, DB Migration Guide, Postgres Migration]
|
||||
created: 2026-05-31
|
||||
source: backend/src (automated multi-agent scan)
|
||||
updated: 2026-05-31 for backend integrate-main-into-development@cab0719
|
||||
updated: 2026-06-01 for backend integrate-main-into-development@6df113d
|
||||
---
|
||||
|
||||
# MongoDB → PostgreSQL Migration Guide
|
||||
@@ -17,7 +17,7 @@ updated: 2026-05-31 for backend integrate-main-into-development@cab0719
|
||||
> **Execution plan:** see the companion [[MongoDB to PostgreSQL Migration Plan (Drizzle)]] for the phased, Drizzle-concrete plan (repository seam, `id_map`, dual-write, per-phase cutover runbook).
|
||||
|
||||
> [!warning] Current implementation delta
|
||||
> This guide started as a migration helper. Backend `integrate-main-into-development@cab0719` now contains the first Postgres implementation layer: Drizzle schemas/migrations through `0008`, `src/db/client.ts`, `id_map`, `pg_dualwrite_gaps`, repository implementations/factory, backfill/verify scripts, conditional `payment_quotes` persistence, and aligned purchase/template request budget validation. The broad service layer is still Mongoose-first and is **not** fully wired through those repositories. Use [[Postgres Runtime Cutover Status]] as the authoritative current-state snapshot.
|
||||
> This guide started as a migration helper. Backend `integrate-main-into-development@6df113d` now contains the first Postgres implementation layer: Drizzle schemas/migrations through `0008`, `src/db/client.ts`, `id_map`, `pg_dualwrite_gaps`, repository implementations/factory, backfill/verify scripts, conditional `payment_quotes` persistence, and aligned purchase/template request budget validation. Backend `2.8.13` also hardens the PurchaseRequest/SellerOffer backfill runner for marketplace-core dry-runs and selected-offer remapping. The broad service layer is still Mongoose-first and is **not** fully wired through those repositories. Use [[Postgres Runtime Cutover Status]] as the authoritative current-state snapshot.
|
||||
|
||||
> [!info] Scan coverage (2026-05-31)
|
||||
> - **23** Mongoose models (collections)
|
||||
|
||||
@@ -4,7 +4,7 @@ tags: [data-model, migration, postgres, drizzle, plan, runbook]
|
||||
aliases: [Drizzle Migration Plan, PG Migration Plan]
|
||||
created: 2026-05-31
|
||||
companion: "[[MongoDB to PostgreSQL Migration Guide]]"
|
||||
updated: 2026-06-01 for backend integrate-main-into-development@1757f1e backend 2.8.9
|
||||
updated: 2026-06-01 for backend integrate-main-into-development@6df113d backend 2.8.13
|
||||
---
|
||||
|
||||
# MongoDB → PostgreSQL Migration Plan (Drizzle)
|
||||
@@ -17,7 +17,7 @@ updated: 2026-06-01 for backend integrate-main-into-development@1757f1e backend
|
||||
> **Scope reminder:** partial migration (Phases 0–5) is the recommended stopping point — ≈16–28 engineer-weeks. Full migration of Chat/Notification/sessions is explicitly deferred.
|
||||
|
||||
> [!warning] Current implementation status
|
||||
> Backend `2.8.9` has started the runtime cutover with store-specific raw Postgres facades: auth-owned users/Telegram auth records behind `AUTH_STORE=postgres`, confirmation-threshold config/history behind `CONFIG_STORE=postgres`, user address CRUD behind `ADDRESS_STORE=postgres`, and the first marketplace/reference domains behind `CATEGORY_STORE=postgres`, `LEVEL_CONFIG_STORE=postgres`, `SHOP_SETTINGS_STORE=postgres`, and `REVIEW_STORE=postgres`. This checkout does **not** currently contain the old `src/db/` Drizzle repository layer described below, so treat the Drizzle sections as the target architecture/runbook, not a statement of current code. Mongo remains authoritative unless a per-store flag is explicitly flipped. See [[Postgres Runtime Cutover Status]].
|
||||
> Backend `2.8.13` has started the runtime cutover with store-specific raw Postgres facades: auth-owned users/Telegram auth records behind `AUTH_STORE=postgres`, confirmation-threshold config/history behind `CONFIG_STORE=postgres`, user address CRUD behind `ADDRESS_STORE=postgres`, and the first marketplace/reference domains behind `CATEGORY_STORE=postgres`, `LEVEL_CONFIG_STORE=postgres`, `SHOP_SETTINGS_STORE=postgres`, and `REVIEW_STORE=postgres`. It also contains the broader `src/db/` Drizzle schemas, repository implementations/factory, id-map bridge, and backfill runner described below, but the broad marketplace/payment/points services are still mostly not wired through that factory. Mongo remains authoritative unless a per-store flag is explicitly flipped. See [[Postgres Runtime Cutover Status]].
|
||||
|
||||
---
|
||||
|
||||
@@ -361,6 +361,7 @@ Same phases as the guide §2, here with Drizzle-concrete entry/exit gates. Each
|
||||
|
||||
### Phase 4 — Money core (6–10 wk) — *the point of the project*
|
||||
- `PurchaseRequest`, `SellerOffer`, `Payment`, `FundsLedgerEntry`, `DerivedDestination`, `TrezorAccount`, `PointTransaction`.
|
||||
- **Status 2026-06-01:** Drizzle schemas/repositories and backfill scripts exist for PurchaseRequest/SellerOffer. Backend `2.8.13` hardens the marketplace-core backfill path with `npm run backfill:marketplace-core:postgres`, fixed PurchaseRequest timestamp/preferred-seller writes, and a post-SellerOffer selected-offer remap step. Runtime marketplace services still call Mongoose directly and must not be flipped with `REPO_MARKETPLACE` until service wiring plus shadow-read checks land.
|
||||
- Apply §4.1 (Mixed→discriminator+FK), §4.2 (offers/preferredSellers junctions, deliveryInfo/serviceInfo child tables), §4.5 (derivation counter).
|
||||
- **Wrap in real PG transactions the multi-doc writes that today have none:** `raiseDispute` (PurchaseRequest + Payment), payment confirm + `FundsLedgerEntry` AML-fee insert, referral reward (points + referralStats), PointsService flows (migrate its 2 `withTransaction` sites to PG `BEGIN/COMMIT`).
|
||||
- Preserve the `Payment` partial-unique idempotency index and `FundsLedgerEntry.idempotencyKey` uniqueness.
|
||||
@@ -402,6 +403,7 @@ async function shadowRead(key, mongoFn, pgFn) {
|
||||
## 8. Cutover & rollback runbook (per collection)
|
||||
|
||||
1. **Backfill** in batches with checkpointing; allocate uuids → `id_map`; remap FKs from already-migrated parents. Re-runnable (idempotent upserts).
|
||||
- Marketplace-core operator path: `MIGRATION_MONGO_URL=... MIGRATION_PG_URL=... npm run backfill:marketplace-core:postgres:dry-run`, then `npm run backfill:marketplace-core:postgres` in non-prod. Run `scripts/smoke/marketplace-core-postgres-backfill.sh` with the same DSNs to exercise the static backfill invariants and dry-run.
|
||||
2. **Enable `dual`** (flag) — writes go to both; shadow-read diffing on. Backfill the delta accumulated during step 1.
|
||||
3. **Soak** until row-count + checksum + shadow-read are clean for the agreed window.
|
||||
4. **Flip reads to `pg`** (flag). Keep dual-write on.
|
||||
|
||||
@@ -3,12 +3,12 @@ title: Postgres Runtime Cutover Status
|
||||
tags: [data-model, postgres, migration, runtime-status]
|
||||
aliases: [Postgres Status, PG Cutover Status, Mongo vs Postgres Runtime]
|
||||
created: 2026-05-31
|
||||
source: backend integrate-main-into-development@ea43862
|
||||
source: backend integrate-main-into-development@6df113d
|
||||
---
|
||||
|
||||
# Postgres Runtime Cutover Status
|
||||
|
||||
> **Current branch:** backend `integrate-main-into-development` at `ea43862`, version `2.8.11`.
|
||||
> **Current branch:** backend `integrate-main-into-development` at `6df113d`, version `2.8.13`.
|
||||
>
|
||||
> **Bottom line:** this branch is **Postgres-capable**, not fully Postgres-backed. Auth-owned user data can run through Postgres when `AUTH_STORE=postgres`; confirmation-threshold runtime config/history can run through Postgres when `CONFIG_STORE=postgres`; user address CRUD can run through Postgres when `ADDRESS_STORE=postgres`; marketplace categories, level config, shop settings, and reviews can run through Postgres with their own store flags. All PG-backed stores require `PG_URL`. Mongo remains the default and the compatibility store for still-Mongo domains.
|
||||
|
||||
@@ -28,7 +28,7 @@ source: backend integrate-main-into-development@ea43862
|
||||
| Marketplace reviews | Opt-in with `REVIEW_STORE=postgres` | Review list/summary/create routes use a review-store facade. PG-mode list responses still hydrate `reviewerId` from the user mirror to preserve frontend shape. |
|
||||
| Repository implementations | Present but partial runtime wiring | `src/db/repositories/*` and Drizzle schemas exist for the target architecture, but this branch's live cutover work is still mostly store-specific raw PG facades plus the conditional oracle quote write path. |
|
||||
| Oracle quote persistence | Conditional runtime PG write | `/api/payment/request-network/intents` lazily imports `quoteRepo` only when `ORACLE_QUOTING_ENABLED=true`; it writes `payment_quotes` if the PG parent payment row exists, mirrors to Mongo `Payment.quote`, and records `pg_dualwrite_gaps` if PG is behind. |
|
||||
| Backfill/verify scripts | Available as operator tooling | `MIGRATION_PG_URL` drives backfill scripts; guards restrict allowed target hosts. These scripts are not run automatically by app startup. |
|
||||
| Backfill/verify scripts | Available as operator tooling | `MIGRATION_PG_URL` drives backfill scripts; guards restrict allowed target hosts. The marketplace-core runner group now backfills users/categories, purchase requests, seller offers, and the post-offer `selectedOfferId` remap in dependency order. These scripts are not run automatically by app startup. |
|
||||
|
||||
## What Is Still Mongo-Backed
|
||||
|
||||
@@ -38,7 +38,7 @@ Most of the service layer still imports Mongoose models directly. Auth-owned pat
|
||||
|---|---|---|
|
||||
| Legacy/broad user consumers | MongoDB mirror | Auth-owned users can be PG-backed, but still-Mongo domains expect Mongo ObjectId user references. PG-mode writes therefore maintain a Mongo mirror until those domains are cut over. |
|
||||
| Admin cleanup / seed address tooling | MongoDB | User-facing address CRUD is PG-capable, but admin cleanup and seed scripts still operate on Mongo first. Seed scripts backfill addresses to PG when `ADDRESS_STORE=postgres`. |
|
||||
| Marketplace requests/offers/templates | MongoDB | Marketplace, checkout, and seller-offer services still call `PurchaseRequest`, `SellerOffer`, and `RequestTemplate` directly. Category and shop-settings reads/writes are PG-capable through facades, but request/template documents still hold Mongo ObjectId references. |
|
||||
| Marketplace requests/offers/templates | MongoDB | Marketplace, checkout, and seller-offer services still call `PurchaseRequest`, `SellerOffer`, and `RequestTemplate` directly. Category and shop-settings reads/writes are PG-capable through facades. PurchaseRequest/SellerOffer backfill tooling is now operator-ready, but runtime reads/writes remain Mongo-first and RequestTemplate still has no PG table/repository. |
|
||||
| Payments and escrow state | MongoDB primary | Request Network, AMN scanner, webhook, admin, release/refund, adapter, reconciliation, and legacy payment paths still create/update `Payment` Mongoose documents directly. |
|
||||
| Funds ledger | MongoDB primary | `FundsLedgerEntry` remains the ledger used by current services; PG ledger tables exist but are not the live write target. |
|
||||
| Derived destinations and sweeps | MongoDB | Wallet destination allocation and sweep services call `DerivedDestination` directly. |
|
||||
@@ -61,7 +61,7 @@ Most of the service layer still imports Mongoose models directly. Auth-owned pat
|
||||
| `SHOP_SETTINGS_STORE` | `mongo` by default. Set `SHOP_SETTINGS_STORE=postgres` to route shop settings, review gates, and seller payment rails through Postgres. |
|
||||
| `REVIEW_STORE` | `mongo` by default. Set `REVIEW_STORE=postgres` to route marketplace reviews through Postgres. |
|
||||
| `PG_URL` | Makes PG code importable/reachable. Required for any `*_STORE=postgres` flag; does not cut over unrelated app domains by itself. |
|
||||
| `MIGRATION_PG_URL` | Used by backfill scripts and migration runbooks; not part of normal request handling. |
|
||||
| `MIGRATION_PG_URL` | Used by backfill scripts and migration runbooks; not part of normal request handling. Marketplace-core dry-run/non-dry backfills also require `MIGRATION_MONGO_URL`. |
|
||||
| `REPO_USER`, `REPO_PAYMENT`, `REPO_POINTS`, `REPO_MARKETPLACE`, `REPO_DEFAULT` | Repository factory flags exist, but broad services are not yet wired through the factory. Treat them as migration controls that need integration verification before relying on them. |
|
||||
| `ORACLE_QUOTING_ENABLED` | Enables server-side quote computation and the only current PG write path in normal checkout: `payment_quotes`, when a PG parent row can be resolved. |
|
||||
|
||||
@@ -77,11 +77,12 @@ Most of the service layer still imports Mongoose models directly. Auth-owned pat
|
||||
- `PG_URL=... npm run backfill:shop-settings:postgres`
|
||||
- `PG_URL=... npm run backfill:review:postgres`
|
||||
6. Run `PG_URL=... MONGODB_URI=... scripts/smoke/reference-stores-postgres.sh`, then set `CATEGORY_STORE=postgres LEVEL_CONFIG_STORE=postgres SHOP_SETTINGS_STORE=postgres REVIEW_STORE=postgres` together in non-prod.
|
||||
7. Run non-prod backfills for the remaining larger domains in dependency order and record row-count/checksum results.
|
||||
8. Wire remaining services to repository interfaces one domain at a time.
|
||||
9. Enable `dual` mode per large domain only after wiring is proven by tests and smoke checks.
|
||||
10. Run shadow-read/reconcile during a soak window.
|
||||
11. Flip reads to `pg` per domain only after zero-diff shadow reads and a rollback plan are in place.
|
||||
7. For marketplace-core data, run `MIGRATION_MONGO_URL=... MIGRATION_PG_URL=... npm run backfill:marketplace-core:postgres:dry-run`, then the non-dry `npm run backfill:marketplace-core:postgres` against non-prod. The group runs root dependencies, PurchaseRequest main rows, SellerOffer rows, then the selected-offer remap.
|
||||
8. Run `scripts/smoke/marketplace-core-postgres-backfill.sh` with the same migration DSNs and record row-count/checksum results.
|
||||
9. Wire remaining services to repository interfaces one domain at a time.
|
||||
10. Enable `dual` mode per large domain only after wiring is proven by tests and smoke checks.
|
||||
11. Run shadow-read/reconcile during a soak window.
|
||||
12. Flip reads to `pg` per domain only after zero-diff shadow reads and a rollback plan are in place.
|
||||
|
||||
## Related Docs
|
||||
|
||||
|
||||
Reference in New Issue
Block a user