docs: sync from backend 2c5c3c7 — pg ledger repo seam
This commit is contained in:
Siavash Sameni
@@ -4,7 +4,7 @@ tags: [data-model, migration, postgres, drizzle, plan, runbook]
|
||||
aliases: [Drizzle Migration Plan, PG Migration Plan]
|
||||
created: 2026-05-31
|
||||
companion: "[[MongoDB to PostgreSQL Migration Guide]]"
|
||||
updated: 2026-06-01 for backend integrate-main-into-development@c5db471 backend 2.8.19 + deployment main@38cb75b
|
||||
updated: 2026-06-01 for backend integrate-main-into-development@2c5c3c7 backend 2.8.20 + deployment main@38cb75b
|
||||
---
|
||||
|
||||
# MongoDB → PostgreSQL Migration Plan (Drizzle)
|
||||
@@ -17,7 +17,7 @@ updated: 2026-06-01 for backend integrate-main-into-development@c5db471 backend
|
||||
> **Scope reminder:** partial migration (Phases 0–5) is the recommended stopping point — ≈16–28 engineer-weeks. Full migration of Chat/Notification/sessions is explicitly deferred.
|
||||
|
||||
> [!warning] Current implementation status
|
||||
> Backend `2.8.19` has started the runtime cutover with store-specific raw Postgres facades: auth-owned users/Telegram auth records behind `AUTH_STORE=postgres`, confirmation-threshold config/history behind `CONFIG_STORE=postgres`, user address CRUD behind `ADDRESS_STORE=postgres`, and the first marketplace/reference domains behind `CATEGORY_STORE=postgres`, `LEVEL_CONFIG_STORE=postgres`, `SHOP_SETTINGS_STORE=postgres`, and `REVIEW_STORE=postgres`. Category PG mode now deactivates duplicate active names and enforces an active normalized-name unique index. It also contains the broader `src/db/` Drizzle schemas through `0010`, repository implementations/factory, id-map bridge, and backfill runner described below. RequestTemplate now has a PG table/backfill, but broad marketplace/payment/points services are still mostly not wired through that factory. Code defaults remain Mongo unless a per-store flag is explicitly flipped; dev deployment `38cb75b` now flips the seven PG-capable store flags to Postgres by default. See [[Postgres Runtime Cutover Status]].
|
||||
> Backend `2.8.20` has started the runtime cutover with store-specific raw Postgres facades: auth-owned users/Telegram auth records behind `AUTH_STORE=postgres`, confirmation-threshold config/history behind `CONFIG_STORE=postgres`, user address CRUD behind `ADDRESS_STORE=postgres`, and the first marketplace/reference domains behind `CATEGORY_STORE=postgres`, `LEVEL_CONFIG_STORE=postgres`, `SHOP_SETTINGS_STORE=postgres`, and `REVIEW_STORE=postgres`. Category PG mode now deactivates duplicate active names and enforces an active normalized-name unique index. It also contains the broader `src/db/` Drizzle schemas through `0010`, repository implementations/factory, id-map bridge, and backfill runner described below. RequestTemplate now has a PG table/backfill. Funds ledger appends and balance reads now route through `getPaymentRepo()` and can be controlled by `REPO_PAYMENT`, but broad marketplace/payment/points services are still mostly not wired through their factory repos. Code defaults remain Mongo unless a per-store flag is explicitly flipped; dev deployment `38cb75b` now flips the seven PG-capable store flags to Postgres by default. See [[Postgres Runtime Cutover Status]].
|
||||
|
||||
---
|
||||
|
||||
@@ -365,7 +365,7 @@ Same phases as the guide §2, here with Drizzle-concrete entry/exit gates. Each
|
||||
|
||||
### Phase 4 — Money core (6–10 wk) — *the point of the project*
|
||||
- `PurchaseRequest`, `SellerOffer`, `RequestTemplate`, `Payment`, `FundsLedgerEntry`, `DerivedDestination`, `TrezorAccount`, `PointTransaction`.
|
||||
- **Status 2026-06-01:** Drizzle schemas and backfill scripts exist for PurchaseRequest/SellerOffer/RequestTemplate. Backend `2.8.19` hardens the marketplace-core backfill path with `npm run backfill:marketplace-core:postgres`, fixed PurchaseRequest timestamp/preferred-seller writes, a RequestTemplate backfill step, a post-SellerOffer selected-offer remap step, and category duplicate cleanup/unique active-name enforcement. Runtime marketplace services still call Mongoose directly and must not be flipped with `REPO_MARKETPLACE` until service wiring plus shadow-read checks land.
|
||||
- **Status 2026-06-01:** Drizzle schemas and backfill scripts exist for PurchaseRequest/SellerOffer/RequestTemplate. Backend `2.8.19` hardens the marketplace-core backfill path with `npm run backfill:marketplace-core:postgres`, fixed PurchaseRequest timestamp/preferred-seller writes, a RequestTemplate backfill step, a post-SellerOffer selected-offer remap step, and category duplicate cleanup/unique active-name enforcement. Backend `2.8.20` wires the funds-ledger service through `getPaymentRepo()`, fixes Mongo/Drizzle payment-stat parity for future service wiring, and makes the repo factory lazy-load PG/dual implementations so Mongo mode does not require `PG_URL`. Runtime marketplace services still call Mongoose directly and must not be flipped with `REPO_MARKETPLACE` until service wiring plus shadow-read checks land.
|
||||
- Apply §4.1 (Mixed→discriminator+FK), §4.2 (offers/preferredSellers junctions, deliveryInfo/serviceInfo child tables), §4.5 (derivation counter).
|
||||
- **Wrap in real PG transactions the multi-doc writes that today have none:** `raiseDispute` (PurchaseRequest + Payment), payment confirm + `FundsLedgerEntry` AML-fee insert, referral reward (points + referralStats), PointsService flows (migrate its 2 `withTransaction` sites to PG `BEGIN/COMMIT`).
|
||||
- Preserve the `Payment` partial-unique idempotency index and `FundsLedgerEntry.idempotencyKey` uniqueness.
|
||||
|
||||
@@ -6,12 +6,12 @@ aliases: [Payment Record, Escrow, IPayment]
|
||||
|
||||
# Payment
|
||||
|
||||
> **Last updated:** 2026-05-31 — added AMN scanner provider, oracle quote mirror, Postgres `payment_quotes` linkage, webhook confirmation persistence, and capped accepted confirmation counts.
|
||||
> **Last updated:** 2026-06-01 — documented the first payment-repo runtime seam for funds ledger appends/balance reads.
|
||||
|
||||
Records every monetary movement in the marketplace: buyer pay-ins, seller payouts, and refunds. The current model is centered on Request Network pay-in, in-house checkout metadata, on-chain transaction verification, escrow state, and provider request IDs. The `provider` and `direction` discriminators let one collection hold incoming buyer payments, outgoing seller releases, refunds, and legacy/other provider records.
|
||||
|
||||
> [!warning] Runtime store
|
||||
> The `Payment` document is still created, read, and updated through Mongoose on normal request paths. Backend `2.6.82` can persist oracle quotes to Postgres `payment_quotes`, but that is conditional on `ORACLE_QUOTING_ENABLED=true` and does not make the whole payment domain PG-authoritative. See [[Postgres Runtime Cutover Status]].
|
||||
> The `Payment` document is still created, read, and updated through Mongoose on most normal request paths. Backend `2.8.20` routes `FundsLedgerEntry` appends and balance reads through the payment repository seam (`REPO_PAYMENT=mongo|dual|pg`), but the default remains Mongo and this does not make the whole payment domain PG-authoritative. Oracle quotes can persist to Postgres `payment_quotes` when `ORACLE_QUOTING_ENABLED=true` and a PG parent payment row can be resolved. See [[Postgres Runtime Cutover Status]].
|
||||
|
||||
> [!note] Source
|
||||
> `backend/src/models/Payment.ts:3` — schema definition
|
||||
@@ -23,12 +23,8 @@ Records every monetary movement in the marketplace: buyer pay-ins, seller payout
|
||||
> [!note] `provider` values
|
||||
> The current backend schema accepts `request.network`, `amn.scanner`, `shkeeper`, and `other`. `amn.scanner` is the in-house scanner provider used for direct on-chain monitoring. Older docs and some frontend types may still mention historical values such as `test` or `decentralized`; treat those as legacy until their active routes are audited again.
|
||||
|
||||
> [!warning] `confirmed` vs `completed` — stats undercount
|
||||
> Payment stats (`paymentService.getPaymentStats`) only increment `successfulPayments` for status **`confirmed`**:
|
||||
> ```ts
|
||||
> case "confirmed": stats.successfulPayments += stat.count; break;
|
||||
> ```
|
||||
> The terminal SHKeeper / DePay state is **`completed`**, which has no case in the switch and is therefore **not** counted as a successful payment. ⚠️ This causes successful-payment stats to undercount any payment that reached `completed`.
|
||||
> [!note] `confirmed` vs `completed` — stats parity
|
||||
> Payment stats should count both **`confirmed`** and **`completed`** as successful. Backend `2.8.20` aligns the Mongo and Drizzle payment repository implementations with that behavior before broader payment-service wiring.
|
||||
|
||||
> [!warning] `SIM_` payment-hash bypass — security concern
|
||||
> In both `payment/paymentRoutes.ts` and `marketplace/routes.ts`, a `paymentHash` that starts with `SIM_` (or a short `0x...` hash under 64 chars) is treated as a simulated transaction and **skips on-chain verification entirely** (`isVerified = true`). There is **no environment guard** (e.g. no `NODE_ENV !== 'production'` check) around this branch, so the bypass is reachable in production. ⚠️ A caller can mark a payment verified without any real on-chain settlement.
|
||||
@@ -126,6 +122,12 @@ Defined at `backend/src/models/Payment.ts:174-188`:
|
||||
|
||||
The Postgres money-core branch can store oracle quotes in `payment_quotes`, a 1:1 child table keyed by `payment_id → payments.id`. Amount/rate columns use `numeric(38,18)` and the route resolves the PG parent through `payments.legacy_object_id` or `id_map` during the Mongo/PG dual-write window. If the PG payment row is missing, the quote is mirrored to this Mongo `quote` subdocument and a `pg_dualwrite_gaps` row is recorded for reconciliation. This table is quote/audit storage only until the payment service itself is wired through the PG repository path.
|
||||
|
||||
## Funds Ledger Repository Seam
|
||||
|
||||
Backend `2.8.20` routes `appendFundsLedgerEntry`, `getFundsBalanceByPurchaseRequestId`, and `getFundsBalanceByPaymentId` through `getPaymentRepo()`. In default mode this is still `MongoPaymentRepo`, preserving the existing `FundsLedgerEntry` collection behavior. `REPO_PAYMENT=dual` can mirror ledger writes to Postgres after backfill/verification; `REPO_PAYMENT=pg` should wait until the surrounding payment services, derived destinations, and webhook/update paths are also wired and soaked.
|
||||
|
||||
The Drizzle ledger balance path supports both UUID entity refs and external/string refs, which matters for template-checkout rows where `purchaseRequestId` or `paymentId` is not a normal Mongo ObjectId.
|
||||
|
||||
## Pre/Post Hooks
|
||||
|
||||
None declared.
|
||||
|
||||
@@ -3,12 +3,12 @@ title: Postgres Runtime Cutover Status
|
||||
tags: [data-model, postgres, migration, runtime-status]
|
||||
aliases: [Postgres Status, PG Cutover Status, Mongo vs Postgres Runtime]
|
||||
created: 2026-05-31
|
||||
source: backend integrate-main-into-development@c5db471 + deployment main@38cb75b
|
||||
source: backend integrate-main-into-development@2c5c3c7 + frontend integrate-main-into-development@775a73b + deployment main@38cb75b
|
||||
---
|
||||
|
||||
# Postgres Runtime Cutover Status
|
||||
|
||||
> **Current branch:** backend `integrate-main-into-development` at `c5db471`, version `2.8.19`; dev deployment `main` at `38cb75b`.
|
||||
> **Current branch:** backend `integrate-main-into-development` at `2c5c3c7`, version `2.8.20`; frontend `integrate-main-into-development` at `775a73b`, version `2.8.20`; dev deployment `main` at `38cb75b`.
|
||||
>
|
||||
> **Bottom line:** this branch is **Postgres-capable**, not fully Postgres-backed. Dev deployment now defaults the seven existing PG-capable runtime stores to Postgres: auth-owned users/Telegram auth, confirmation-threshold config/history, user addresses, categories, level config, shop settings, and reviews. Code-level defaults remain Mongo outside that deployment override, and Mongo remains the compatibility store for still-Mongo domains. The category PG path enforces one active visible category per normalized name. All PG-backed stores require `PG_URL`.
|
||||
|
||||
@@ -26,21 +26,21 @@ source: backend integrate-main-into-development@c5db471 + deployment main@38cb75
|
||||
| Level configuration | PG-backed in dev deployment; code opt-in with `LEVEL_CONFIG_STORE=postgres` | `PointsService` level reads use a level-config facade. `PointTransaction` and user points remain Mongo-backed. |
|
||||
| Shop settings | PG-backed in dev deployment; code opt-in with `SHOP_SETTINGS_STORE=postgres` | Shop settings controller, seller payment rail resolution, and review enable/disable checks use a shop-settings facade. PG-mode writes mirror back to Mongo. |
|
||||
| Marketplace reviews | PG-backed in dev deployment; code opt-in with `REVIEW_STORE=postgres` | Review list/summary/create routes use a review-store facade. PG-mode list responses still hydrate `reviewerId` from the user mirror to preserve frontend shape. |
|
||||
| Repository implementations | Present but partial runtime wiring | `src/db/repositories/*` and Drizzle schemas exist for the target architecture, but this branch's live cutover work is still mostly store-specific raw PG facades plus the conditional oracle quote write path. |
|
||||
| Repository implementations | Present with first payment-ledger runtime seam | `src/db/repositories/*` and Drizzle schemas exist for the target architecture. Backend `2.8.20` wires `fundsLedgerService` appends/balance reads through `getPaymentRepo()`, making that ledger slice controllable by `REPO_PAYMENT=mongo|dual|pg`. The broader payment, marketplace, and points services still need method-by-method service wiring before their repo flags are safe runtime cutovers. |
|
||||
| Oracle quote persistence | Conditional runtime PG write | `/api/payment/request-network/intents` lazily imports `quoteRepo` only when `ORACLE_QUOTING_ENABLED=true`; it writes `payment_quotes` if the PG parent payment row exists, mirrors to Mongo `Payment.quote`, and records `pg_dualwrite_gaps` if PG is behind. |
|
||||
| Backfill/verify scripts | Available as operator tooling | `MIGRATION_PG_URL` drives backfill scripts; guards restrict allowed target hosts. The marketplace-core runner group now backfills users/categories, request templates, purchase requests, seller offers, and the post-offer `selectedOfferId` remap in dependency order. These scripts are not run automatically by app startup. |
|
||||
|
||||
## What Is Still Mongo-Backed
|
||||
|
||||
Most of the service layer still imports Mongoose models directly. Auth-owned paths now have an auth-store boundary; confirmation-threshold config, user addresses, categories, level config, shop settings, and reviews have store boundaries. Broad marketplace requests/offers/templates, payment, funds ledger, points transactions, chat, notification, and admin paths remain Mongo-first.
|
||||
Most of the service layer still imports Mongoose models directly. Auth-owned paths now have an auth-store boundary; confirmation-threshold config, user addresses, categories, level config, shop settings, and reviews have store boundaries. Funds ledger appends/balance reads now use the payment repository seam, but default to Mongo unless `REPO_PAYMENT` is flipped. Broad marketplace requests/offers/templates, most payment paths, points transactions, chat, notification, and admin paths remain Mongo-first.
|
||||
|
||||
| Domain | Current live store | Why not Postgres yet |
|
||||
|---|---|---|
|
||||
| Legacy/broad user consumers | MongoDB mirror | Auth-owned users can be PG-backed, but still-Mongo domains expect Mongo ObjectId user references. PG-mode writes therefore maintain a Mongo mirror until those domains are cut over. |
|
||||
| Admin cleanup / seed address tooling | MongoDB | User-facing address CRUD is PG-capable, but admin cleanup and seed scripts still operate on Mongo first. Seed scripts backfill addresses to PG when `ADDRESS_STORE=postgres`. |
|
||||
| Marketplace requests/offers/templates | MongoDB | Marketplace, checkout, and seller-offer services still call `PurchaseRequest`, `SellerOffer`, and `RequestTemplate` directly. Category and shop-settings reads/writes are PG-capable through facades. PurchaseRequest/SellerOffer/RequestTemplate backfill tooling is now operator-ready, but runtime reads/writes remain Mongo-first and RequestTemplate still has no runtime repo/service wiring. |
|
||||
| Payments and escrow state | MongoDB primary | Request Network, AMN scanner, webhook, admin, release/refund, adapter, reconciliation, and legacy payment paths still create/update `Payment` Mongoose documents directly. |
|
||||
| Funds ledger | MongoDB primary | `FundsLedgerEntry` remains the ledger used by current services; PG ledger tables exist but are not the live write target. |
|
||||
| Payments and escrow state | MongoDB primary | Request Network, AMN scanner, webhook, admin, release/refund, adapter, reconciliation, and legacy payment paths still create/update `Payment` Mongoose documents directly. Payment repository methods exist but are not broadly wired into runtime services yet. |
|
||||
| Funds ledger | Repository-backed, default Mongo | `appendFundsLedgerEntry` and `getFundsBalanceBy*` now call `getPaymentRepo()`. In default mode that is `MongoPaymentRepo`; `REPO_PAYMENT=dual`/`pg` can exercise the PG ledger implementation after backfill/soak. Drizzle balance reads support both UUID refs and external/string refs used by template checkout. |
|
||||
| Derived destinations and sweeps | MongoDB | Wallet destination allocation and sweep services call `DerivedDestination` directly. |
|
||||
| Points/referrals/transactions | MongoDB | Level configuration is PG-capable, but `User` points fields, `PointTransaction`, referral aggregation, and Mongo transactions remain Mongo-backed. |
|
||||
| Chat/messages | MongoDB | Chat intentionally remains document-shaped and is not part of the current PG cutover. |
|
||||
@@ -64,7 +64,8 @@ The backend code defaults every store flag below to `mongo`. Dev deployment over
|
||||
| `REVIEW_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes marketplace reviews through Postgres. |
|
||||
| `PG_URL` | Makes PG code importable/reachable. Required for any `*_STORE=postgres` flag; does not cut over unrelated app domains by itself. |
|
||||
| `MIGRATION_PG_URL` | Used by backfill scripts and migration runbooks; not part of normal request handling. Marketplace-core dry-run/non-dry backfills also require `MIGRATION_MONGO_URL`. |
|
||||
| `REPO_USER`, `REPO_PAYMENT`, `REPO_POINTS`, `REPO_MARKETPLACE`, `REPO_DEFAULT` | Repository factory flags exist, but broad services are not yet wired through the factory. Treat them as migration controls that need integration verification before relying on them. |
|
||||
| `REPO_PAYMENT` | Code default `mongo`. As of backend `2.8.20`, funds ledger appends and balance reads use this flag through `getPaymentRepo()`. Do not flip broad payment runtime to `pg` yet; most payment services still call Mongoose directly. |
|
||||
| `REPO_USER`, `REPO_POINTS`, `REPO_MARKETPLACE`, `REPO_DEFAULT` | Repository factory flags exist, but broad services are not yet wired through the factory. Treat them as migration controls that need integration verification before relying on them. The factory lazy-loads PG/dual implementations so importing it in Mongo mode does not require `PG_URL`. |
|
||||
| `ORACLE_QUOTING_ENABLED` | Enables server-side quote computation and the only current PG write path in normal checkout: `payment_quotes`, when a PG parent row can be resolved. |
|
||||
|
||||
## Next Cutover Work
|
||||
@@ -82,10 +83,12 @@ The backend code defaults every store flag below to `mongo`. Dev deployment over
|
||||
7. Run `PG_URL=... scripts/smoke/categories-postgres-unique.sh` and `PG_URL=... MONGODB_URI=... scripts/smoke/reference-stores-postgres.sh`, then confirm `CATEGORY_STORE=postgres LEVEL_CONFIG_STORE=postgres SHOP_SETTINGS_STORE=postgres REVIEW_STORE=postgres` in non-prod.
|
||||
8. For marketplace-core data, run `MIGRATION_MONGO_URL=... MIGRATION_PG_URL=... npm run backfill:marketplace-core:postgres:dry-run`, then the non-dry `npm run backfill:marketplace-core:postgres` against non-prod. The group runs root dependencies, RequestTemplate rows, PurchaseRequest main rows, SellerOffer rows, then the selected-offer remap.
|
||||
9. Run `scripts/smoke/marketplace-core-postgres-backfill.sh` with the same migration DSNs and record row-count/checksum results.
|
||||
10. Wire remaining services to repository interfaces one domain at a time.
|
||||
11. Enable `dual` mode per large domain only after wiring is proven by tests and smoke checks.
|
||||
12. Run shadow-read/reconcile during a soak window.
|
||||
13. Flip reads to `pg` per domain only after zero-diff shadow reads and a rollback plan are in place.
|
||||
10. Continue payment-domain wiring after the ledger seam: add the missing payment repo methods for provider lookups, transaction-hash/webhook lookups, metadata/blockchain patching, template duplicate cleanup, and quote updates before moving `paymentService`, `paymentCoordinator`, RN, or AMN scanner routes.
|
||||
11. Add a derived-destination/sweep repository seam before payment PG cutover; destination allocation is payment-address state and should not stay Mongo-only once payments become PG-backed.
|
||||
12. Wire remaining services to repository interfaces one domain at a time.
|
||||
13. Enable `dual` mode per large domain only after wiring is proven by tests and smoke checks.
|
||||
14. Run shadow-read/reconcile during a soak window.
|
||||
15. Flip reads to `pg` per domain only after zero-diff shadow reads and a rollback plan are in place.
|
||||
|
||||
## Related Docs
|
||||
|
||||
|
||||
@@ -11,6 +11,18 @@ entries on top. Maintained by agents per the rule in `../AGENTS.md`.
|
||||
|
||||
---
|
||||
|
||||
### 2026-06-01 — backend@2c5c3c7, frontend@775a73b — route funds ledger through payment repo seam
|
||||
|
||||
**Commits:** backend `2c5c3c7`, frontend `775a73b` (backend `2.8.20`, frontend `2.8.20`)
|
||||
**Touched:**
|
||||
- Backend: `src/services/payment/ledger/fundsLedgerService.ts`, `src/db/repositories/factory.ts`, `src/db/repositories/drizzle/DrizzlePaymentRepo.ts`, `src/db/repositories/mongo/MongoPaymentRepo.ts`, `__tests__/payment-ledger.service.test.ts`, `__tests__/mongo-payment-repo.test.ts`, `scripts/smoke/funds-ledger-repo.sh`, `package.json`, `package-lock.json`
|
||||
- Frontend: `package.json`, `Dockerfile` version metadata only.
|
||||
**Why:** Start replacing the remaining PG-capable repository stores with a low-risk money-core slice. Funds ledger appends and balance reads now go through `getPaymentRepo()`, so `REPO_PAYMENT=mongo|dual|pg` can control the ledger path. The repo factory now lazy-loads PG/dual implementations so importing it in Mongo mode no longer requires `PG_URL`. Mongo/Drizzle payment repo stats were aligned with live behavior (`buyerId`, nested `amount.amount`, and `completed` counted as successful), and Drizzle ledger balance reads now match external/string refs as well as UUID refs.
|
||||
**Verification:** Backend `git pull --rebase --autostash` (already up to date); backend `scripts/smoke/funds-ledger-repo.sh`; backend `npm test -- --runTestsByPath __tests__/payment-release-refund-orchestration.test.ts __tests__/money-safety.test.ts --runInBand` (`money-safety` skipped by its own env guard); backend `npm run typecheck -- --pretty false`; backend `npm run build:server`; frontend `git pull --rebase --autostash` (already up to date); frontend `npx tsc --noEmit --project tsconfig.json`; frontend `npm run build` (passed with the existing non-fatal SSR `getPosts` fetch refusal during static page generation); backend/frontend `git diff --check`.
|
||||
**Linked docs updated:** [[Postgres Runtime Cutover Status]], [[MongoDB to PostgreSQL Migration Plan (Drizzle)]], [[Payment]]
|
||||
|
||||
---
|
||||
|
||||
### 2026-06-01 — deployment@38cb75b — default PG-capable stores to Postgres in dev
|
||||
|
||||
**Commits:** deployment `38cb75b`
|
||||
|
||||
Reference in New Issue
Block a user