nick/nick-doc
3
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs: sync from backend f1ba14b — notification pg backfill tooling

Browse Source
This commit is contained in:
Siavash Sameni
2026-06-02 10:44:18 +04:00
parent c90f286b12
commit bf82e7d628
2 changed files with 25 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
02 - Data Models/Postgres Runtime Cutover Status.md
View File

@@ -3,12 +3,12 @@ title: Postgres Runtime Cutover Status
tags: [data-model, postgres, migration, runtime-status]
aliases: [Postgres Status, PG Cutover Status, Mongo vs Postgres Runtime]
created: 2026-05-31
source: backend integrate-main-into-development@10de752 + frontend integrate-main-into-development@3dfbac2 + deployment main@38cb75b
source: backend integrate-main-into-development@f1ba14b + frontend integrate-main-into-development@b94d8a9 + deployment main@38cb75b
---
# Postgres Runtime Cutover Status
> **Current branch:** backend `integrate-main-into-development` at `10de752`, version `2.8.33`; frontend `integrate-main-into-development` at `3dfbac2`, version `2.8.33`; dev deployment `main` at `38cb75b`.
> **Current branch:** backend `integrate-main-into-development` at `f1ba14b`, version `2.8.34`; frontend `integrate-main-into-development` at `b94d8a9`, version `2.8.34`; dev deployment `main` at `38cb75b`.
>
> **Bottom line:** this branch is **Postgres-capable**, not fully Postgres-backed. Dev deployment now defaults the seven existing PG-capable runtime stores to Postgres: auth-owned users/Telegram auth, confirmation-threshold config/history, user addresses, categories, level config, shop settings, and reviews. Code-level defaults remain Mongo outside that deployment override, and Mongo remains the compatibility store for still-Mongo domains. The category PG path enforces one active visible category per normalized name. As of backend `2.8.33`, the active startup/health/admin/report import surface no longer has non-type top-level `mongoose` or `models/*` imports; legacy Mongo models are lazy-loaded only when fallback/backfill/maintenance actions run. All PG-backed stores require `PG_URL`.
@@ -26,13 +26,14 @@ source: backend integrate-main-into-development@10de752 + frontend integrate-mai
| Level configuration | PG-backed in dev deployment; code opt-in with `LEVEL_CONFIG_STORE=postgres` | `PointsService` level reads use a level-config facade. `PointTransaction` and user points remain Mongo-backed. |
| Shop settings | PG-backed in dev deployment; code opt-in with `SHOP_SETTINGS_STORE=postgres` | Shop settings controller, seller payment rail resolution, and review enable/disable checks use a shop-settings facade. PG-mode writes mirror back to Mongo. Backend `2.8.32` removed top-level Mongo model imports from this facade; legacy models load only for Mongo fallback/backfill/mirror paths. |
| Marketplace reviews | PG-backed in dev deployment; code opt-in with `REVIEW_STORE=postgres` | Review list/summary/create routes use a review-store facade. PG-mode list responses still hydrate `reviewerId` from the user mirror to preserve frontend shape. Backend `2.8.32` removed top-level Mongo model imports from this facade; legacy models load only for Mongo fallback/backfill/mirror paths. |
| Notifications | Repository-backed, code opt-in with `NOTIFICATION_STORE=postgres` or `REPO_NOTIFICATION=pg` | `NotificationService` uses `getNotificationRepo()` for create/list/read/delete/count paths. Backend `2.8.34` adds Mongo→Postgres notification backfill tooling, ordered-runner support, a dry-run path, and `scripts/smoke/notifications-postgres.sh`. Dev deployment has not yet flipped this store to Postgres. |
| Repository implementations | Present with first payment-ledger runtime seam | `src/db/repositories/*` and Drizzle schemas exist for the target architecture. Backend `2.8.20` wires `fundsLedgerService` appends/balance reads through `getPaymentRepo()`, making that ledger slice controllable by `REPO_PAYMENT=mongo|dual|pg`. The broader payment, marketplace, and points services still need method-by-method service wiring before their repo flags are safe runtime cutovers. |
| Oracle quote persistence | Conditional runtime PG write | `/api/payment/request-network/intents` lazily imports `quoteRepo` only when `ORACLE_QUOTING_ENABLED=true`; it writes `payment_quotes` if the PG parent payment row exists, mirrors to Mongo `Payment.quote`, and records `pg_dualwrite_gaps` if PG is behind. |
| Backfill/verify scripts | Available as operator tooling | `MIGRATION_PG_URL` drives backfill scripts; guards restrict allowed target hosts. The marketplace-core runner group now backfills users/categories, request templates, purchase requests, seller offers, and the post-offer `selectedOfferId` remap in dependency order. These scripts are not run automatically by app startup. |
## What Is Still Mongo-Backed
Active app startup, health, and the PG-capable store facades no longer top-load Mongoose models. Auth-owned paths now have an auth-store boundary; confirmation-threshold config, user addresses, categories, level config, shop settings, and reviews have store boundaries, and their PG-capable facades lazy-load legacy Mongo fallbacks instead of top-loading them. Funds ledger appends/balance reads now use the payment repository seam, but default to Mongo unless `REPO_PAYMENT` is flipped. Broad marketplace requests/offers/templates, most payment paths, points transactions, chat, notification, and admin maintenance actions remain Mongo-first when exercised.
Active app startup, health, and the PG-capable store facades no longer top-load Mongoose models. Auth-owned paths now have an auth-store boundary; confirmation-threshold config, user addresses, categories, level config, shop settings, and reviews have store boundaries, and their PG-capable facades lazy-load legacy Mongo fallbacks instead of top-loading them. Notification runtime paths are repository-backed and now have backfill/smoke tooling, but dev deployment still leaves the store in Mongo mode. Funds ledger appends/balance reads now use the payment repository seam, but default to Mongo unless `REPO_PAYMENT` is flipped. Broad marketplace requests/offers/templates, most payment paths, points transactions, chat, and admin maintenance actions remain Mongo-first when exercised.
| Domain | Current live store | Why not Postgres yet |
|---|---|---|
@@ -44,7 +45,7 @@ Active app startup, health, and the PG-capable store facades no longer top-load
| Derived destinations and sweeps | MongoDB | Wallet destination allocation and sweep services call `DerivedDestination` directly. |
| Points/referrals/transactions | MongoDB | Level configuration is PG-capable, but `User` points fields, `PointTransaction`, referral aggregation, and Mongo transactions remain Mongo-backed. |
| Chat/messages | MongoDB | Chat intentionally remains document-shaped and is not part of the current PG cutover. |
| Notifications | MongoDB | Notification TTL/read-state paths remain Mongo-backed. |
| Notifications | Repository-backed, default Mongo in dev deployment | `NotificationService` routes through `getNotificationRepo()`, so `NOTIFICATION_STORE=postgres` / `REPO_NOTIFICATION=pg` can exercise the Drizzle repo. Backend `2.8.34` adds backfill and smoke coverage. Dev compose/Gatus have not yet been flipped from the seven-store PG baseline to include notification. |
| Disputes/blog/content/admin cleanup | MongoDB | Reviews are PG-capable; disputes, blog/content, and admin cleanup still call their Mongoose models directly. |
| Runtime config outside confirmation thresholds | MongoDB | `ConfigSetting` and `ConfigSettingHistory` are PG-capable for confirmation thresholds only; any future admin-editable settings need to route through the same config-store boundary before they count as cut over. |
| Telegram link/session/temp verification | PG-backed in dev deployment; code default MongoDB | These records move with `AUTH_STORE=postgres`. Dev compose defaults that flag to `postgres`; environments without the override remain Mongo until the flag is flipped. |
@@ -62,6 +63,7 @@ The backend code defaults every store flag below to `mongo`. Dev deployment over
| `LEVEL_CONFIG_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes level configuration reads and seed replacement through Postgres. `LEVEL_STORE=postgres` is accepted as a compatibility alias. |
| `SHOP_SETTINGS_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes shop settings, review gates, and seller payment rails through Postgres. |
| `REVIEW_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes marketplace reviews through Postgres. |
| `NOTIFICATION_STORE` / `REPO_NOTIFICATION` | Code default `mongo`; `postgres`/`pg` routes notification inbox create/list/read/delete/count through the Drizzle notification repo. Backend `2.8.34` adds `npm run backfill:notification:postgres`, ordered-runner step `notifications`, and `scripts/smoke/notifications-postgres.sh`. |
| `PG_URL` | Makes PG code importable/reachable. Required for any `*_STORE=postgres` flag; does not cut over unrelated app domains by itself. |
| `MIGRATION_PG_URL` | Used by backfill scripts and migration runbooks; not part of normal request handling. Marketplace-core dry-run/non-dry backfills also require `MIGRATION_MONGO_URL`. |
| `REPO_PAYMENT` | Code default `mongo`. As of backend `2.8.20`, funds ledger appends and balance reads use this flag through `getPaymentRepo()`. Do not flip broad payment runtime to `pg` yet; most payment services still call Mongoose directly. |
@@ -83,12 +85,13 @@ The backend code defaults every store flag below to `mongo`. Dev deployment over
7. Run `PG_URL=... scripts/smoke/categories-postgres-unique.sh` and `PG_URL=... MONGODB_URI=... scripts/smoke/reference-stores-postgres.sh`, then confirm `CATEGORY_STORE=postgres LEVEL_CONFIG_STORE=postgres SHOP_SETTINGS_STORE=postgres REVIEW_STORE=postgres` in non-prod.
8. For marketplace-core data, run `MIGRATION_MONGO_URL=... MIGRATION_PG_URL=... npm run backfill:marketplace-core:postgres:dry-run`, then the non-dry `npm run backfill:marketplace-core:postgres` against non-prod. The group runs root dependencies, RequestTemplate rows, PurchaseRequest main rows, SellerOffer rows, then the selected-offer remap.
9. Run `scripts/smoke/marketplace-core-postgres-backfill.sh` with the same migration DSNs and record row-count/checksum results.
10. Continue payment-domain wiring after the ledger seam: add the missing payment repo methods for provider lookups, transaction-hash/webhook lookups, metadata/blockchain patching, template duplicate cleanup, and quote updates before moving `paymentService`, `paymentCoordinator`, RN, or AMN scanner routes.
11. Add a derived-destination/sweep repository seam before payment PG cutover; destination allocation is payment-address state and should not stay Mongo-only once payments become PG-backed.
12. Wire remaining services to repository interfaces one domain at a time.
13. Enable `dual` mode per large domain only after wiring is proven by tests and smoke checks.
14. Run shadow-read/reconcile during a soak window.
15. Flip reads to `pg` per domain only after zero-diff shadow reads and a rollback plan are in place.
10. For notifications, run `PG_URL=... npm run backfill:notification:postgres`, run `PG_URL=... scripts/smoke/notifications-postgres.sh`, then flip `NOTIFICATION_STORE=postgres` in non-prod and update Gatus from seven to eight required PG stores.
11. Continue payment-domain wiring after the ledger seam: add the missing payment repo methods for provider lookups, transaction-hash/webhook lookups, metadata/blockchain patching, template duplicate cleanup, and quote updates before moving `paymentService`, `paymentCoordinator`, RN, or AMN scanner routes.
12. Add a derived-destination/sweep repository seam before payment PG cutover; destination allocation is payment-address state and should not stay Mongo-only once payments become PG-backed.
13. Wire remaining services to repository interfaces one domain at a time.
14. Enable `dual` mode per large domain only after wiring is proven by tests and smoke checks.
15. Run shadow-read/reconcile during a soak window.
16. Flip reads to `pg` per domain only after zero-diff shadow reads and a rollback plan are in place.
## Related Docs