nick/nick-doc
3
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
bf82e7d6283c022256a3d96d4005ea16244bdc6a
nick-doc/02 - Data Models/Postgres Runtime Cutover Status.md

16 KiB
Raw Blame History

title, tags, aliases, created, source
title tags aliases created source
Postgres Runtime Cutover Status
data-model
postgres
migration
runtime-status
Postgres Status
PG Cutover Status
Mongo vs Postgres Runtime
2026-05-31 backend integrate-main-into-development@f1ba14b + frontend integrate-main-into-development@b94d8a9 + deployment main@38cb75b

Postgres Runtime Cutover Status

Current branch: backend integrate-main-into-development at f1ba14b, version 2.8.34; frontend integrate-main-into-development at b94d8a9, version 2.8.34; dev deployment main at 38cb75b.

Bottom line: this branch is Postgres-capable, not fully Postgres-backed. Dev deployment now defaults the seven existing PG-capable runtime stores to Postgres: auth-owned users/Telegram auth, confirmation-threshold config/history, user addresses, categories, level config, shop settings, and reviews. Code-level defaults remain Mongo outside that deployment override, and Mongo remains the compatibility store for still-Mongo domains. The category PG path enforces one active visible category per normalized name. As of backend 2.8.33, the active startup/health/admin/report import surface no longer has non-type top-level mongoose or models/* imports; legacy Mongo models are lazy-loaded only when fallback/backfill/maintenance actions run. All PG-backed stores require PG_URL.

What Uses Postgres Now

Area Runtime status Notes
Postgres connection Available when PG_URL is set Current store facades use src/infrastructure/postgres/client.ts; the broader src/db/ Drizzle layer and repository factory exist, but most live services are not wired through that factory yet.
Runtime schema bootstrap Implemented for auth, config, address, and reference stores Auth tables are bootstrapped from src/services/auth/postgresAuthSchema.ts; store facades bootstrap their own tables at startup when their *_STORE=postgres flag is enabled.
Health observability Implemented in /api/health checks.postgres reports configured, required, storeModes, enabledStores, and enabledStoreCount, so Gatus/operators can verify both PG reachability and which runtime stores are actively PG-backed. Dev Gatus now asserts all seven PG-capable store modes are postgres. Backend 2.8.33 lazy-loads Mongoose for the legacy Mongo health check and skips it when Mongo is optional under MONGO_CONNECT_MODE=auto/never.
Auth-owned user store PG-backed in dev deployment; code opt-in with AUTH_STORE=postgres Auth, passkey, Telegram auth/link/session/temp-verification, and /api/user profile paths use an auth-store facade. In PG mode, users are stored in Postgres and mirrored back to Mongo through legacy_object_id for compatibility with still-Mongo services.
Confirmation-threshold runtime config PG-backed in dev deployment; code opt-in with CONFIG_STORE=postgres ConfigSetting / ConfigSettingHistory access for /api/admin/settings/confirmation-thresholds and transaction-safety confirmation thresholds uses a config-store facade. PG-mode writes mirror back to Mongo for rollback. Backend 2.8.32 removed top-level Mongo model imports from this facade; legacy models load only for Mongo fallback/backfill/mirror paths.
User addresses PG-backed in dev deployment; code opt-in with ADDRESS_STORE=postgres /api/addresses CRUD uses an address-store facade. PG mode enforces one primary address per user with a partial unique index and mirrors writes/deletes back to Mongo for rollback.
Marketplace categories PG-backed in dev deployment; code opt-in with CATEGORY_STORE=postgres CategoryService and the default General category path use a category-store facade. PG-mode writes mirror back to Mongo for rollback and still-Mongo request/template references. PG schema bootstrap/migration deactivates duplicate active category labels, repoints existing category references to the kept row, and enforces categories_active_name_norm_uq on lower(btrim(name)) WHERE is_active = true. List/cache reads also dedupe by normalized name.
Level configuration PG-backed in dev deployment; code opt-in with LEVEL_CONFIG_STORE=postgres PointsService level reads use a level-config facade. PointTransaction and user points remain Mongo-backed.
Shop settings PG-backed in dev deployment; code opt-in with SHOP_SETTINGS_STORE=postgres Shop settings controller, seller payment rail resolution, and review enable/disable checks use a shop-settings facade. PG-mode writes mirror back to Mongo. Backend 2.8.32 removed top-level Mongo model imports from this facade; legacy models load only for Mongo fallback/backfill/mirror paths.
Marketplace reviews PG-backed in dev deployment; code opt-in with REVIEW_STORE=postgres Review list/summary/create routes use a review-store facade. PG-mode list responses still hydrate reviewerId from the user mirror to preserve frontend shape. Backend 2.8.32 removed top-level Mongo model imports from this facade; legacy models load only for Mongo fallback/backfill/mirror paths.
Notifications Repository-backed, code opt-in with NOTIFICATION_STORE=postgres or REPO_NOTIFICATION=pg NotificationService uses getNotificationRepo() for create/list/read/delete/count paths. Backend 2.8.34 adds Mongo→Postgres notification backfill tooling, ordered-runner support, a dry-run path, and scripts/smoke/notifications-postgres.sh. Dev deployment has not yet flipped this store to Postgres.
Repository implementations Present with first payment-ledger runtime seam src/db/repositories/* and Drizzle schemas exist for the target architecture. Backend 2.8.20 wires fundsLedgerService appends/balance reads through getPaymentRepo(), making that ledger slice controllable by `REPO_PAYMENT=mongo
Oracle quote persistence Conditional runtime PG write /api/payment/request-network/intents lazily imports quoteRepo only when ORACLE_QUOTING_ENABLED=true; it writes payment_quotes if the PG parent payment row exists, mirrors to Mongo Payment.quote, and records pg_dualwrite_gaps if PG is behind.
Backfill/verify scripts Available as operator tooling MIGRATION_PG_URL drives backfill scripts; guards restrict allowed target hosts. The marketplace-core runner group now backfills users/categories, request templates, purchase requests, seller offers, and the post-offer selectedOfferId remap in dependency order. These scripts are not run automatically by app startup.

What Is Still Mongo-Backed

Active app startup, health, and the PG-capable store facades no longer top-load Mongoose models. Auth-owned paths now have an auth-store boundary; confirmation-threshold config, user addresses, categories, level config, shop settings, and reviews have store boundaries, and their PG-capable facades lazy-load legacy Mongo fallbacks instead of top-loading them. Notification runtime paths are repository-backed and now have backfill/smoke tooling, but dev deployment still leaves the store in Mongo mode. Funds ledger appends/balance reads now use the payment repository seam, but default to Mongo unless REPO_PAYMENT is flipped. Broad marketplace requests/offers/templates, most payment paths, points transactions, chat, and admin maintenance actions remain Mongo-first when exercised.

Domain Current live store Why not Postgres yet
Legacy/broad user consumers MongoDB mirror Auth-owned users can be PG-backed, but still-Mongo domains expect Mongo ObjectId user references. PG-mode writes therefore maintain a Mongo mirror until those domains are cut over.
Admin cleanup / seed address tooling MongoDB User-facing address CRUD is PG-capable, but admin cleanup and seed scripts still operate on Mongo first. Backend 2.8.33 makes admin cleanup lazy-load its Mongo models only when cleanup/stat/user-data maintenance actions run. Seed scripts backfill addresses to PG when ADDRESS_STORE=postgres.
Marketplace requests/offers/templates MongoDB Marketplace, checkout, and seller-offer services still call PurchaseRequest, SellerOffer, and RequestTemplate directly. Category and shop-settings reads/writes are PG-capable through facades. PurchaseRequest/SellerOffer/RequestTemplate backfill tooling is now operator-ready, but runtime reads/writes remain Mongo-first and RequestTemplate still has no runtime repo/service wiring.
Payments and escrow state MongoDB primary Request Network, AMN scanner, webhook, admin, release/refund, adapter, reconciliation, and legacy payment paths still create/update Payment Mongoose documents directly. Payment repository methods exist but are not broadly wired into runtime services yet. The SHKeeper migration report lazy-loads Payment and FundsLedgerEntry as of backend 2.8.33, but the report remains Mongo-backed.
Funds ledger Repository-backed, default Mongo appendFundsLedgerEntry and getFundsBalanceBy* now call getPaymentRepo(). In default mode that is MongoPaymentRepo; REPO_PAYMENT=dual/pg can exercise the PG ledger implementation after backfill/soak. Drizzle balance reads support both UUID refs and external/string refs used by template checkout.
Derived destinations and sweeps MongoDB Wallet destination allocation and sweep services call DerivedDestination directly.
Points/referrals/transactions MongoDB Level configuration is PG-capable, but User points fields, PointTransaction, referral aggregation, and Mongo transactions remain Mongo-backed.
Chat/messages MongoDB Chat intentionally remains document-shaped and is not part of the current PG cutover.
Notifications Repository-backed, default Mongo in dev deployment NotificationService routes through getNotificationRepo(), so NOTIFICATION_STORE=postgres / REPO_NOTIFICATION=pg can exercise the Drizzle repo. Backend 2.8.34 adds backfill and smoke coverage. Dev compose/Gatus have not yet been flipped from the seven-store PG baseline to include notification.
Disputes/blog/content/admin cleanup MongoDB Reviews are PG-capable; disputes, blog/content, and admin cleanup still call their Mongoose models directly.
Runtime config outside confirmation thresholds MongoDB ConfigSetting and ConfigSettingHistory are PG-capable for confirmation thresholds only; any future admin-editable settings need to route through the same config-store boundary before they count as cut over.
Telegram link/session/temp verification PG-backed in dev deployment; code default MongoDB These records move with AUTH_STORE=postgres. Dev compose defaults that flag to postgres; environments without the override remain Mongo until the flag is flipped.

Env Flag Reality

The backend code defaults every store flag below to mongo. Dev deployment overrides all seven PG-capable store flags to postgres in deployment/docker-compose.yml as of deployment@38cb75b.

Flag Current meaning
AUTH_STORE Code default mongo; dev deployment default postgres. Routes auth-owned users, refresh tokens, passkeys, Telegram links/sessions, and temp verifications through Postgres.
CONFIG_STORE Code default mongo; dev deployment default postgres. Routes confirmation-threshold settings/history through Postgres.
ADDRESS_STORE Code default mongo; dev deployment default postgres. Routes /api/addresses through Postgres.
CATEGORY_STORE Code default mongo; dev deployment default postgres. Routes marketplace category reads/writes through Postgres. Active PG categories are unique by normalized visible name.
LEVEL_CONFIG_STORE Code default mongo; dev deployment default postgres. Routes level configuration reads and seed replacement through Postgres. LEVEL_STORE=postgres is accepted as a compatibility alias.
SHOP_SETTINGS_STORE Code default mongo; dev deployment default postgres. Routes shop settings, review gates, and seller payment rails through Postgres.
REVIEW_STORE Code default mongo; dev deployment default postgres. Routes marketplace reviews through Postgres.
NOTIFICATION_STORE / REPO_NOTIFICATION Code default mongo; postgres/pg routes notification inbox create/list/read/delete/count through the Drizzle notification repo. Backend 2.8.34 adds npm run backfill:notification:postgres, ordered-runner step notifications, and scripts/smoke/notifications-postgres.sh.
PG_URL Makes PG code importable/reachable. Required for any *_STORE=postgres flag; does not cut over unrelated app domains by itself.
MIGRATION_PG_URL Used by backfill scripts and migration runbooks; not part of normal request handling. Marketplace-core dry-run/non-dry backfills also require MIGRATION_MONGO_URL.
REPO_PAYMENT Code default mongo. As of backend 2.8.20, funds ledger appends and balance reads use this flag through getPaymentRepo(). Do not flip broad payment runtime to pg yet; most payment services still call Mongoose directly.
REPO_USER, REPO_POINTS, REPO_MARKETPLACE, REPO_DEFAULT Repository factory flags exist, but broad services are not yet wired through the factory. Treat them as migration controls that need integration verification before relying on them. The factory lazy-loads PG/dual implementations so importing it in Mongo mode does not require PG_URL.
ORACLE_QUOTING_ENABLED Enables server-side quote computation and the only current PG write path in normal checkout: payment_quotes, when a PG parent row can be resolved.

Next Cutover Work

  1. Apply Drizzle migrations to the target Postgres database.
  2. For dev/test data, either run the existing backfills below or reseed acceptable test data before relying on the PG-backed stores. The deployment default flip does not move historical Mongo rows by itself.
  3. For auth cutover, run PG_URL=... npm run backfill:auth:postgres, verify counts, and confirm AUTH_STORE=postgres in the target runtime.
  4. For confirmation-threshold config cutover, run PG_URL=... npm run backfill:config:postgres, verify counts/history, and confirm CONFIG_STORE=postgres.
  5. For address cutover, run PG_URL=... npm run backfill:address:postgres, verify one-primary invariants, and confirm ADDRESS_STORE=postgres.
  6. For reference-domain cutover, run:
    • PG_URL=... npm run backfill:category:postgres
    • PG_URL=... npm run backfill:level-config:postgres
    • PG_URL=... npm run backfill:shop-settings:postgres
    • PG_URL=... npm run backfill:review:postgres
  7. Run PG_URL=... scripts/smoke/categories-postgres-unique.sh and PG_URL=... MONGODB_URI=... scripts/smoke/reference-stores-postgres.sh, then confirm CATEGORY_STORE=postgres LEVEL_CONFIG_STORE=postgres SHOP_SETTINGS_STORE=postgres REVIEW_STORE=postgres in non-prod.
  8. For marketplace-core data, run MIGRATION_MONGO_URL=... MIGRATION_PG_URL=... npm run backfill:marketplace-core:postgres:dry-run, then the non-dry npm run backfill:marketplace-core:postgres against non-prod. The group runs root dependencies, RequestTemplate rows, PurchaseRequest main rows, SellerOffer rows, then the selected-offer remap.
  9. Run scripts/smoke/marketplace-core-postgres-backfill.sh with the same migration DSNs and record row-count/checksum results.
  10. For notifications, run PG_URL=... npm run backfill:notification:postgres, run PG_URL=... scripts/smoke/notifications-postgres.sh, then flip NOTIFICATION_STORE=postgres in non-prod and update Gatus from seven to eight required PG stores.
  11. Continue payment-domain wiring after the ledger seam: add the missing payment repo methods for provider lookups, transaction-hash/webhook lookups, metadata/blockchain patching, template duplicate cleanup, and quote updates before moving paymentService, paymentCoordinator, RN, or AMN scanner routes.
  12. Add a derived-destination/sweep repository seam before payment PG cutover; destination allocation is payment-address state and should not stay Mongo-only once payments become PG-backed.
  13. Wire remaining services to repository interfaces one domain at a time.
  14. Enable dual mode per large domain only after wiring is proven by tests and smoke checks.
  15. Run shadow-read/reconcile during a soak window.
  16. Flip reads to pg per domain only after zero-diff shadow reads and a rollback plan are in place.

Related Docs

Reference in New Issue View Git Blame Copy Permalink