Files
nick-doc/02 - Data Models/Postgres Runtime Cutover Status.md
2026-06-01 17:22:53 +04:00

95 lines
12 KiB
Markdown

---
title: Postgres Runtime Cutover Status
tags: [data-model, postgres, migration, runtime-status]
aliases: [Postgres Status, PG Cutover Status, Mongo vs Postgres Runtime]
created: 2026-05-31
source: backend integrate-main-into-development@1543b53
---
# Postgres Runtime Cutover Status
> **Current branch:** backend `integrate-main-into-development` at `1543b53`, version `2.8.17`.
>
> **Bottom line:** this branch is **Postgres-capable**, not fully Postgres-backed. Auth-owned user data can run through Postgres when `AUTH_STORE=postgres`; confirmation-threshold runtime config/history can run through Postgres when `CONFIG_STORE=postgres`; user address CRUD can run through Postgres when `ADDRESS_STORE=postgres`; marketplace categories, level config, shop settings, and reviews can run through Postgres with their own store flags. The category PG path now enforces one active visible category per normalized name. All PG-backed stores require `PG_URL`. Mongo remains the default and the compatibility store for still-Mongo domains.
## What Uses Postgres Now
| Area | Runtime status | Notes |
|---|---|---|
| Postgres connection | Available when `PG_URL` is set | Current store facades use `src/infrastructure/postgres/client.ts`; the broader `src/db/` Drizzle layer and repository factory exist, but most live services are not wired through that factory yet. |
| Runtime schema bootstrap | Implemented for auth, config, address, and reference stores | Auth tables are bootstrapped from `src/services/auth/postgresAuthSchema.ts`; store facades bootstrap their own tables at startup when their `*_STORE=postgres` flag is enabled. |
| Health observability | Implemented in `/api/health` | `checks.postgres` reports `configured`, `required`, `storeModes`, `enabledStores`, and `enabledStoreCount`, so Gatus/operators can verify both PG reachability and which runtime stores are actively PG-backed. |
| Auth-owned user store | Opt-in with `AUTH_STORE=postgres` | Auth, passkey, Telegram auth/link/session/temp-verification, and `/api/user` profile paths use an auth-store facade. In PG mode, users are stored in Postgres and mirrored back to Mongo through `legacy_object_id` for compatibility with still-Mongo services. |
| Confirmation-threshold runtime config | Opt-in with `CONFIG_STORE=postgres` | `ConfigSetting` / `ConfigSettingHistory` access for `/api/admin/settings/confirmation-thresholds` and transaction-safety confirmation thresholds uses a config-store facade. PG-mode writes mirror back to Mongo for rollback. |
| User addresses | Opt-in with `ADDRESS_STORE=postgres` | `/api/addresses` CRUD uses an address-store facade. PG mode enforces one primary address per user with a partial unique index and mirrors writes/deletes back to Mongo for rollback. |
| Marketplace categories | Opt-in with `CATEGORY_STORE=postgres` | `CategoryService` and the default `General` category path use a category-store facade. PG-mode writes mirror back to Mongo for rollback and still-Mongo request/template references. PG schema bootstrap/migration deactivates duplicate active category labels, repoints existing category references to the kept row, and enforces `categories_active_name_norm_uq` on `lower(btrim(name)) WHERE is_active = true`. List/cache reads also dedupe by normalized name. |
| Level configuration | Opt-in with `LEVEL_CONFIG_STORE=postgres` | `PointsService` level reads use a level-config facade. `PointTransaction` and user points remain Mongo-backed. |
| Shop settings | Opt-in with `SHOP_SETTINGS_STORE=postgres` | Shop settings controller, seller payment rail resolution, and review enable/disable checks use a shop-settings facade. PG-mode writes mirror back to Mongo. |
| Marketplace reviews | Opt-in with `REVIEW_STORE=postgres` | Review list/summary/create routes use a review-store facade. PG-mode list responses still hydrate `reviewerId` from the user mirror to preserve frontend shape. |
| Repository implementations | Present but partial runtime wiring | `src/db/repositories/*` and Drizzle schemas exist for the target architecture, but this branch's live cutover work is still mostly store-specific raw PG facades plus the conditional oracle quote write path. |
| Oracle quote persistence | Conditional runtime PG write | `/api/payment/request-network/intents` lazily imports `quoteRepo` only when `ORACLE_QUOTING_ENABLED=true`; it writes `payment_quotes` if the PG parent payment row exists, mirrors to Mongo `Payment.quote`, and records `pg_dualwrite_gaps` if PG is behind. |
| Backfill/verify scripts | Available as operator tooling | `MIGRATION_PG_URL` drives backfill scripts; guards restrict allowed target hosts. The marketplace-core runner group now backfills users/categories, purchase requests, seller offers, and the post-offer `selectedOfferId` remap in dependency order. These scripts are not run automatically by app startup. |
## What Is Still Mongo-Backed
Most of the service layer still imports Mongoose models directly. Auth-owned paths now have an auth-store boundary; confirmation-threshold config, user addresses, categories, level config, shop settings, and reviews have store boundaries. Broad marketplace requests/offers/templates, payment, funds ledger, points transactions, chat, notification, and admin paths remain Mongo-first.
| Domain | Current live store | Why not Postgres yet |
|---|---|---|
| Legacy/broad user consumers | MongoDB mirror | Auth-owned users can be PG-backed, but still-Mongo domains expect Mongo ObjectId user references. PG-mode writes therefore maintain a Mongo mirror until those domains are cut over. |
| Admin cleanup / seed address tooling | MongoDB | User-facing address CRUD is PG-capable, but admin cleanup and seed scripts still operate on Mongo first. Seed scripts backfill addresses to PG when `ADDRESS_STORE=postgres`. |
| Marketplace requests/offers/templates | MongoDB | Marketplace, checkout, and seller-offer services still call `PurchaseRequest`, `SellerOffer`, and `RequestTemplate` directly. Category and shop-settings reads/writes are PG-capable through facades. PurchaseRequest/SellerOffer backfill tooling is now operator-ready, but runtime reads/writes remain Mongo-first and RequestTemplate still has no PG table/repository. |
| Payments and escrow state | MongoDB primary | Request Network, AMN scanner, webhook, admin, release/refund, adapter, reconciliation, and legacy payment paths still create/update `Payment` Mongoose documents directly. |
| Funds ledger | MongoDB primary | `FundsLedgerEntry` remains the ledger used by current services; PG ledger tables exist but are not the live write target. |
| Derived destinations and sweeps | MongoDB | Wallet destination allocation and sweep services call `DerivedDestination` directly. |
| Points/referrals/transactions | MongoDB | Level configuration is PG-capable, but `User` points fields, `PointTransaction`, referral aggregation, and Mongo transactions remain Mongo-backed. |
| Chat/messages | MongoDB | Chat intentionally remains document-shaped and is not part of the current PG cutover. |
| Notifications | MongoDB | Notification TTL/read-state paths remain Mongo-backed. |
| Disputes/blog/content/admin cleanup | MongoDB | Reviews are PG-capable; disputes, blog/content, and admin cleanup still call their Mongoose models directly. |
| Runtime config outside confirmation thresholds | MongoDB | `ConfigSetting` and `ConfigSettingHistory` are PG-capable for confirmation thresholds only; any future admin-editable settings need to route through the same config-store boundary before they count as cut over. |
| Telegram link/session/temp verification | Default MongoDB, PG-capable with auth store | These records move with `AUTH_STORE=postgres`; default runtime remains Mongo until the environment flag is flipped. |
## Env Flag Reality
| Flag | Current meaning |
|---|---|
| `AUTH_STORE` | `mongo` by default. Set `AUTH_STORE=postgres` to route auth-owned users, refresh tokens, passkeys, Telegram links/sessions, and temp verifications through Postgres. |
| `CONFIG_STORE` | `mongo` by default. Set `CONFIG_STORE=postgres` to route confirmation-threshold settings/history through Postgres. |
| `ADDRESS_STORE` | `mongo` by default. Set `ADDRESS_STORE=postgres` to route `/api/addresses` through Postgres. |
| `CATEGORY_STORE` | `mongo` by default. Set `CATEGORY_STORE=postgres` to route marketplace category reads/writes through Postgres. Active PG categories are unique by normalized visible name. |
| `LEVEL_CONFIG_STORE` | `mongo` by default. Set `LEVEL_CONFIG_STORE=postgres` to route level configuration reads and seed replacement through Postgres. `LEVEL_STORE=postgres` is accepted as a compatibility alias. |
| `SHOP_SETTINGS_STORE` | `mongo` by default. Set `SHOP_SETTINGS_STORE=postgres` to route shop settings, review gates, and seller payment rails through Postgres. |
| `REVIEW_STORE` | `mongo` by default. Set `REVIEW_STORE=postgres` to route marketplace reviews through Postgres. |
| `PG_URL` | Makes PG code importable/reachable. Required for any `*_STORE=postgres` flag; does not cut over unrelated app domains by itself. |
| `MIGRATION_PG_URL` | Used by backfill scripts and migration runbooks; not part of normal request handling. Marketplace-core dry-run/non-dry backfills also require `MIGRATION_MONGO_URL`. |
| `REPO_USER`, `REPO_PAYMENT`, `REPO_POINTS`, `REPO_MARKETPLACE`, `REPO_DEFAULT` | Repository factory flags exist, but broad services are not yet wired through the factory. Treat them as migration controls that need integration verification before relying on them. |
| `ORACLE_QUOTING_ENABLED` | Enables server-side quote computation and the only current PG write path in normal checkout: `payment_quotes`, when a PG parent row can be resolved. |
## Next Cutover Work
1. Apply Drizzle migrations to the target Postgres database.
2. For auth cutover, run `PG_URL=... npm run backfill:auth:postgres`, verify counts, then set `AUTH_STORE=postgres`.
3. For confirmation-threshold config cutover, run `PG_URL=... npm run backfill:config:postgres`, verify counts/history, then set `CONFIG_STORE=postgres`.
4. For address cutover, run `PG_URL=... npm run backfill:address:postgres`, verify one-primary invariants, then set `ADDRESS_STORE=postgres`.
5. For reference-domain cutover, run:
- `PG_URL=... npm run backfill:category:postgres`
- `PG_URL=... npm run backfill:level-config:postgres`
- `PG_URL=... npm run backfill:shop-settings:postgres`
- `PG_URL=... npm run backfill:review:postgres`
6. Run `PG_URL=... scripts/smoke/categories-postgres-unique.sh` and `PG_URL=... MONGODB_URI=... scripts/smoke/reference-stores-postgres.sh`, then set `CATEGORY_STORE=postgres LEVEL_CONFIG_STORE=postgres SHOP_SETTINGS_STORE=postgres REVIEW_STORE=postgres` together in non-prod.
7. For marketplace-core data, run `MIGRATION_MONGO_URL=... MIGRATION_PG_URL=... npm run backfill:marketplace-core:postgres:dry-run`, then the non-dry `npm run backfill:marketplace-core:postgres` against non-prod. The group runs root dependencies, PurchaseRequest main rows, SellerOffer rows, then the selected-offer remap.
8. Run `scripts/smoke/marketplace-core-postgres-backfill.sh` with the same migration DSNs and record row-count/checksum results.
9. Wire remaining services to repository interfaces one domain at a time.
10. Enable `dual` mode per large domain only after wiring is proven by tests and smoke checks.
11. Run shadow-read/reconcile during a soak window.
12. Flip reads to `pg` per domain only after zero-diff shadow reads and a rollback plan are in place.
## Related Docs
- [[Database Strategy - Mongo vs Postgres Assessment]]
- [[MongoDB to PostgreSQL Migration Plan (Drizzle)]]
- [[Payment]]
- [[Payment API]]
- [[Environment Variables]]
- [[Database Operations]]