nick/nick-doc
3
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs: sync from deployment 38cb75b — pg store defaults

Browse Source
This commit is contained in:
Siavash Sameni
2026-06-01 21:40:42 +04:00
parent 8a90bb69be
commit e908cfce63
5 changed files with 57 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
02 - Data Models/MongoDB to PostgreSQL Migration Plan (Drizzle).md
View File

@@ -4,7 +4,7 @@ tags: [data-model, migration, postgres, drizzle, plan, runbook]
aliases: [Drizzle Migration Plan, PG Migration Plan]
created: 2026-05-31
companion: "[[MongoDB to PostgreSQL Migration Guide]]"
updated: 2026-06-01 for backend integrate-main-into-development@c5db471 backend 2.8.19
updated: 2026-06-01 for backend integrate-main-into-development@c5db471 backend 2.8.19 + deployment main@38cb75b
---
# MongoDB → PostgreSQL Migration Plan (Drizzle)
@@ -17,7 +17,7 @@ updated: 2026-06-01 for backend integrate-main-into-development@c5db471 backend
> **Scope reminder:** partial migration (Phases 0–5) is the recommended stopping point — ≈16–28 engineer-weeks. Full migration of Chat/Notification/sessions is explicitly deferred.
> [!warning] Current implementation status
> Backend `2.8.19` has started the runtime cutover with store-specific raw Postgres facades: auth-owned users/Telegram auth records behind `AUTH_STORE=postgres`, confirmation-threshold config/history behind `CONFIG_STORE=postgres`, user address CRUD behind `ADDRESS_STORE=postgres`, and the first marketplace/reference domains behind `CATEGORY_STORE=postgres`, `LEVEL_CONFIG_STORE=postgres`, `SHOP_SETTINGS_STORE=postgres`, and `REVIEW_STORE=postgres`. Category PG mode now deactivates duplicate active names and enforces an active normalized-name unique index. It also contains the broader `src/db/` Drizzle schemas through `0010`, repository implementations/factory, id-map bridge, and backfill runner described below. RequestTemplate now has a PG table/backfill, but broad marketplace/payment/points services are still mostly not wired through that factory. Mongo remains authoritative unless a per-store flag is explicitly flipped. See [[Postgres Runtime Cutover Status]].
> Backend `2.8.19` has started the runtime cutover with store-specific raw Postgres facades: auth-owned users/Telegram auth records behind `AUTH_STORE=postgres`, confirmation-threshold config/history behind `CONFIG_STORE=postgres`, user address CRUD behind `ADDRESS_STORE=postgres`, and the first marketplace/reference domains behind `CATEGORY_STORE=postgres`, `LEVEL_CONFIG_STORE=postgres`, `SHOP_SETTINGS_STORE=postgres`, and `REVIEW_STORE=postgres`. Category PG mode now deactivates duplicate active names and enforces an active normalized-name unique index. It also contains the broader `src/db/` Drizzle schemas through `0010`, repository implementations/factory, id-map bridge, and backfill runner described below. RequestTemplate now has a PG table/backfill, but broad marketplace/payment/points services are still mostly not wired through that factory. Code defaults remain Mongo unless a per-store flag is explicitly flipped; dev deployment `38cb75b` now flips the seven PG-capable store flags to Postgres by default. See [[Postgres Runtime Cutover Status]].
---

63
02 - Data Models/Postgres Runtime Cutover Status.md
View File

@@ -3,14 +3,14 @@ title: Postgres Runtime Cutover Status
tags: [data-model, postgres, migration, runtime-status]
aliases: [Postgres Status, PG Cutover Status, Mongo vs Postgres Runtime]
created: 2026-05-31
source: backend integrate-main-into-development@c5db471
source: backend integrate-main-into-development@c5db471 + deployment main@38cb75b
---
# Postgres Runtime Cutover Status
> **Current branch:** backend `integrate-main-into-development` at `c5db471`, version `2.8.19`.
> **Current branch:** backend `integrate-main-into-development` at `c5db471`, version `2.8.19`; dev deployment `main` at `38cb75b`.
>
> **Bottom line:** this branch is **Postgres-capable**, not fully Postgres-backed. Auth-owned user data can run through Postgres when `AUTH_STORE=postgres`; confirmation-threshold runtime config/history can run through Postgres when `CONFIG_STORE=postgres`; user address CRUD can run through Postgres when `ADDRESS_STORE=postgres`; marketplace categories, level config, shop settings, and reviews can run through Postgres with their own store flags. The category PG path now enforces one active visible category per normalized name. All PG-backed stores require `PG_URL`. Mongo remains the default and the compatibility store for still-Mongo domains.
> **Bottom line:** this branch is **Postgres-capable**, not fully Postgres-backed. Dev deployment now defaults the seven existing PG-capable runtime stores to Postgres: auth-owned users/Telegram auth, confirmation-threshold config/history, user addresses, categories, level config, shop settings, and reviews. Code-level defaults remain Mongo outside that deployment override, and Mongo remains the compatibility store for still-Mongo domains. The category PG path enforces one active visible category per normalized name. All PG-backed stores require `PG_URL`.
## What Uses Postgres Now
@@ -18,14 +18,14 @@ source: backend integrate-main-into-development@c5db471
|---|---|---|
| Postgres connection | Available when `PG_URL` is set | Current store facades use `src/infrastructure/postgres/client.ts`; the broader `src/db/` Drizzle layer and repository factory exist, but most live services are not wired through that factory yet. |
| Runtime schema bootstrap | Implemented for auth, config, address, and reference stores | Auth tables are bootstrapped from `src/services/auth/postgresAuthSchema.ts`; store facades bootstrap their own tables at startup when their `*_STORE=postgres` flag is enabled. |
| Health observability | Implemented in `/api/health` | `checks.postgres` reports `configured`, `required`, `storeModes`, `enabledStores`, and `enabledStoreCount`, so Gatus/operators can verify both PG reachability and which runtime stores are actively PG-backed. |
| Auth-owned user store | Opt-in with `AUTH_STORE=postgres` | Auth, passkey, Telegram auth/link/session/temp-verification, and `/api/user` profile paths use an auth-store facade. In PG mode, users are stored in Postgres and mirrored back to Mongo through `legacy_object_id` for compatibility with still-Mongo services. |
| Confirmation-threshold runtime config | Opt-in with `CONFIG_STORE=postgres` | `ConfigSetting` / `ConfigSettingHistory` access for `/api/admin/settings/confirmation-thresholds` and transaction-safety confirmation thresholds uses a config-store facade. PG-mode writes mirror back to Mongo for rollback. |
| User addresses | Opt-in with `ADDRESS_STORE=postgres` | `/api/addresses` CRUD uses an address-store facade. PG mode enforces one primary address per user with a partial unique index and mirrors writes/deletes back to Mongo for rollback. |
| Marketplace categories | Opt-in with `CATEGORY_STORE=postgres` | `CategoryService` and the default `General` category path use a category-store facade. PG-mode writes mirror back to Mongo for rollback and still-Mongo request/template references. PG schema bootstrap/migration deactivates duplicate active category labels, repoints existing category references to the kept row, and enforces `categories_active_name_norm_uq` on `lower(btrim(name)) WHERE is_active = true`. List/cache reads also dedupe by normalized name. |
| Level configuration | Opt-in with `LEVEL_CONFIG_STORE=postgres` | `PointsService` level reads use a level-config facade. `PointTransaction` and user points remain Mongo-backed. |
| Shop settings | Opt-in with `SHOP_SETTINGS_STORE=postgres` | Shop settings controller, seller payment rail resolution, and review enable/disable checks use a shop-settings facade. PG-mode writes mirror back to Mongo. |
| Marketplace reviews | Opt-in with `REVIEW_STORE=postgres` | Review list/summary/create routes use a review-store facade. PG-mode list responses still hydrate `reviewerId` from the user mirror to preserve frontend shape. |
| Health observability | Implemented in `/api/health` | `checks.postgres` reports `configured`, `required`, `storeModes`, `enabledStores`, and `enabledStoreCount`, so Gatus/operators can verify both PG reachability and which runtime stores are actively PG-backed. Dev Gatus now asserts all seven PG-capable store modes are `postgres`. |
| Auth-owned user store | PG-backed in dev deployment; code opt-in with `AUTH_STORE=postgres` | Auth, passkey, Telegram auth/link/session/temp-verification, and `/api/user` profile paths use an auth-store facade. In PG mode, users are stored in Postgres and mirrored back to Mongo through `legacy_object_id` for compatibility with still-Mongo services. |
| Confirmation-threshold runtime config | PG-backed in dev deployment; code opt-in with `CONFIG_STORE=postgres` | `ConfigSetting` / `ConfigSettingHistory` access for `/api/admin/settings/confirmation-thresholds` and transaction-safety confirmation thresholds uses a config-store facade. PG-mode writes mirror back to Mongo for rollback. |
| User addresses | PG-backed in dev deployment; code opt-in with `ADDRESS_STORE=postgres` | `/api/addresses` CRUD uses an address-store facade. PG mode enforces one primary address per user with a partial unique index and mirrors writes/deletes back to Mongo for rollback. |
| Marketplace categories | PG-backed in dev deployment; code opt-in with `CATEGORY_STORE=postgres` | `CategoryService` and the default `General` category path use a category-store facade. PG-mode writes mirror back to Mongo for rollback and still-Mongo request/template references. PG schema bootstrap/migration deactivates duplicate active category labels, repoints existing category references to the kept row, and enforces `categories_active_name_norm_uq` on `lower(btrim(name)) WHERE is_active = true`. List/cache reads also dedupe by normalized name. |
| Level configuration | PG-backed in dev deployment; code opt-in with `LEVEL_CONFIG_STORE=postgres` | `PointsService` level reads use a level-config facade. `PointTransaction` and user points remain Mongo-backed. |
| Shop settings | PG-backed in dev deployment; code opt-in with `SHOP_SETTINGS_STORE=postgres` | Shop settings controller, seller payment rail resolution, and review enable/disable checks use a shop-settings facade. PG-mode writes mirror back to Mongo. |
| Marketplace reviews | PG-backed in dev deployment; code opt-in with `REVIEW_STORE=postgres` | Review list/summary/create routes use a review-store facade. PG-mode list responses still hydrate `reviewerId` from the user mirror to preserve frontend shape. |
| Repository implementations | Present but partial runtime wiring | `src/db/repositories/*` and Drizzle schemas exist for the target architecture, but this branch's live cutover work is still mostly store-specific raw PG facades plus the conditional oracle quote write path. |
| Oracle quote persistence | Conditional runtime PG write | `/api/payment/request-network/intents` lazily imports `quoteRepo` only when `ORACLE_QUOTING_ENABLED=true`; it writes `payment_quotes` if the PG parent payment row exists, mirrors to Mongo `Payment.quote`, and records `pg_dualwrite_gaps` if PG is behind. |
| Backfill/verify scripts | Available as operator tooling | `MIGRATION_PG_URL` drives backfill scripts; guards restrict allowed target hosts. The marketplace-core runner group now backfills users/categories, request templates, purchase requests, seller offers, and the post-offer `selectedOfferId` remap in dependency order. These scripts are not run automatically by app startup. |
@@ -47,19 +47,21 @@ Most of the service layer still imports Mongoose models directly. Auth-owned pat
| Notifications | MongoDB | Notification TTL/read-state paths remain Mongo-backed. |
| Disputes/blog/content/admin cleanup | MongoDB | Reviews are PG-capable; disputes, blog/content, and admin cleanup still call their Mongoose models directly. |
| Runtime config outside confirmation thresholds | MongoDB | `ConfigSetting` and `ConfigSettingHistory` are PG-capable for confirmation thresholds only; any future admin-editable settings need to route through the same config-store boundary before they count as cut over. |
| Telegram link/session/temp verification | Default MongoDB, PG-capable with auth store | These records move with `AUTH_STORE=postgres`; default runtime remains Mongo until the environment flag is flipped. |
| Telegram link/session/temp verification | PG-backed in dev deployment; code default MongoDB | These records move with `AUTH_STORE=postgres`. Dev compose defaults that flag to `postgres`; environments without the override remain Mongo until the flag is flipped. |
## Env Flag Reality
The backend code defaults every store flag below to `mongo`. Dev deployment overrides all seven PG-capable store flags to `postgres` in `deployment/docker-compose.yml` as of `deployment@38cb75b`.
| Flag | Current meaning |
|---|---|
| `AUTH_STORE` | `mongo` by default. Set `AUTH_STORE=postgres` to route auth-owned users, refresh tokens, passkeys, Telegram links/sessions, and temp verifications through Postgres. |
| `CONFIG_STORE` | `mongo` by default. Set `CONFIG_STORE=postgres` to route confirmation-threshold settings/history through Postgres. |
| `ADDRESS_STORE` | `mongo` by default. Set `ADDRESS_STORE=postgres` to route `/api/addresses` through Postgres. |
| `CATEGORY_STORE` | `mongo` by default. Set `CATEGORY_STORE=postgres` to route marketplace category reads/writes through Postgres. Active PG categories are unique by normalized visible name. |
| `LEVEL_CONFIG_STORE` | `mongo` by default. Set `LEVEL_CONFIG_STORE=postgres` to route level configuration reads and seed replacement through Postgres. `LEVEL_STORE=postgres` is accepted as a compatibility alias. |
| `SHOP_SETTINGS_STORE` | `mongo` by default. Set `SHOP_SETTINGS_STORE=postgres` to route shop settings, review gates, and seller payment rails through Postgres. |
| `REVIEW_STORE` | `mongo` by default. Set `REVIEW_STORE=postgres` to route marketplace reviews through Postgres. |
| `AUTH_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes auth-owned users, refresh tokens, passkeys, Telegram links/sessions, and temp verifications through Postgres. |
| `CONFIG_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes confirmation-threshold settings/history through Postgres. |
| `ADDRESS_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes `/api/addresses` through Postgres. |
| `CATEGORY_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes marketplace category reads/writes through Postgres. Active PG categories are unique by normalized visible name. |
| `LEVEL_CONFIG_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes level configuration reads and seed replacement through Postgres. `LEVEL_STORE=postgres` is accepted as a compatibility alias. |
| `SHOP_SETTINGS_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes shop settings, review gates, and seller payment rails through Postgres. |
| `REVIEW_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes marketplace reviews through Postgres. |
| `PG_URL` | Makes PG code importable/reachable. Required for any `*_STORE=postgres` flag; does not cut over unrelated app domains by itself. |
| `MIGRATION_PG_URL` | Used by backfill scripts and migration runbooks; not part of normal request handling. Marketplace-core dry-run/non-dry backfills also require `MIGRATION_MONGO_URL`. |
| `REPO_USER`, `REPO_PAYMENT`, `REPO_POINTS`, `REPO_MARKETPLACE`, `REPO_DEFAULT` | Repository factory flags exist, but broad services are not yet wired through the factory. Treat them as migration controls that need integration verification before relying on them. |
@@ -68,21 +70,22 @@ Most of the service layer still imports Mongoose models directly. Auth-owned pat
## Next Cutover Work
1. Apply Drizzle migrations to the target Postgres database.
2. For auth cutover, run `PG_URL=... npm run backfill:auth:postgres`, verify counts, then set `AUTH_STORE=postgres`.
3. For confirmation-threshold config cutover, run `PG_URL=... npm run backfill:config:postgres`, verify counts/history, then set `CONFIG_STORE=postgres`.
4. For address cutover, run `PG_URL=... npm run backfill:address:postgres`, verify one-primary invariants, then set `ADDRESS_STORE=postgres`.
5. For reference-domain cutover, run:
2. For dev/test data, either run the existing backfills below or reseed acceptable test data before relying on the PG-backed stores. The deployment default flip does not move historical Mongo rows by itself.
3. For auth cutover, run `PG_URL=... npm run backfill:auth:postgres`, verify counts, and confirm `AUTH_STORE=postgres` in the target runtime.
4. For confirmation-threshold config cutover, run `PG_URL=... npm run backfill:config:postgres`, verify counts/history, and confirm `CONFIG_STORE=postgres`.
5. For address cutover, run `PG_URL=... npm run backfill:address:postgres`, verify one-primary invariants, and confirm `ADDRESS_STORE=postgres`.
6. For reference-domain cutover, run:
- `PG_URL=... npm run backfill:category:postgres`
- `PG_URL=... npm run backfill:level-config:postgres`
- `PG_URL=... npm run backfill:shop-settings:postgres`
- `PG_URL=... npm run backfill:review:postgres`
6. Run `PG_URL=... scripts/smoke/categories-postgres-unique.sh` and `PG_URL=... MONGODB_URI=... scripts/smoke/reference-stores-postgres.sh`, then set `CATEGORY_STORE=postgres LEVEL_CONFIG_STORE=postgres SHOP_SETTINGS_STORE=postgres REVIEW_STORE=postgres` together in non-prod.
7. For marketplace-core data, run `MIGRATION_MONGO_URL=... MIGRATION_PG_URL=... npm run backfill:marketplace-core:postgres:dry-run`, then the non-dry `npm run backfill:marketplace-core:postgres` against non-prod. The group runs root dependencies, RequestTemplate rows, PurchaseRequest main rows, SellerOffer rows, then the selected-offer remap.
8. Run `scripts/smoke/marketplace-core-postgres-backfill.sh` with the same migration DSNs and record row-count/checksum results.
9. Wire remaining services to repository interfaces one domain at a time.
10. Enable `dual` mode per large domain only after wiring is proven by tests and smoke checks.
11. Run shadow-read/reconcile during a soak window.
12. Flip reads to `pg` per domain only after zero-diff shadow reads and a rollback plan are in place.
7. Run `PG_URL=... scripts/smoke/categories-postgres-unique.sh` and `PG_URL=... MONGODB_URI=... scripts/smoke/reference-stores-postgres.sh`, then confirm `CATEGORY_STORE=postgres LEVEL_CONFIG_STORE=postgres SHOP_SETTINGS_STORE=postgres REVIEW_STORE=postgres` in non-prod.
8. For marketplace-core data, run `MIGRATION_MONGO_URL=... MIGRATION_PG_URL=... npm run backfill:marketplace-core:postgres:dry-run`, then the non-dry `npm run backfill:marketplace-core:postgres` against non-prod. The group runs root dependencies, RequestTemplate rows, PurchaseRequest main rows, SellerOffer rows, then the selected-offer remap.
9. Run `scripts/smoke/marketplace-core-postgres-backfill.sh` with the same migration DSNs and record row-count/checksum results.
10. Wire remaining services to repository interfaces one domain at a time.
11. Enable `dual` mode per large domain only after wiring is proven by tests and smoke checks.
12. Run shadow-read/reconcile during a soak window.
13. Flip reads to `pg` per domain only after zero-diff shadow reads and a rollback plan are in place.
## Related Docs