From 85fe50aca06ea07e50f8a198953dd2885cb8ce44 Mon Sep 17 00:00:00 2001 From: Siavash Sameni Date: Tue, 2 Jun 2026 12:33:58 +0400 Subject: [PATCH] =?UTF-8?q?docs:=20sync=20from=20backend=20882096f=20?= =?UTF-8?q?=E2=80=94=20notification=20pg=20dev=20cutover?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../Postgres Runtime Cutover Status.md | 30 +++++++++---------- 09 - Audits/Activity Log.md | 12 ++++++++ 2 files changed, 27 insertions(+), 15 deletions(-) diff --git a/02 - Data Models/Postgres Runtime Cutover Status.md b/02 - Data Models/Postgres Runtime Cutover Status.md index 3098beb..51010e2 100644 --- a/02 - Data Models/Postgres Runtime Cutover Status.md +++ b/02 - Data Models/Postgres Runtime Cutover Status.md @@ -3,14 +3,14 @@ title: Postgres Runtime Cutover Status tags: [data-model, postgres, migration, runtime-status] aliases: [Postgres Status, PG Cutover Status, Mongo vs Postgres Runtime] created: 2026-05-31 -source: backend integrate-main-into-development@f1ba14b + frontend integrate-main-into-development@b94d8a9 + deployment main@38cb75b +source: backend integrate-main-into-development@882096f + frontend integrate-main-into-development@28ad8e6 + deployment main@8764fdf --- # Postgres Runtime Cutover Status -> **Current branch:** backend `integrate-main-into-development` at `f1ba14b`, version `2.8.34`; frontend `integrate-main-into-development` at `b94d8a9`, version `2.8.34`; dev deployment `main` at `38cb75b`. +> **Current branch:** backend `integrate-main-into-development` at `882096f`, version `2.8.36`; frontend `integrate-main-into-development` at `28ad8e6`, version `2.8.36`; dev deployment `main` at `8764fdf`. > -> **Bottom line:** this branch is **Postgres-capable**, not fully Postgres-backed. Dev deployment now defaults the seven existing PG-capable runtime stores to Postgres: auth-owned users/Telegram auth, confirmation-threshold config/history, user addresses, categories, level config, shop settings, and reviews. Code-level defaults remain Mongo outside that deployment override, and Mongo remains the compatibility store for still-Mongo domains. The category PG path enforces one active visible category per normalized name. As of backend `2.8.33`, the active startup/health/admin/report import surface no longer has non-type top-level `mongoose` or `models/*` imports; legacy Mongo models are lazy-loaded only when fallback/backfill/maintenance actions run. All PG-backed stores require `PG_URL`. +> **Bottom line:** this branch is **Postgres-capable**, not fully Postgres-backed. Dev deployment now defaults eight existing PG-capable runtime stores to Postgres: auth-owned users/Telegram auth, confirmation-threshold config/history, user addresses, categories, level config, shop settings, reviews, and notifications. Code-level defaults remain Mongo outside that deployment override, and Mongo remains the compatibility store for still-Mongo domains. The category PG path enforces one active visible category per normalized name. As of backend `2.8.36`, the active startup/health/admin/report import surface no longer has non-type top-level `mongoose` or `models/*` imports, and the unmounted legacy marketplace router is no longer re-exported from the marketplace service index; legacy Mongo models are lazy-loaded only when fallback/backfill/maintenance actions run. All PG-backed stores require `PG_URL`. ## What Uses Postgres Now @@ -18,7 +18,7 @@ source: backend integrate-main-into-development@f1ba14b + frontend integrate-mai |---|---|---| | Postgres connection | Available when `PG_URL` is set | Current store facades use `src/infrastructure/postgres/client.ts`; the broader `src/db/` Drizzle layer and repository factory exist, but most live services are not wired through that factory yet. | | Runtime schema bootstrap | Implemented for auth, config, address, and reference stores | Auth tables are bootstrapped from `src/services/auth/postgresAuthSchema.ts`; store facades bootstrap their own tables at startup when their `*_STORE=postgres` flag is enabled. | -| Health observability | Implemented in `/api/health` | `checks.postgres` reports `configured`, `required`, `storeModes`, `enabledStores`, and `enabledStoreCount`, so Gatus/operators can verify both PG reachability and which runtime stores are actively PG-backed. Dev Gatus now asserts all seven PG-capable store modes are `postgres`. Backend `2.8.33` lazy-loads Mongoose for the legacy Mongo health check and skips it when Mongo is optional under `MONGO_CONNECT_MODE=auto/never`. | +| Health observability | Implemented in `/api/health` | `checks.postgres` reports `configured`, `required`, `storeModes`, `enabledStores`, and `enabledStoreCount`, so Gatus/operators can verify both PG reachability and which runtime stores are actively PG-backed. Dev Gatus now asserts all eight dev PG-backed store modes are `postgres`, including notifications. Backend `2.8.33` lazy-loads Mongoose for the legacy Mongo health check and skips it when Mongo is optional under `MONGO_CONNECT_MODE=auto/never`. | | Auth-owned user store | PG-backed in dev deployment; code opt-in with `AUTH_STORE=postgres` | Auth, passkey, Telegram auth/link/session/temp-verification, and `/api/user` profile paths use an auth-store facade. In PG mode, users are stored in Postgres and mirrored back to Mongo through `legacy_object_id` for compatibility with still-Mongo services. | | Confirmation-threshold runtime config | PG-backed in dev deployment; code opt-in with `CONFIG_STORE=postgres` | `ConfigSetting` / `ConfigSettingHistory` access for `/api/admin/settings/confirmation-thresholds` and transaction-safety confirmation thresholds uses a config-store facade. PG-mode writes mirror back to Mongo for rollback. Backend `2.8.32` removed top-level Mongo model imports from this facade; legacy models load only for Mongo fallback/backfill/mirror paths. | | User addresses | PG-backed in dev deployment; code opt-in with `ADDRESS_STORE=postgres` | `/api/addresses` CRUD uses an address-store facade. PG mode enforces one primary address per user with a partial unique index and mirrors writes/deletes back to Mongo for rollback. | @@ -26,33 +26,33 @@ source: backend integrate-main-into-development@f1ba14b + frontend integrate-mai | Level configuration | PG-backed in dev deployment; code opt-in with `LEVEL_CONFIG_STORE=postgres` | `PointsService` level reads use a level-config facade. `PointTransaction` and user points remain Mongo-backed. | | Shop settings | PG-backed in dev deployment; code opt-in with `SHOP_SETTINGS_STORE=postgres` | Shop settings controller, seller payment rail resolution, and review enable/disable checks use a shop-settings facade. PG-mode writes mirror back to Mongo. Backend `2.8.32` removed top-level Mongo model imports from this facade; legacy models load only for Mongo fallback/backfill/mirror paths. | | Marketplace reviews | PG-backed in dev deployment; code opt-in with `REVIEW_STORE=postgres` | Review list/summary/create routes use a review-store facade. PG-mode list responses still hydrate `reviewerId` from the user mirror to preserve frontend shape. Backend `2.8.32` removed top-level Mongo model imports from this facade; legacy models load only for Mongo fallback/backfill/mirror paths. | -| Notifications | Repository-backed, code opt-in with `NOTIFICATION_STORE=postgres` or `REPO_NOTIFICATION=pg` | `NotificationService` uses `getNotificationRepo()` for create/list/read/delete/count paths. Backend `2.8.34` adds Mongo→Postgres notification backfill tooling, ordered-runner support, a dry-run path, and `scripts/smoke/notifications-postgres.sh`. Dev deployment has not yet flipped this store to Postgres. | +| Notifications | PG-backed in dev deployment; code opt-in with `NOTIFICATION_STORE=postgres` or `REPO_NOTIFICATION=pg` | `NotificationService` uses `getNotificationRepo()` for create/list/read/delete/count paths. Backend `2.8.34` adds Mongo→Postgres notification backfill tooling, ordered-runner support, a dry-run path, and `scripts/smoke/notifications-postgres.sh`. Deployment `8764fdf` defaults `NOTIFICATION_STORE=postgres` in dev and Gatus requires the notification store mode. | | Repository implementations | Present with first payment-ledger runtime seam | `src/db/repositories/*` and Drizzle schemas exist for the target architecture. Backend `2.8.20` wires `fundsLedgerService` appends/balance reads through `getPaymentRepo()`, making that ledger slice controllable by `REPO_PAYMENT=mongo|dual|pg`. The broader payment, marketplace, and points services still need method-by-method service wiring before their repo flags are safe runtime cutovers. | | Oracle quote persistence | Conditional runtime PG write | `/api/payment/request-network/intents` lazily imports `quoteRepo` only when `ORACLE_QUOTING_ENABLED=true`; it writes `payment_quotes` if the PG parent payment row exists, mirrors to Mongo `Payment.quote`, and records `pg_dualwrite_gaps` if PG is behind. | | Backfill/verify scripts | Available as operator tooling | `MIGRATION_PG_URL` drives backfill scripts; guards restrict allowed target hosts. The marketplace-core runner group now backfills users/categories, request templates, purchase requests, seller offers, and the post-offer `selectedOfferId` remap in dependency order. These scripts are not run automatically by app startup. | ## What Is Still Mongo-Backed -Active app startup, health, and the PG-capable store facades no longer top-load Mongoose models. Auth-owned paths now have an auth-store boundary; confirmation-threshold config, user addresses, categories, level config, shop settings, and reviews have store boundaries, and their PG-capable facades lazy-load legacy Mongo fallbacks instead of top-loading them. Notification runtime paths are repository-backed and now have backfill/smoke tooling, but dev deployment still leaves the store in Mongo mode. Funds ledger appends/balance reads now use the payment repository seam, but default to Mongo unless `REPO_PAYMENT` is flipped. Broad marketplace requests/offers/templates, most payment paths, points transactions, chat, and admin maintenance actions remain Mongo-first when exercised. +Active app startup, health, and the PG-capable store facades no longer top-load Mongoose models. Auth-owned paths now have an auth-store boundary; confirmation-threshold config, user addresses, categories, level config, shop settings, reviews, and notifications have store/repository boundaries, and their PG-capable facades lazy-load legacy Mongo fallbacks instead of top-loading them. Funds ledger appends/balance reads now use the payment repository seam, but default to Mongo unless `REPO_PAYMENT` is flipped. Broad marketplace requests/offers/templates, most payment paths, points transactions, chat, and admin maintenance actions remain Mongo-first when exercised. | Domain | Current live store | Why not Postgres yet | |---|---|---| | Legacy/broad user consumers | MongoDB mirror | Auth-owned users can be PG-backed, but still-Mongo domains expect Mongo ObjectId user references. PG-mode writes therefore maintain a Mongo mirror until those domains are cut over. | | Admin cleanup / seed address tooling | MongoDB | User-facing address CRUD is PG-capable, but admin cleanup and seed scripts still operate on Mongo first. Backend `2.8.33` makes admin cleanup lazy-load its Mongo models only when cleanup/stat/user-data maintenance actions run. Seed scripts backfill addresses to PG when `ADDRESS_STORE=postgres`. | -| Marketplace requests/offers/templates | MongoDB | Marketplace, checkout, and seller-offer services still call `PurchaseRequest`, `SellerOffer`, and `RequestTemplate` directly. Category and shop-settings reads/writes are PG-capable through facades. PurchaseRequest/SellerOffer/RequestTemplate backfill tooling is now operator-ready, but runtime reads/writes remain Mongo-first and RequestTemplate still has no runtime repo/service wiring. | +| Marketplace requests/offers/templates | Repository-backed, default Mongo | Controller/service paths route through `getMarketplaceRepo()`, and PurchaseRequest/SellerOffer/RequestTemplate backfill tooling is operator-ready. `REPO_MARKETPLACE` still defaults to Mongo, the old unmounted route file remains present for now, and full PG/dual marketplace runtime cutover still needs route/service smoke coverage before flipping. Backend `2.8.36` stops re-exporting the old Mongo-heavy `marketplaceRouter` from the marketplace service index. | | Payments and escrow state | MongoDB primary | Request Network, AMN scanner, webhook, admin, release/refund, adapter, reconciliation, and legacy payment paths still create/update `Payment` Mongoose documents directly. Payment repository methods exist but are not broadly wired into runtime services yet. The SHKeeper migration report lazy-loads `Payment` and `FundsLedgerEntry` as of backend `2.8.33`, but the report remains Mongo-backed. | | Funds ledger | Repository-backed, default Mongo | `appendFundsLedgerEntry` and `getFundsBalanceBy*` now call `getPaymentRepo()`. In default mode that is `MongoPaymentRepo`; `REPO_PAYMENT=dual`/`pg` can exercise the PG ledger implementation after backfill/soak. Drizzle balance reads support both UUID refs and external/string refs used by template checkout. | -| Derived destinations and sweeps | MongoDB | Wallet destination allocation and sweep services call `DerivedDestination` directly. | -| Points/referrals/transactions | MongoDB | Level configuration is PG-capable, but `User` points fields, `PointTransaction`, referral aggregation, and Mongo transactions remain Mongo-backed. | -| Chat/messages | MongoDB | Chat intentionally remains document-shaped and is not part of the current PG cutover. | -| Notifications | Repository-backed, default Mongo in dev deployment | `NotificationService` routes through `getNotificationRepo()`, so `NOTIFICATION_STORE=postgres` / `REPO_NOTIFICATION=pg` can exercise the Drizzle repo. Backend `2.8.34` adds backfill and smoke coverage. Dev compose/Gatus have not yet been flipped from the seven-store PG baseline to include notification. | -| Disputes/blog/content/admin cleanup | MongoDB | Reviews are PG-capable; disputes, blog/content, and admin cleanup still call their Mongoose models directly. | +| Derived destinations and sweeps | Repository-backed, default Mongo | Wallet destination allocation and sweep paths use `getDerivedDestinationRepo()` / `getPaymentRepo()`, but `REPO_DERIVED_DESTINATION` still defaults to Mongo and has not been flipped in dev. | +| Points/referrals/transactions | Repository-backed, default Mongo | `PointsService` uses `getPointsRepo()` and level configuration is PG-capable, but `REPO_POINTS` defaults to Mongo and point transaction/user-point flows have not been flipped in dev. | +| Chat/messages | Repository-backed, default Mongo | Chat service uses `getChatRepo()`, but `REPO_CHAT` / `CHAT_STORE` defaults to Mongo and chat is still treated as a document-shaped domain until a deliberate PG cutover. | +| Notifications | PG-backed in dev deployment; code default Mongo | `NotificationService` routes through `getNotificationRepo()`, so `NOTIFICATION_STORE=postgres` / `REPO_NOTIFICATION=pg` can exercise the Drizzle repo. Backend `2.8.34` adds backfill and smoke coverage; deployment `8764fdf` includes notification in the dev PG baseline. | +| Disputes/blog/content/admin cleanup | Mixed | Disputes and blog are repository-backed, but their code defaults still resolve to Mongo until `REPO_DISPUTE`/`BLOG_STORE` are flipped. Admin cleanup still lazy-loads and calls Mongoose models directly for maintenance cleanup/stat/user-data actions. | | Runtime config outside confirmation thresholds | MongoDB | `ConfigSetting` and `ConfigSettingHistory` are PG-capable for confirmation thresholds only; any future admin-editable settings need to route through the same config-store boundary before they count as cut over. | | Telegram link/session/temp verification | PG-backed in dev deployment; code default MongoDB | These records move with `AUTH_STORE=postgres`. Dev compose defaults that flag to `postgres`; environments without the override remain Mongo until the flag is flipped. | ## Env Flag Reality -The backend code defaults every store flag below to `mongo`. Dev deployment overrides all seven PG-capable store flags to `postgres` in `deployment/docker-compose.yml` as of `deployment@38cb75b`. +The backend code defaults every store flag below to `mongo`. Dev deployment overrides eight PG-capable store flags to `postgres` in `deployment/docker-compose.yml` as of `deployment@8764fdf`. | Flag | Current meaning | |---|---| @@ -63,7 +63,7 @@ The backend code defaults every store flag below to `mongo`. Dev deployment over | `LEVEL_CONFIG_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes level configuration reads and seed replacement through Postgres. `LEVEL_STORE=postgres` is accepted as a compatibility alias. | | `SHOP_SETTINGS_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes shop settings, review gates, and seller payment rails through Postgres. | | `REVIEW_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes marketplace reviews through Postgres. | -| `NOTIFICATION_STORE` / `REPO_NOTIFICATION` | Code default `mongo`; `postgres`/`pg` routes notification inbox create/list/read/delete/count through the Drizzle notification repo. Backend `2.8.34` adds `npm run backfill:notification:postgres`, ordered-runner step `notifications`, and `scripts/smoke/notifications-postgres.sh`. | +| `NOTIFICATION_STORE` / `REPO_NOTIFICATION` | Code default `mongo`; dev deployment default `postgres`. Routes notification inbox create/list/read/delete/count through the Drizzle notification repo. Backend `2.8.34` adds `npm run backfill:notification:postgres`, ordered-runner step `notifications`, and `scripts/smoke/notifications-postgres.sh`. | | `PG_URL` | Makes PG code importable/reachable. Required for any `*_STORE=postgres` flag; does not cut over unrelated app domains by itself. | | `MIGRATION_PG_URL` | Used by backfill scripts and migration runbooks; not part of normal request handling. Marketplace-core dry-run/non-dry backfills also require `MIGRATION_MONGO_URL`. | | `REPO_PAYMENT` | Code default `mongo`. As of backend `2.8.20`, funds ledger appends and balance reads use this flag through `getPaymentRepo()`. Do not flip broad payment runtime to `pg` yet; most payment services still call Mongoose directly. | @@ -85,7 +85,7 @@ The backend code defaults every store flag below to `mongo`. Dev deployment over 7. Run `PG_URL=... scripts/smoke/categories-postgres-unique.sh` and `PG_URL=... MONGODB_URI=... scripts/smoke/reference-stores-postgres.sh`, then confirm `CATEGORY_STORE=postgres LEVEL_CONFIG_STORE=postgres SHOP_SETTINGS_STORE=postgres REVIEW_STORE=postgres` in non-prod. 8. For marketplace-core data, run `MIGRATION_MONGO_URL=... MIGRATION_PG_URL=... npm run backfill:marketplace-core:postgres:dry-run`, then the non-dry `npm run backfill:marketplace-core:postgres` against non-prod. The group runs root dependencies, RequestTemplate rows, PurchaseRequest main rows, SellerOffer rows, then the selected-offer remap. 9. Run `scripts/smoke/marketplace-core-postgres-backfill.sh` with the same migration DSNs and record row-count/checksum results. -10. For notifications, run `PG_URL=... npm run backfill:notification:postgres`, run `PG_URL=... scripts/smoke/notifications-postgres.sh`, then flip `NOTIFICATION_STORE=postgres` in non-prod and update Gatus from seven to eight required PG stores. +10. For notifications, run `PG_URL=... npm run backfill:notification:postgres` and `PG_URL=... scripts/smoke/notifications-postgres.sh` against dev to validate the new default. 11. Continue payment-domain wiring after the ledger seam: add the missing payment repo methods for provider lookups, transaction-hash/webhook lookups, metadata/blockchain patching, template duplicate cleanup, and quote updates before moving `paymentService`, `paymentCoordinator`, RN, or AMN scanner routes. 12. Add a derived-destination/sweep repository seam before payment PG cutover; destination allocation is payment-address state and should not stay Mongo-only once payments become PG-backed. 13. Wire remaining services to repository interfaces one domain at a time. diff --git a/09 - Audits/Activity Log.md b/09 - Audits/Activity Log.md index d3dc6aa..e0317ad 100644 --- a/09 - Audits/Activity Log.md +++ b/09 - Audits/Activity Log.md @@ -11,6 +11,18 @@ entries on top. Maintained by agents per the rule in `../AGENTS.md`. --- +### 2026-06-02 — backend@882096f, deployment@8764fdf — enable notification Postgres dev cutover + +**Commits:** backend `b64995a` and `882096f` (version `2.8.36`), frontend `28ad8e6` (version `2.8.36`, already on remote), deployment `8764fdf` +**Touched:** +- Backend: `src/services/marketplace/index.ts`, `__tests__/marketplace-runtime-import-surface.test.ts`, `package.json`, `package-lock.json` +- Deployment: `docker-compose.yml`, `gatus/config.yaml` +**Why:** Continue MongoDB removal by detaching the unmounted legacy marketplace router export that still pulled in top-level Mongoose/model imports, and move notifications from PG-capable/operator-ready to PG-backed in the dev runtime. Dev Gatus now requires eight enabled Postgres stores and explicitly asserts `storeModes.notification == "postgres"`. +**Verification:** Backend `npm run typecheck`; backend `npm test -- --runTestsByPath __tests__/marketplace-runtime-import-surface.test.ts __tests__/blog-service-repo.test.ts __tests__/health-check-service.test.ts --runInBand`; backend active-surface static scan for non-type top-level `mongoose`/`models/*` imports returned no matches; backend/frontend/deployment `git diff --check`. Frontend push was not needed because remote frontend was already at `2.8.36`. +**Linked docs updated:** [[Postgres Runtime Cutover Status]] + +--- + ### 2026-06-02 — backend@f1ba14b, frontend@b94d8a9 — add notification Postgres backfill tooling **Commits:** backend `f1ba14b` (version `2.8.34`), frontend `b94d8a9` (version `2.8.34`)