docs: sync from backend c5db471 — request templates
This commit is contained in:
Siavash Sameni
@@ -4,7 +4,7 @@ tags: [data-model, migration, mongodb, postgres, mongoose, helper]
|
||||
aliases: [Mongo to Postgres, DB Migration Guide, Postgres Migration]
|
||||
created: 2026-05-31
|
||||
source: backend/src (automated multi-agent scan)
|
||||
updated: 2026-06-01 for backend integrate-main-into-development@1543b53
|
||||
updated: 2026-06-01 for backend integrate-main-into-development@c5db471
|
||||
---
|
||||
|
||||
# MongoDB → PostgreSQL Migration Guide
|
||||
@@ -17,7 +17,7 @@ updated: 2026-06-01 for backend integrate-main-into-development@1543b53
|
||||
> **Execution plan:** see the companion [[MongoDB to PostgreSQL Migration Plan (Drizzle)]] for the phased, Drizzle-concrete plan (repository seam, `id_map`, dual-write, per-phase cutover runbook).
|
||||
|
||||
> [!warning] Current implementation delta
|
||||
> This guide started as a migration helper. Backend `integrate-main-into-development@1543b53` now contains the first Postgres implementation layer: Drizzle schemas/migrations through `0009`, `src/db/client.ts`, `id_map`, `pg_dualwrite_gaps`, repository implementations/factory, backfill/verify scripts, conditional `payment_quotes` persistence, and aligned purchase/template request budget validation. Backend `2.8.17` also hardens the PurchaseRequest/SellerOffer backfill runner for marketplace-core dry-runs and selected-offer remapping, and enforces unique active marketplace categories by normalized visible name in Postgres mode. The broad service layer is still Mongoose-first and is **not** fully wired through those repositories. Use [[Postgres Runtime Cutover Status]] as the authoritative current-state snapshot.
|
||||
> This guide started as a migration helper. Backend `integrate-main-into-development@c5db471` now contains the first Postgres implementation layer: Drizzle schemas/migrations through `0010`, `src/db/client.ts`, `id_map`, `pg_dualwrite_gaps`, repository implementations/factory, backfill/verify scripts, conditional `payment_quotes` persistence, and aligned purchase/template request budget validation. Backend `2.8.19` also hardens the marketplace-core backfill runner for RequestTemplate/PurchaseRequest/SellerOffer dry-runs and selected-offer remapping, and enforces unique active marketplace categories by normalized visible name in Postgres mode. The broad service layer is still Mongoose-first and is **not** fully wired through those repositories. Use [[Postgres Runtime Cutover Status]] as the authoritative current-state snapshot.
|
||||
|
||||
> [!info] Scan coverage (2026-05-31)
|
||||
> - **23** Mongoose models (collections)
|
||||
@@ -2123,6 +2123,9 @@ CREATE TABLE purchase_request_service_info (
|
||||
|
||||
Template marketplace objects that sellers create to define reusable service request specifications. Each template includes product/service details, budget expectations, delivery requirements, and optional default proposals used in marketplace discovery.
|
||||
|
||||
> [!note] Implementation status
|
||||
> Backend `2.8.19` added `src/db/schema/requestTemplate.ts`, migration `0010_request_templates.sql`, and `src/db/backfill/backfill-requestTemplates.ts`. Runtime `RequestTemplateService` still uses Mongoose directly; this is a backfill/schema slice, not a read/write cutover.
|
||||
|
||||
#### Fields
|
||||
|
||||
| Path | Mongo Type | PG Type | Required | Notes |
|
||||
@@ -2181,7 +2184,7 @@ Template marketplace objects that sellers create to define reusable service requ
|
||||
|
||||
#### Gotchas & Migration Challenges
|
||||
|
||||
1. **Embedded Subdocuments**: budget, deliveryInfo, serviceInfo, proposal, and metadata are nested objects. Choose JSONB columns for simplicity or denormalize to separate tables for relational queries (not needed initially).
|
||||
1. **Embedded Subdocuments**: budget, deliveryInfo, serviceInfo, proposal, and metadata are nested objects. Backend `2.8.19` denormalizes budget, delivery, service, proposal, metadata, and payment rails into scalar/array columns, while keeping `specifications` as JSONB.
|
||||
|
||||
2. **String Arrays**: tags, attachments, images, specifications, and serviceInfo.requirements can use native PostgreSQL ARRAY type (faster queries) or JSONB (more flexible).
|
||||
|
||||
@@ -4747,7 +4750,8 @@ Each phase has explicit entry/exit criteria. The unit of migration is a bounded
|
||||
- **Entry:** Phase 4 exit. **Exit:** dispute lifecycle on PG; release-hold sync transactional.
|
||||
|
||||
#### Phase 6 (optional / deferred) — RequestTemplate, BlogPost
|
||||
- Move behind a search abstraction; replace `$regex` with PG trigram/FTS only if you choose to migrate. Otherwise leave on Mongo.
|
||||
- RequestTemplate now has a PG table/backfill in backend `2.8.19` because template checkout creates marketplace-core rows; runtime service wiring is still Mongo-first.
|
||||
- BlogPost remains optional/deferred. Move it behind a search abstraction; replace `$regex` with PG trigram/FTS only if you choose to migrate. Otherwise leave on Mongo.
|
||||
|
||||
#### Permanent on Mongo (no phase): Chat, Notification, TelegramSession, TempVerification, TelegramLink (link state)
|
||||
- Document-shaped + TTL-driven. Revisit only if operational cost of dual-stack outweighs migration cost.
|
||||
@@ -4827,7 +4831,7 @@ Engineer-weeks (one mid/senior backend engineer-week). Ranges reflect genuine un
|
||||
| Aggregation pipeline rewrites (18, incl. $lookup/$facet) | 2 | 4 | Golden-output testing. |
|
||||
| Verification, load-testing, cutover ops per context | 2 | 4 | Shadow-read tooling, soak monitoring. |
|
||||
| **Partial migration — money/relational core only** (Phases 0–5 + cross-cutting) | **~16** | **~28** | This delivers the stated goal: ACID for money + relational integrity. |
|
||||
| Phase 6 RequestTemplate + BlogPost (search/aggregation, $regex→FTS) | 2 | 4 | Optional. |
|
||||
| Phase 6 BlogPost (search/aggregation, $regex→FTS) | 1 | 2 | Optional. RequestTemplate schema/backfill moved into the marketplace-core phase; runtime wiring remains. |
|
||||
| Chat + Notification + sessions full migration (embedded arrays, 3 TTL→pg_cron) | 5 | 9 | Only if going full-PG; this is the expensive long tail. |
|
||||
| **Full migration (all 23 collections)** | **~23** | **~40** | Plus integration/hardening contingency. |
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ tags: [data-model, migration, postgres, drizzle, plan, runbook]
|
||||
aliases: [Drizzle Migration Plan, PG Migration Plan]
|
||||
created: 2026-05-31
|
||||
companion: "[[MongoDB to PostgreSQL Migration Guide]]"
|
||||
updated: 2026-06-01 for backend integrate-main-into-development@1543b53 backend 2.8.17
|
||||
updated: 2026-06-01 for backend integrate-main-into-development@c5db471 backend 2.8.19
|
||||
---
|
||||
|
||||
# MongoDB → PostgreSQL Migration Plan (Drizzle)
|
||||
@@ -17,7 +17,7 @@ updated: 2026-06-01 for backend integrate-main-into-development@1543b53 backend
|
||||
> **Scope reminder:** partial migration (Phases 0–5) is the recommended stopping point — ≈16–28 engineer-weeks. Full migration of Chat/Notification/sessions is explicitly deferred.
|
||||
|
||||
> [!warning] Current implementation status
|
||||
> Backend `2.8.17` has started the runtime cutover with store-specific raw Postgres facades: auth-owned users/Telegram auth records behind `AUTH_STORE=postgres`, confirmation-threshold config/history behind `CONFIG_STORE=postgres`, user address CRUD behind `ADDRESS_STORE=postgres`, and the first marketplace/reference domains behind `CATEGORY_STORE=postgres`, `LEVEL_CONFIG_STORE=postgres`, `SHOP_SETTINGS_STORE=postgres`, and `REVIEW_STORE=postgres`. Category PG mode now deactivates duplicate active names and enforces an active normalized-name unique index. It also contains the broader `src/db/` Drizzle schemas, repository implementations/factory, id-map bridge, and backfill runner described below, but the broad marketplace/payment/points services are still mostly not wired through that factory. Mongo remains authoritative unless a per-store flag is explicitly flipped. See [[Postgres Runtime Cutover Status]].
|
||||
> Backend `2.8.19` has started the runtime cutover with store-specific raw Postgres facades: auth-owned users/Telegram auth records behind `AUTH_STORE=postgres`, confirmation-threshold config/history behind `CONFIG_STORE=postgres`, user address CRUD behind `ADDRESS_STORE=postgres`, and the first marketplace/reference domains behind `CATEGORY_STORE=postgres`, `LEVEL_CONFIG_STORE=postgres`, `SHOP_SETTINGS_STORE=postgres`, and `REVIEW_STORE=postgres`. Category PG mode now deactivates duplicate active names and enforces an active normalized-name unique index. It also contains the broader `src/db/` Drizzle schemas through `0010`, repository implementations/factory, id-map bridge, and backfill runner described below. RequestTemplate now has a PG table/backfill, but broad marketplace/payment/points services are still mostly not wired through that factory. Mongo remains authoritative unless a per-store flag is explicitly flipped. See [[Postgres Runtime Cutover Status]].
|
||||
|
||||
---
|
||||
|
||||
@@ -364,8 +364,8 @@ Same phases as the guide §2, here with Drizzle-concrete entry/exit gates. Each
|
||||
- **Exit:** `users` in `pg` mode; referral self-FK intact; all auth flows pass; user uuids authoritative in `id_map`.
|
||||
|
||||
### Phase 4 — Money core (6–10 wk) — *the point of the project*
|
||||
- `PurchaseRequest`, `SellerOffer`, `Payment`, `FundsLedgerEntry`, `DerivedDestination`, `TrezorAccount`, `PointTransaction`.
|
||||
- **Status 2026-06-01:** Drizzle schemas/repositories and backfill scripts exist for PurchaseRequest/SellerOffer. Backend `2.8.17` hardens the marketplace-core backfill path with `npm run backfill:marketplace-core:postgres`, fixed PurchaseRequest timestamp/preferred-seller writes, a post-SellerOffer selected-offer remap step, and category duplicate cleanup/unique active-name enforcement. Runtime marketplace services still call Mongoose directly and must not be flipped with `REPO_MARKETPLACE` until service wiring plus shadow-read checks land.
|
||||
- `PurchaseRequest`, `SellerOffer`, `RequestTemplate`, `Payment`, `FundsLedgerEntry`, `DerivedDestination`, `TrezorAccount`, `PointTransaction`.
|
||||
- **Status 2026-06-01:** Drizzle schemas and backfill scripts exist for PurchaseRequest/SellerOffer/RequestTemplate. Backend `2.8.19` hardens the marketplace-core backfill path with `npm run backfill:marketplace-core:postgres`, fixed PurchaseRequest timestamp/preferred-seller writes, a RequestTemplate backfill step, a post-SellerOffer selected-offer remap step, and category duplicate cleanup/unique active-name enforcement. Runtime marketplace services still call Mongoose directly and must not be flipped with `REPO_MARKETPLACE` until service wiring plus shadow-read checks land.
|
||||
- Apply §4.1 (Mixed→discriminator+FK), §4.2 (offers/preferredSellers junctions, deliveryInfo/serviceInfo child tables), §4.5 (derivation counter).
|
||||
- **Wrap in real PG transactions the multi-doc writes that today have none:** `raiseDispute` (PurchaseRequest + Payment), payment confirm + `FundsLedgerEntry` AML-fee insert, referral reward (points + referralStats), PointsService flows (migrate its 2 `withTransaction` sites to PG `BEGIN/COMMIT`).
|
||||
- Preserve the `Payment` partial-unique idempotency index and `FundsLedgerEntry.idempotencyKey` uniqueness.
|
||||
@@ -376,8 +376,8 @@ Same phases as the guide §2, here with Drizzle-concrete entry/exit gates. Each
|
||||
- `Dispute ↔ Chat` becomes a **cross-store call** (Chat stays on Mongo) — define the boundary API.
|
||||
- **Exit:** dispute lifecycle in `pg` mode; release-hold sync transactional.
|
||||
|
||||
### Phase 6 (deferred / optional) — `RequestTemplate`, `BlogPost`
|
||||
- Behind a search abstraction; `$regex` → PG trigram/FTS only if migrated. Otherwise leave on Mongo.
|
||||
### Phase 6 (deferred / optional) — `BlogPost`
|
||||
- Behind a search abstraction; `$regex` → PG trigram/FTS only if migrated. Otherwise leave on Mongo. RequestTemplate schema/backfill moved into Phase 4 because template checkout creates PurchaseRequest/SellerOffer rows.
|
||||
|
||||
### Permanent on Mongo
|
||||
`Chat`, `Notification`, `TelegramSession`, `TempVerification`, `TelegramLink` link-state. Revisit only if dual-stack ops cost exceeds migration cost.
|
||||
@@ -407,7 +407,7 @@ async function shadowRead(key, mongoFn, pgFn) {
|
||||
## 8. Cutover & rollback runbook (per collection)
|
||||
|
||||
1. **Backfill** in batches with checkpointing; allocate uuids → `id_map`; remap FKs from already-migrated parents. Re-runnable (idempotent upserts).
|
||||
- Marketplace-core operator path: `MIGRATION_MONGO_URL=... MIGRATION_PG_URL=... npm run backfill:marketplace-core:postgres:dry-run`, then `npm run backfill:marketplace-core:postgres` in non-prod. Run `scripts/smoke/marketplace-core-postgres-backfill.sh` with the same DSNs to exercise the static backfill invariants and dry-run.
|
||||
- Marketplace-core operator path: `MIGRATION_MONGO_URL=... MIGRATION_PG_URL=... npm run backfill:marketplace-core:postgres:dry-run`, then `npm run backfill:marketplace-core:postgres` in non-prod. The group now includes RequestTemplate before PurchaseRequest/SellerOffer. Run `scripts/smoke/marketplace-core-postgres-backfill.sh` with the same DSNs to exercise the static backfill invariants and dry-run.
|
||||
2. **Enable `dual`** (flag) — writes go to both; shadow-read diffing on. Backfill the delta accumulated during step 1.
|
||||
3. **Soak** until row-count + checksum + shadow-read are clean for the agreed window.
|
||||
4. **Flip reads to `pg`** (flag). Keep dual-write on.
|
||||
|
||||
@@ -3,12 +3,12 @@ title: Postgres Runtime Cutover Status
|
||||
tags: [data-model, postgres, migration, runtime-status]
|
||||
aliases: [Postgres Status, PG Cutover Status, Mongo vs Postgres Runtime]
|
||||
created: 2026-05-31
|
||||
source: backend integrate-main-into-development@1543b53
|
||||
source: backend integrate-main-into-development@c5db471
|
||||
---
|
||||
|
||||
# Postgres Runtime Cutover Status
|
||||
|
||||
> **Current branch:** backend `integrate-main-into-development` at `1543b53`, version `2.8.17`.
|
||||
> **Current branch:** backend `integrate-main-into-development` at `c5db471`, version `2.8.19`.
|
||||
>
|
||||
> **Bottom line:** this branch is **Postgres-capable**, not fully Postgres-backed. Auth-owned user data can run through Postgres when `AUTH_STORE=postgres`; confirmation-threshold runtime config/history can run through Postgres when `CONFIG_STORE=postgres`; user address CRUD can run through Postgres when `ADDRESS_STORE=postgres`; marketplace categories, level config, shop settings, and reviews can run through Postgres with their own store flags. The category PG path now enforces one active visible category per normalized name. All PG-backed stores require `PG_URL`. Mongo remains the default and the compatibility store for still-Mongo domains.
|
||||
|
||||
@@ -28,7 +28,7 @@ source: backend integrate-main-into-development@1543b53
|
||||
| Marketplace reviews | Opt-in with `REVIEW_STORE=postgres` | Review list/summary/create routes use a review-store facade. PG-mode list responses still hydrate `reviewerId` from the user mirror to preserve frontend shape. |
|
||||
| Repository implementations | Present but partial runtime wiring | `src/db/repositories/*` and Drizzle schemas exist for the target architecture, but this branch's live cutover work is still mostly store-specific raw PG facades plus the conditional oracle quote write path. |
|
||||
| Oracle quote persistence | Conditional runtime PG write | `/api/payment/request-network/intents` lazily imports `quoteRepo` only when `ORACLE_QUOTING_ENABLED=true`; it writes `payment_quotes` if the PG parent payment row exists, mirrors to Mongo `Payment.quote`, and records `pg_dualwrite_gaps` if PG is behind. |
|
||||
| Backfill/verify scripts | Available as operator tooling | `MIGRATION_PG_URL` drives backfill scripts; guards restrict allowed target hosts. The marketplace-core runner group now backfills users/categories, purchase requests, seller offers, and the post-offer `selectedOfferId` remap in dependency order. These scripts are not run automatically by app startup. |
|
||||
| Backfill/verify scripts | Available as operator tooling | `MIGRATION_PG_URL` drives backfill scripts; guards restrict allowed target hosts. The marketplace-core runner group now backfills users/categories, request templates, purchase requests, seller offers, and the post-offer `selectedOfferId` remap in dependency order. These scripts are not run automatically by app startup. |
|
||||
|
||||
## What Is Still Mongo-Backed
|
||||
|
||||
@@ -38,7 +38,7 @@ Most of the service layer still imports Mongoose models directly. Auth-owned pat
|
||||
|---|---|---|
|
||||
| Legacy/broad user consumers | MongoDB mirror | Auth-owned users can be PG-backed, but still-Mongo domains expect Mongo ObjectId user references. PG-mode writes therefore maintain a Mongo mirror until those domains are cut over. |
|
||||
| Admin cleanup / seed address tooling | MongoDB | User-facing address CRUD is PG-capable, but admin cleanup and seed scripts still operate on Mongo first. Seed scripts backfill addresses to PG when `ADDRESS_STORE=postgres`. |
|
||||
| Marketplace requests/offers/templates | MongoDB | Marketplace, checkout, and seller-offer services still call `PurchaseRequest`, `SellerOffer`, and `RequestTemplate` directly. Category and shop-settings reads/writes are PG-capable through facades. PurchaseRequest/SellerOffer backfill tooling is now operator-ready, but runtime reads/writes remain Mongo-first and RequestTemplate still has no PG table/repository. |
|
||||
| Marketplace requests/offers/templates | MongoDB | Marketplace, checkout, and seller-offer services still call `PurchaseRequest`, `SellerOffer`, and `RequestTemplate` directly. Category and shop-settings reads/writes are PG-capable through facades. PurchaseRequest/SellerOffer/RequestTemplate backfill tooling is now operator-ready, but runtime reads/writes remain Mongo-first and RequestTemplate still has no runtime repo/service wiring. |
|
||||
| Payments and escrow state | MongoDB primary | Request Network, AMN scanner, webhook, admin, release/refund, adapter, reconciliation, and legacy payment paths still create/update `Payment` Mongoose documents directly. |
|
||||
| Funds ledger | MongoDB primary | `FundsLedgerEntry` remains the ledger used by current services; PG ledger tables exist but are not the live write target. |
|
||||
| Derived destinations and sweeps | MongoDB | Wallet destination allocation and sweep services call `DerivedDestination` directly. |
|
||||
@@ -77,7 +77,7 @@ Most of the service layer still imports Mongoose models directly. Auth-owned pat
|
||||
- `PG_URL=... npm run backfill:shop-settings:postgres`
|
||||
- `PG_URL=... npm run backfill:review:postgres`
|
||||
6. Run `PG_URL=... scripts/smoke/categories-postgres-unique.sh` and `PG_URL=... MONGODB_URI=... scripts/smoke/reference-stores-postgres.sh`, then set `CATEGORY_STORE=postgres LEVEL_CONFIG_STORE=postgres SHOP_SETTINGS_STORE=postgres REVIEW_STORE=postgres` together in non-prod.
|
||||
7. For marketplace-core data, run `MIGRATION_MONGO_URL=... MIGRATION_PG_URL=... npm run backfill:marketplace-core:postgres:dry-run`, then the non-dry `npm run backfill:marketplace-core:postgres` against non-prod. The group runs root dependencies, PurchaseRequest main rows, SellerOffer rows, then the selected-offer remap.
|
||||
7. For marketplace-core data, run `MIGRATION_MONGO_URL=... MIGRATION_PG_URL=... npm run backfill:marketplace-core:postgres:dry-run`, then the non-dry `npm run backfill:marketplace-core:postgres` against non-prod. The group runs root dependencies, RequestTemplate rows, PurchaseRequest main rows, SellerOffer rows, then the selected-offer remap.
|
||||
8. Run `scripts/smoke/marketplace-core-postgres-backfill.sh` with the same migration DSNs and record row-count/checksum results.
|
||||
9. Wire remaining services to repository interfaces one domain at a time.
|
||||
10. Enable `dual` mode per large domain only after wiring is proven by tests and smoke checks.
|
||||
|
||||
@@ -1,18 +1,20 @@
|
||||
---
|
||||
title: RequestTemplate
|
||||
tags: [data-model, mongoose]
|
||||
tags: [data-model, mongoose, postgres]
|
||||
aliases: [Template, Request Template, IRequestTemplate]
|
||||
---
|
||||
|
||||
# RequestTemplate
|
||||
|
||||
> **Last updated:** 2026-05-31 — seller-owned delivery mode and per-template payment rails documented.
|
||||
> **Last updated:** 2026-06-01 — Postgres schema and backfill surface documented.
|
||||
|
||||
A reusable template authored by a seller. When a buyer visits the template's `shareableLink`, the front-end pre-fills a new [[PurchaseRequest]] with the template's category, urgency, specs, seller-selected delivery mode, payment rail allowlist, and an optional default seller `proposal`. The schema mirrors `PurchaseRequest` for fast cloning, plus template-specific bookkeeping (`isActive`, `usageCount`, `maxUsage`, `expiresAt`).
|
||||
|
||||
> [!note] Source
|
||||
> `backend/src/models/RequestTemplate.ts:65` — schema definition
|
||||
> `backend/src/models/RequestTemplate.ts:295` — model export
|
||||
> `backend/src/models/RequestTemplate.ts:83` — Mongoose schema definition
|
||||
> `backend/src/models/RequestTemplate.ts:335` — model export
|
||||
> `backend/src/db/schema/requestTemplate.ts:35` — Drizzle table definition
|
||||
> `backend/src/db/backfill/backfill-requestTemplates.ts:1` — Mongo → Postgres backfill
|
||||
|
||||
## Schema
|
||||
|
||||
@@ -83,6 +85,16 @@ Defined at `backend/src/models/RequestTemplate.ts:283-293`:
|
||||
|
||||
`shareableLink` and `sellerId` already get indexes from `unique: true` / field-level conventions (see source comment at line 282).
|
||||
|
||||
Postgres migration `0010_request_templates.sql` creates `request_templates` with:
|
||||
|
||||
- `request_templates_legacy_object_id_uq`: idempotent Mongo bridge for backfill.
|
||||
- `request_templates_shareable_link_uq`: public slug uniqueness.
|
||||
- FK columns `seller_id → users.id` and `category_id → categories.id`.
|
||||
- Matching single/compound indexes for seller, category, product type, active state, expiry, and public slug lookups.
|
||||
- JSONB `specifications` and scalar/array columns for delivery, service, proposal, payment rails, images, and attachments.
|
||||
|
||||
Runtime service wiring is not cut over yet; `RequestTemplateService` still uses Mongoose directly.
|
||||
|
||||
## Pre/Post Hooks
|
||||
|
||||
None declared.
|
||||
|
||||
Reference in New Issue
Block a user