docs: sync from backend cab0719 - align request budget validation
This commit is contained in:
Siavash Sameni
@@ -6,7 +6,7 @@ aliases: [Models Index, Schema Overview]
|
||||
|
||||
# Data Model Overview
|
||||
|
||||
This section documents every Mongoose model that backs the marketplace. The persistence layer lives in `backend/src/models/` and is exported through a single barrel file at `backend/src/models/index.ts`. All models target MongoDB via Mongoose, lean on `timestamps: true` for `createdAt` / `updatedAt`, and follow a consistent pattern: one schema per file, an exported `I<Name>` TypeScript interface, and named exports for the compiled model.
|
||||
This section documents every Mongoose model that backs the marketplace. On backend `integrate-main-into-development@cab0719`, these Mongoose models are still the live application persistence layer. The repo also contains a Drizzle/Postgres migration layer, but most services still call `backend/src/models/*` directly.
|
||||
|
||||
> [!note] Scope
|
||||
> Twenty-two models are present in `backend/src/models/`. The "File" concept exists only at the service layer (`backend/src/services/file/`) and is not persisted as its own Mongoose collection, so it is not listed below.
|
||||
@@ -14,6 +14,9 @@ This section documents every Mongoose model that backs the marketplace. The pers
|
||||
> [!note] Documentation freshness
|
||||
> The 2026-05-24 audit note that marked `Dispute`, `BlogPost`, `Review`, `PointTransaction`, `LevelConfig`, and `ShopSettings` as missing is now stale: schema files exist for those models. Newer operational models such as [[ConfigSetting]], [[DerivedDestination]], [[FundsLedgerEntry]], and [[TrezorAccount]] should be expanded into dedicated model pages when the docs are next deepened.
|
||||
|
||||
> [!warning] Mongo vs Postgres runtime status
|
||||
> Postgres schemas and repositories exist for the money/relational core, but normal app traffic is not fully cut over. Payment quote rows are the only current conditional PG write in checkout, and even that requires `ORACLE_QUOTING_ENABLED=true` plus a resolvable PG payment row. See [[Postgres Runtime Cutover Status]].
|
||||
|
||||
## Index of Models
|
||||
|
||||
- [[User]] — Core identity. Stores credentials, profile, preferences, referral data, points, and WebAuthn passkeys. Every other model that records "who did what" points back at a `User._id`. Buyers, sellers, and admins all live in this collection, differentiated by a `role` enum.
|
||||
|
||||
120
02 - Data Models/Money-Core Migration Workflow — Audit Report.md
Normal file
120
02 - Data Models/Money-Core Migration Workflow — Audit Report.md
Normal file
@@ -0,0 +1,120 @@
|
||||
---
|
||||
title: Money-Core Migration Workflow — Audit Report
|
||||
tags: [migration, audit, workflow, postgres, drizzle]
|
||||
created: 2026-05-31
|
||||
target: .claude/workflows/pg-money-core-migration.js
|
||||
verdict: go-with-changes (0 blockers)
|
||||
companion: "[[MongoDB to PostgreSQL Migration Plan (Drizzle)]]"
|
||||
updated: 2026-05-31 after backend integrate-main-into-development@3a50dc4
|
||||
---
|
||||
|
||||
> [!info] Provenance
|
||||
> Generated 2026-05-31 by a 29-agent read-only audit workflow (4 review dimensions → adversarial verification → synthesis) over the `pg-money-core-migration` workflow design + the real money-core backend code. All recommended changes below have since been **applied** to the workflow script.
|
||||
|
||||
# Audit Report — `pg-money-core-migration.js` Workflow
|
||||
|
||||
**Target:** `/Users/manwe/CascadeProjects/escrow/.claude/workflows/pg-money-core-migration.js`
|
||||
**Status:** Not yet run. 8 phases (Preflight → Infra → Schema → Repos → Backfill → Tests → VerifyGate → SelfAudit).
|
||||
**Adversarial verification:** Every `high`/`blocker` finding was challenged against the actual script. **All of them were refuted.** The remaining open items are `medium`/`low` enforcement-gap concerns that were not independently re-verified (`n/a`).
|
||||
|
||||
> [!warning] Historical scope
|
||||
> This is a workflow audit, not the current runtime status. The backend branch now contains the generated Postgres layer, but Mongo remains authoritative for normal traffic and service-layer wiring is still incomplete. Use [[Postgres Runtime Cutover Status]] for the current code/runtime boundary.
|
||||
|
||||
---
|
||||
|
||||
## 1. Verdict: **GO (with recommended changes)**
|
||||
|
||||
The workflow **cannot itself cause fraud or fund/order dataloss**, and that is the load-bearing question. Three independent, code-level facts make it safe to *run*:
|
||||
|
||||
1. **It never touches production data.** SAFETY rule 1 (lines 34–36, 67) binds every agent to code/SQL/test generation only; any DB the harness spins up is an ephemeral local throwaway. The backfill scripts are generated as *artifacts for humans*, explicitly never auto-run (line 160), and are gated on a non-prod `MIGRATION_PG_URL` allowlist.
|
||||
2. **It cannot deploy.** It works on an isolated branch `feat/pg-money-core-migration` cut from `origin/main` (lines 54–55, 91), is additive-only under `backend/src/db/**` (rule 2), never bumps version (rule 3), and never pushes / opens a PR / flips a flag (rule 4). Per the CI audit (BRANCH_001, DEPLOY_001), creating or committing to this branch fires no CI; only a human push to `main`/`master` or a `v*` tag deploys — outside the workflow's reach.
|
||||
3. **The worst realistic failure is recoverable.** VerifyGate commits only on green typecheck+tests (lines 216–217), and to a *local* branch. Even the post-commit SelfAudit ordering (SAFETY_001, refuted) is benign because the commit is local and reversible; the verdict still surfaces to the human before any push.
|
||||
|
||||
Every adversarially-tested `blocker`/`high` claim was **refuted** because the corresponding safety constraint *is* present in the prompts (SAFETY rules 1–7, plus per-phase reinforcement on schema indexes line 127, transactions line 145, dual-write line 146, decimals line 126, backfill guard line 160). The legitimate residual risk is that these are **prompt-level instructions, not code-enforced gates** — an agent could under-implement one and VerifyGate (which only runs typecheck+tests) would not catch it. That is a *quality/trust* gap, not a *safety* gap, and it is fully contained by the mandatory human review before cutover. Hence: go, but tighten the gates below first so the human reviewer is handed verifiable evidence instead of having to re-derive it.
|
||||
|
||||
**Confirmed blockers: 0.**
|
||||
|
||||
---
|
||||
|
||||
## 2. Confirmed Blockers (must fix before first run)
|
||||
|
||||
**None.** All `blocker`/`high`-severity findings were adversarially verified and **refuted** — the safety contract they claimed was missing is in fact specified in the workflow prompts. The workflow is safe to run as-is. The items in §3 are improvements that raise trust and catch agent under-implementation; they are not gating.
|
||||
|
||||
| ID | Why it is NOT a blocker |
|
||||
|---|---|
|
||||
| IDEMPOTENCY_INDEX_PRESERVATION / FK-010 | Line 127 explicitly mandates the partial `WHERE provider='request.network' AND direction='in' AND status='pending'` and the ledger sparse-unique. Not enforced in code, but specified + audited by SelfAudit (line 240). |
|
||||
| DECIMAL_PRECISION_TRUNCATION_RISK / FK-009 | Line 126 mandates `numeric(38,18)`, never float; SelfAudit hunts float money (line 239). Token-decimal scale flagged as a risk to surface (line 131). |
|
||||
| MULTI_DOC_WRITE / FK-011 | Line 145 requires `db.transaction(...)` for payment+ledger, referral reward, dispute hold/release; SAFETY rule 7 repeats it. |
|
||||
| LOCKFILE_001 / BRANCH_002 / SAFETY_002 | Workflow never pushes; lockfile/`npm audit` are CI-merge-time concerns the human owns. Refuted as in-workflow blockers. |
|
||||
| MIXED_ID / immutability / prod-guard / version-bump | All specified in prompts (lines 124, 48/line 70, 160, rule 3). |
|
||||
|
||||
---
|
||||
|
||||
## 3. Recommended Changes Before Running (high / medium), by phase
|
||||
|
||||
These convert prompt-level promises into **verifiable evidence** so the human reviewer and VerifyGate can confirm them, and they fix the two genuine structural risks (factory race, test-skip-as-pass).
|
||||
|
||||
| Phase | ID | Change | Severity |
|
||||
|---|---|---|---|
|
||||
| **Preflight** | PREFLIGHT_BRANCH_ALREADY_EXISTS_REUSE_RISK | After `git switch ${BRANCH}` (line 91, reuse path), run `git status --porcelain` on the target branch and STOP if dirty — re-running must not accumulate prior uncommitted work. | medium |
|
||||
| **Repos** | PARALLEL_AGENTS_RACE / CONFLICT_001 | **Structural fix.** Four parallel domain agents all "update `factory.ts`" (lines 138, 147); last writer wins, silently dropping 3 domains' wiring. Split Repos into: (a) parallel — each agent writes only its own repo files, no `factory.ts`; (b) sequential aggregator agent that builds one unified `factory.ts`. Then have VerifyGate assert `factory.ts` imports/exports all 4 domains. | medium |
|
||||
| **Repos** | REPOS_PHASE_MONGO_WRAPPER_NOT_VERIFIED | Require the MongoRepo to carry the original Mongoose call as an inline comment (or a method→source map) so the human can confirm "verbatim, no behavior change" (line 144). | medium |
|
||||
| **Repos** | FK-012 | For multi-doc money writes, the dual-write "log+alert, don't throw" (line 146) can leave Mongo funds released with no PG ledger row during the dual-write window. Add an explicit escalation/alert requirement and a reconcile sweep for these gaps (covered by reconcile.ts, line 170). | medium |
|
||||
| **Schema** | IDEMPOTENCY_INDEX_REPRODUCTION_NOT_TESTED / SCHEMA_AGENTS_INDEX_REPRODUCTION_NO_VERIFY / FK-001/FK-006/FK-008 | Have each schema agent emit the generated SQL (or index/CHECK list) in its return. Add a schema-audit step in VerifyGate that greps the generated migration for: (a) the partial `WHERE` on the RN idempotency index, (b) ledger `idempotency_key IS NOT NULL`, (c) each Mixed-id CHECK constraint, (d) presence of every Mongo index. Missing partial-`WHERE` or CHECK → blocker. | high (where) / medium |
|
||||
| **Backfill** | NO_PROD_BACKFILL_RUNAWAY_GUARD / FK-003/FK-004/FK-005/FK-007 | Make the non-prod guard a **whitelist** (`localhost`/`127.0.0.1`/named staging hosts), not a `!includes('prod')` blacklist, and require a guard unit-test that aborts on a mock prod DSN. Add per-FK parent-completion checks (TrezorAccount `addresses[].paymentId`, PointTransaction `order`/`referredUser`, Category `parentId` string audit) before child upsert. | high (where) / medium |
|
||||
| **Backfill (verify)** | VERIFICATION_CHECKSUM_PRECISION / LEDGER_IMMUTABILITY | In `checksums.ts` (line 168) compare fund sums as `::numeric` cast to **text**, not float. In `reconcile.ts` (line 170) add an assertion that ledger rows are never UPDATE/DELETE (replicating the Mongo pre-save immutability hook) — via a PG trigger or a repo guard plus a test. | medium / high (where) |
|
||||
| **Tests** | TEST_SKIP_WITHOUT_TEST_PG_SILENTLY_PASSES / TEST_001 | **Structural fix.** Tests "skip if no test PG" (line 190) but VerifyGate treats skip as pass. Return `{testsPassed, testsSkipped, skipReason}`; VerifyGate must **fail the commit if `testsSkipped` is true**, so money-safety tests are never silently bypassed. | medium |
|
||||
| **Tests** | PAYMENT_CONFIRM_LEDGER_ATOMICITY / MISSING_CONSTRAINT_SERIALIZATION / DUAL_WRITE idempotency | Expand the atomicity test (line 182) to a concrete template: inject a mid-transaction ledger failure → assert payment status rolled back → retry → assert success (idempotent). Add a 2-concurrent-confirm test and document the required isolation level / `FOR UPDATE` locking to prevent double-release. | high (where) / medium |
|
||||
| **VerifyGate** | VERSION_BUMP_DETECTION / VERSION_001 | Add a hard diff check: `git diff ${BASE}...HEAD -- backend/package.json | grep '"version"'` → if non-empty, BLOCKER and do not commit. Cheap insurance against an accidental deploy-triggering bump. | high (where) / low |
|
||||
| **SelfAudit** | SELF_AUDIT_VERDICT_NOT_GATED | After SelfAudit, gate the final return: if `verdict === 'unsafe'` or `mustFixBeforeBackfill` is non-empty, return an error/flag prominently (the commit is local and reversible, so this is advisory, but it must not be buried in the JSON). | medium |
|
||||
| **New phase** | NO_MANUAL_BACKFILL_RUNBOOK_GENERATED | Add a Documentation phase that emits `backend/src/db/BACKFILL_RUNBOOK.md`: dependency order, per-script invocation + env, checksum/row-count verification, dual-write enablement, soak/monitor, rollback. This is the cutover team's source of truth. | medium |
|
||||
|
||||
---
|
||||
|
||||
## 4. What the Workflow Does Well (trust these parts)
|
||||
|
||||
- **Airtight non-prod / non-deploy posture.** SAFETY rules 1–4 are stated globally and re-stated in every phase prompt. Branch off `origin/main`, additive-only, no version bump, no push, no auto-backfill. CI audit independently confirms the branch cannot trigger a deploy.
|
||||
- **Correct scope and FK ordering.** Tier A (money/orders) + Tier B parents (User, Category) migrated first; `COLLECTIONS` ordered parents-first (line 59); backfill runner enforces `User, Category → PurchaseRequest, SellerOffer → Payment, ...` (line 159). Out-of-scope set (Chat, Notification, Address, Review) is sensible.
|
||||
- **Money-safety modeling is explicitly specified.** `numeric(38,18)` not float (line 126); the exact RN partial-unique `WHERE`, ledger sparse-unique, and derived-destination uniqueness (line 127); Mixed-id discriminator+typed-FK+external-ref+CHECK pattern (line 124); ledger append-only (rule 6); multi-doc writes in real PG transactions (rule 7, line 145).
|
||||
- **Dual-write direction is correct.** Reads authoritative from Mongo, writes Mongo-first then PG idempotent upsert, PG failure does not block Mongo (line 146) — the safe ordering for a not-yet-cut-over store.
|
||||
- **Real verification harness + reconciliation.** Row counts, fund-sum checksums, shadow-read diffing, and a money-specific reconcile (released/refunded payments have matching ledger entries, no double-release, monotonic escrow state) — lines 167–170.
|
||||
- **Layered safety net.** A `BULK`(sonnet)/`BRAIN`(opus) model split that reserves Opus for the commit gate and the adversarial fund-safety SelfAudit (lines 16–20, 247), which hunts exactly the right failure modes (float money, dropped idempotency, non-transactional writes, collapsed Mixed ids, prod backfill).
|
||||
|
||||
---
|
||||
|
||||
## 5. Refuted / Non-Issues (do not re-raise)
|
||||
|
||||
All of the following were challenged and confirmed already handled by the workflow prompts or by the no-push/no-prod design. They are **not** action items:
|
||||
|
||||
- **Prod backfill guard, version-bump, client.ts throw-on-unset-PG_URL, ledger immutability, Mixed-id CHECK, idempotency index `WHERE`, decimal-as-numeric, multi-doc transactions** — all specified in SAFETY rules 1–7 and per-phase prompts (lines 105, 124, 126, 127, 145, 146, 160, rule 3/6/7). The "not code-enforced" critique is valid as a *trust* improvement (see §3) but does not make the workflow unsafe to run.
|
||||
- **SAFETY_001 (self-audit after commit):** refuted — commit is to a local, reversible, non-deploying branch; the verdict still reaches the human.
|
||||
- **LOCKFILE_001 / BRANCH_002 / SAFETY_002 (lockfile, npm audit):** refuted as in-workflow blockers — the workflow never installs, never pushes; these are CI-merge-time concerns owned by the human at integration. (Note: CI is Gitea Actions, not Woodpecker as the SAFETY comment says — a cosmetic doc fix only.)
|
||||
- **BRANCH_001 / BRANCH_003 / DEPLOY_001 / VERSION_001 / MERGE_001 / PARALLEL_001:** refuted — branch isolation, additive package.json edits, and "never push" are operationally sound; the residual is human discipline at push time, covered by §6.
|
||||
|
||||
---
|
||||
|
||||
## 6. Pre-Run Checklist & Human-Only Gates
|
||||
|
||||
### Before invoking the workflow
|
||||
1. **Apply the two structural §3 fixes first** — they are cheap and prevent silent breakage:
|
||||
- Repos `factory.ts` race: split into parallel-repo-files + sequential-aggregator.
|
||||
- Tests skip-as-pass: return `testsSkipped` and have VerifyGate fail on skip.
|
||||
2. **Confirm a clean working tree** on the repo (Preflight will STOP otherwise — line 90) and that `origin` is fetched.
|
||||
3. **Confirm branch state:** if `feat/pg-money-core-migration` already exists, verify it has no stray uncommitted work (PREFLIGHT_BRANCH_ALREADY_EXISTS).
|
||||
4. **Provide an ephemeral local test PG** (container/throwaway) so money-safety tests actually run instead of skipping. If you cannot, accept that VerifyGate (post-fix) will refuse to commit.
|
||||
5. **Verify parallel-agent scope is disjoint** — per repo memory, the moojttaba agent pushes to the same branches. Ensure it is NOT touching `user/payment/points/marketplace` domains or `backend/src/db/repositories/factory.ts`.
|
||||
6. **Set no production env vars** in the harness shell. Ensure `PG_URL` (if set) and `MIGRATION_PG_URL` point only at local/staging.
|
||||
|
||||
### Gates enforced during the run
|
||||
- VerifyGate commits **only** on green typecheck + tests; otherwise leaves work uncommitted with verbatim blockers (lines 216–217). Trust this — but read the `blockers` array.
|
||||
- (After §3) VerifyGate also fails on version-bump diff, skipped tests, and missing partial-`WHERE`/CHECK in generated SQL.
|
||||
|
||||
### After the run — human reviews REQUIRED before anything leaves the branch
|
||||
1. Read `selfAudit.verdict` and `selfAudit.mustFixBeforeBackfill` (lines 222–248). **Do not proceed if `unsafe` or non-empty must-fix.**
|
||||
2. Code-review the generated schema for: partial-unique `WHERE` clauses, ledger immutability enforcement, Mixed-id CHECK constraints, `numeric` (no float), and `db.transaction(...)` around payment+ledger / referral / dispute writes.
|
||||
3. Confirm `package.json` version is unchanged and the lockfile situation is understood (update lockfiles + run `npm/yarn audit` at integration time, not in this workflow).
|
||||
|
||||
### STAYS HUMAN-ONLY (workflow must never do these — and does not)
|
||||
- **Running any backfill against real/staging data.** Scripts are artifacts; a human runs them, in dependency order, against a whitelisted non-prod DSN, with `--dry-run` first and checksum verification after each step (use the §3 runbook).
|
||||
- **Cutover / flipping any `mongo|dual|pg` rollout flag.**
|
||||
- **Pushing the branch, opening/merging a PR, tagging `v*`** — any of which can trigger CI/deploy. Cherry-pick reviewed changes into a proper feature branch if needed; never push `feat/pg-money-core-migration` to `main`.
|
||||
4864
02 - Data Models/MongoDB to PostgreSQL Migration Guide.md
Normal file
4864
02 - Data Models/MongoDB to PostgreSQL Migration Guide.md
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,437 @@
|
||||
---
|
||||
title: MongoDB → PostgreSQL Migration Plan (Drizzle)
|
||||
tags: [data-model, migration, postgres, drizzle, plan, runbook]
|
||||
aliases: [Drizzle Migration Plan, PG Migration Plan]
|
||||
created: 2026-05-31
|
||||
companion: "[[MongoDB to PostgreSQL Migration Guide]]"
|
||||
updated: 2026-05-31 for backend integrate-main-into-development@cab0719
|
||||
---
|
||||
|
||||
# MongoDB → PostgreSQL Migration Plan (Drizzle)
|
||||
|
||||
> [!abstract] What this is
|
||||
> The **execution plan** for the recommendation in [[MongoDB to PostgreSQL Migration Guide]]: a **hybrid target** (Postgres for the money/relational core, Mongo retained for Chat/Notification/TTL-session collections) reached via the **strangler pattern with dual-write**, using **Drizzle ORM** + **drizzle-kit** migrations.
|
||||
>
|
||||
> It is opinionated and concrete: a repository seam, an `id_map` bridge, Drizzle schema sketches for the hard cases (Mixed ids, embedded arrays, partial-unique idempotency, TTL), per-phase backfill/verify/cutover mechanics, and a rollback runbook. Where it references fields it uses the **real schema** from `backend/src/models/`.
|
||||
>
|
||||
> **Scope reminder:** partial migration (Phases 0–5) is the recommended stopping point — ≈16–28 engineer-weeks. Full migration of Chat/Notification/sessions is explicitly deferred.
|
||||
|
||||
> [!warning] Current implementation status
|
||||
> Backend `2.6.80` has completed the first implementation slice of this plan: Postgres/Drizzle infra, schemas/migrations through `0008`, `id_map`, `pg_dualwrite_gaps`, Drizzle/Mongo/Dual repo implementations, backfill/verify tooling, conditional oracle `payment_quotes` persistence, and the `PurchaseRequest`/`RequestTemplate` budget enum alignment with PG `budget_currency`. It has **not** completed service-layer wiring or runtime cutover. Mongo remains authoritative for normal traffic. See [[Postgres Runtime Cutover Status]].
|
||||
|
||||
---
|
||||
|
||||
## 0. Guiding principles
|
||||
|
||||
1. **Never cut over without a soak.** Every collection goes through backfill → dual-write → shadow-read verify → flip reads → soak → decommission. Rollback at any point = flip reads back to Mongo.
|
||||
2. **The repository layer is the only thing that knows where data lives.** Services must stop calling Mongoose directly. This seam is what makes the swap invisible and per-collection reversible.
|
||||
3. **Parents before children.** FK remapping flows through `id_map`; you cannot migrate `Payment` before `User` exists in PG with stable uuids.
|
||||
4. **Money correctness is the point.** The migration's payoff is real ACID transactions around payment + ledger + dispute flows that today lean on Mongo per-document atomicity. Treat every money write as transactional from day one in PG.
|
||||
5. **No feature work during migration.** No new fields, no behavior changes. A migration that also ships features cannot be verified by row-count + checksum equality.
|
||||
6. **Mongo stays authoritative until cutover.** Dual-write writes both; reads come from Mongo until a collection's shadow-read window is clean.
|
||||
|
||||
---
|
||||
|
||||
## 1. Target architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ Service layer │
|
||||
│ (marketplace, payment, dispute, points, …) │
|
||||
└───────────────────────┬─────────────────────┘
|
||||
│ calls interfaces only
|
||||
┌───────────────────────▼─────────────────────┐
|
||||
│ Repository layer │
|
||||
│ IUserRepo, IPaymentRepo, IPurchaseRepo, … │
|
||||
│ ── feature-flagged per collection ── │
|
||||
└───────┬───────────────────────────┬─────────┘
|
||||
reads/writes reads/writes
|
||||
│ │
|
||||
┌───────────▼─────────┐ ┌───────────▼─────────┐
|
||||
│ MongoRepo (today) │ │ DrizzleRepo (new) │
|
||||
│ Mongoose models │ │ Postgres + Drizzle │
|
||||
└─────────────────────┘ └─────────────────────┘
|
||||
│ │
|
||||
┌─────▼─────┐ ┌─────▼─────┐
|
||||
│ MongoDB │◄── id_map ──────►│ Postgres │
|
||||
└───────────┘ (bridge) └───────────┘
|
||||
|
||||
Permanent on Mongo: Chat, Notification, TelegramSession,
|
||||
TempVerification, TelegramLink-state. Redis untouched.
|
||||
```
|
||||
|
||||
Each domain gets an interface (`IPaymentRepo`), a `MongoPaymentRepo` (wraps today's Mongoose calls verbatim), a `DrizzlePaymentRepo` (new), and a `DualWritePaymentRepo` (delegates reads to one, writes to both, behind a flag). A factory picks the implementation per collection from config:
|
||||
|
||||
```ts
|
||||
// repos/factory.ts
|
||||
type Mode = 'mongo' | 'dual' | 'pg';
|
||||
const MODE: Record<string, Mode> = {
|
||||
user: env.REPO_USER ?? 'mongo',
|
||||
payment: env.REPO_PAYMENT ?? 'mongo',
|
||||
// …per collection
|
||||
};
|
||||
export const paymentRepo: IPaymentRepo =
|
||||
MODE.payment === 'pg' ? new DrizzlePaymentRepo()
|
||||
: MODE.payment === 'dual' ? new DualWritePaymentRepo(new MongoPaymentRepo(), new DrizzlePaymentRepo())
|
||||
: new MongoPaymentRepo();
|
||||
```
|
||||
|
||||
A collection's migration is then just three flag flips: `mongo → dual → pg`.
|
||||
|
||||
---
|
||||
|
||||
## 2. Drizzle & infra setup (Phase 0)
|
||||
|
||||
### Packages
|
||||
```
|
||||
pnpm add drizzle-orm pg
|
||||
pnpm add -D drizzle-kit @types/pg
|
||||
```
|
||||
|
||||
### Layout
|
||||
```
|
||||
backend/src/db/
|
||||
schema/ # one file per table group
|
||||
users.ts
|
||||
payments.ts
|
||||
purchaseRequests.ts
|
||||
...
|
||||
idMap.ts
|
||||
index.ts # re-exports all tables + relations
|
||||
client.ts # drizzle(pg.Pool) singleton
|
||||
migrations/ # drizzle-kit generated SQL
|
||||
repositories/
|
||||
interfaces/ # IUserRepo, IPaymentRepo, …
|
||||
mongo/ # MongoUserRepo (wraps existing Mongoose)
|
||||
drizzle/ # DrizzleUserRepo
|
||||
dual/ # DualWriteUserRepo
|
||||
factory.ts
|
||||
backfill/ # per-collection batch copiers
|
||||
verify/ # row-count + checksum + shadow-read harness
|
||||
drizzle.config.ts
|
||||
```
|
||||
|
||||
### `drizzle.config.ts`
|
||||
```ts
|
||||
import { defineConfig } from 'drizzle-kit';
|
||||
export default defineConfig({
|
||||
schema: './src/db/schema/index.ts',
|
||||
out: './src/db/migrations',
|
||||
dialect: 'postgresql',
|
||||
dbCredentials: { url: process.env.PG_URL! },
|
||||
strict: true,
|
||||
verbose: true,
|
||||
});
|
||||
```
|
||||
|
||||
### Client
|
||||
```ts
|
||||
// src/db/client.ts
|
||||
import { drizzle } from 'drizzle-orm/node-postgres';
|
||||
import { Pool } from 'pg';
|
||||
import * as schema from './schema';
|
||||
export const pool = new Pool({ connectionString: process.env.PG_URL, max: 10 });
|
||||
export const db = drizzle(pool, { schema });
|
||||
```
|
||||
|
||||
> Mirror the current Mongo pool size (`maxPoolSize: 10` in `connection.ts`). Keep `mongoose.connect` alive in parallel — both drivers run for the whole migration.
|
||||
|
||||
### Migration workflow
|
||||
- Author tables in `schema/*.ts` → `pnpm drizzle-kit generate` → review the SQL in `migrations/` → `pnpm drizzle-kit migrate` in CI per environment.
|
||||
- **Migrations are versioned, reviewed, and reversible.** This is brand-new discipline — there is no migration framework today.
|
||||
|
||||
---
|
||||
|
||||
## 3. The `id_map` bridge
|
||||
|
||||
ObjectIds become uuids. Every legacy id is recorded so FKs can be remapped and dual-writes stay idempotent.
|
||||
|
||||
```ts
|
||||
// src/db/schema/idMap.ts
|
||||
import { pgTable, uuid, text, timestamp, uniqueIndex } from 'drizzle-orm/pg-core';
|
||||
export const idMap = pgTable('id_map', {
|
||||
collection: text('collection').notNull(), // 'users', 'payments', …
|
||||
legacyId: text('legacy_object_id').notNull(), // 24-char hex
|
||||
newId: uuid('new_id').notNull().defaultRandom(),
|
||||
createdAt: timestamp('created_at', { withTimezone: true }).defaultNow(),
|
||||
}, (t) => ({
|
||||
uq: uniqueIndex('id_map_collection_legacy_uq').on(t.collection, t.legacyId),
|
||||
}));
|
||||
```
|
||||
|
||||
Rules:
|
||||
- Backfill allocates `new_id` once per `(collection, legacyId)` and upserts here. Re-running backfill is safe.
|
||||
- Resolving a foreign reference = look up the parent's `legacyId` in `id_map` to get its `new_id`. **A child cannot backfill until its parents are mapped** (enforces parents-before-children).
|
||||
- Keep `legacy_object_id` as a real column on each migrated table too, for traceability and for the dual-write path to match Mongo docs.
|
||||
|
||||
---
|
||||
|
||||
## 4. Resolving the hard data-modeling cases in Drizzle
|
||||
|
||||
These are the patterns from §3 of the guide, made concrete. Get these right once; they recur.
|
||||
|
||||
### 4.1 Mixed / polymorphic ids — `Payment`, `FundsLedgerEntry`, `DerivedDestination`
|
||||
|
||||
Today `Payment.purchaseRequestId`, `sellerOfferId`, `sellerId` are `Schema.Types.Mixed` — an ObjectId for normal flows, a **string** for template checkout. **Never** store "uuid-or-string" in one PG column. Split into a typed FK + a nullable free-text ref + a discriminator.
|
||||
|
||||
```ts
|
||||
// src/db/schema/payments.ts
|
||||
import { pgTable, uuid, text, numeric, boolean, timestamp, jsonb, pgEnum, index, uniqueIndex } from 'drizzle-orm/pg-core';
|
||||
|
||||
export const paymentProvider = pgEnum('payment_provider', ['request.network','amn.scanner','shkeeper','other']);
|
||||
export const paymentDirection = pgEnum('payment_direction', ['in','out','refund']);
|
||||
export const paymentStatus = pgEnum('payment_status', ['pending','processing','completed','failed','cancelled','refunded']); // confirm full enum from model
|
||||
export const escrowState = pgEnum('escrow_state', ['funded','releasable','released','refunded','releasing','failed','cancelled','partial']);
|
||||
export const refKind = pgEnum('ref_kind', ['entity','template']); // discriminator
|
||||
|
||||
export const payments = pgTable('payments', {
|
||||
id: uuid('id').primaryKey().defaultRandom(),
|
||||
legacyObjectId: text('legacy_object_id'),
|
||||
|
||||
// purchaseRequestId (Mixed) → typed FK OR free string
|
||||
purchaseRequestRefKind: refKind('purchase_request_ref_kind').notNull(),
|
||||
purchaseRequestId: uuid('purchase_request_id').references(() => purchaseRequests.id), // null when template
|
||||
purchaseRequestExternalRef: text('purchase_request_external_ref'), // set when template
|
||||
|
||||
// sellerOfferId (Mixed) → same shape
|
||||
sellerOfferRefKind: refKind('seller_offer_ref_kind').notNull(),
|
||||
sellerOfferId: uuid('seller_offer_id').references(() => sellerOffers.id),
|
||||
sellerOfferExternalRef: text('seller_offer_external_ref'),
|
||||
|
||||
buyerId: uuid('buyer_id').notNull().references(() => users.id),
|
||||
|
||||
// sellerId (Mixed)
|
||||
sellerRefKind: refKind('seller_ref_kind').notNull(),
|
||||
sellerId: uuid('seller_id').references(() => users.id),
|
||||
sellerExternalRef: text('seller_external_ref'),
|
||||
|
||||
// amount subdoc → inline columns
|
||||
amount: numeric('amount', { precision: 38, scale: 18 }).notNull(),
|
||||
currency: text('currency').notNull().default('USDT'),
|
||||
|
||||
provider: paymentProvider('provider').notNull().default('request.network'),
|
||||
direction: paymentDirection('direction').notNull().default('in'),
|
||||
status: paymentStatus('status').notNull().default('pending'),
|
||||
escrowState: escrowState('escrow_state'),
|
||||
|
||||
providerPaymentId: text('provider_payment_id'),
|
||||
blockchain: jsonb('blockchain'), // transactionHash etc. — read-as-blob, GIN if filtered
|
||||
metadata: jsonb('metadata'), // provider-specific, schema-varying
|
||||
|
||||
isRefunded: boolean('is_refunded').notNull().default(false),
|
||||
completedAt: timestamp('completed_at', { withTimezone: true }),
|
||||
createdAt: timestamp('created_at', { withTimezone: true }).defaultNow(),
|
||||
updatedAt: timestamp('updated_at', { withTimezone: true }).defaultNow(),
|
||||
}, (t) => ({
|
||||
byStatusCreated: index('payments_status_created_idx').on(t.status, t.createdAt),
|
||||
byBuyerStatus: index('payments_buyer_status_idx').on(t.buyerId, t.status),
|
||||
bySellerStatus: index('payments_seller_status_idx').on(t.sellerId, t.status),
|
||||
txHash: index('payments_tx_hash_idx').on(t.providerPaymentId),
|
||||
// Partial-unique idempotency — the real Mongo index 'uniq_pending_request_network_by_buyer_session_offer'
|
||||
pendingRnUq: uniqueIndex('uniq_pending_rn_by_buyer_offer')
|
||||
.on(t.buyerId, t.purchaseRequestId, t.sellerOfferId, t.provider, t.direction)
|
||||
.where(sql`provider = 'request.network' AND direction = 'in' AND status = 'pending'`),
|
||||
}));
|
||||
```
|
||||
|
||||
Add a CHECK so a discriminator always agrees with which column is populated:
|
||||
```sql
|
||||
ALTER TABLE payments ADD CONSTRAINT payments_pr_ref_ck CHECK (
|
||||
(purchase_request_ref_kind = 'entity' AND purchase_request_id IS NOT NULL AND purchase_request_external_ref IS NULL) OR
|
||||
(purchase_request_ref_kind = 'template' AND purchase_request_id IS NULL AND purchase_request_external_ref IS NOT NULL)
|
||||
);
|
||||
```
|
||||
|
||||
`FundsLedgerEntry` has the same Mixed `purchaseRequestId`/`paymentId` plus a **`idempotencyKey` sparse-unique** → partial unique index `WHERE idempotency_key IS NOT NULL`.
|
||||
|
||||
### 4.2 Embedded arrays → child tables
|
||||
|
||||
| Source (embedded) | PG | Notes |
|
||||
|---|---|---|
|
||||
| `PurchaseRequest.offers[]` (array of SellerOffer ids) | junction `purchase_request_offers(pr_id, offer_id)` | FK integrity; also drop the denormalized array. |
|
||||
| `PurchaseRequest.preferredSellerIds[]` | junction `pr_preferred_sellers(pr_id, user_id)` | — |
|
||||
| `PurchaseRequest.deliveryInfo / serviceInfo` (nested subdocs) | child tables `pr_delivery_info`, `pr_service_info` (1:1) | queried logistics; not blobbed. |
|
||||
| `Dispute.evidence[]`, `Dispute.timeline[]` | `dispute_evidence`, `dispute_timeline` | timeline pre-save append → explicit INSERT. |
|
||||
| `User.passkeys[]`, `User.refreshTokens[]` | `user_passkeys`, `user_refresh_tokens` | append/revoke + lookup semantics. |
|
||||
| `DerivedDestination` sweep history, `TrezorAccount.addresses[]` | child tables | per-address rows referenced by payments. |
|
||||
| `Payment.blockchain`, `Payment.metadata`, `Notification.metadata`, `PointTransaction.metadata` | **JSONB** | read-as-blob, never filtered/joined. |
|
||||
|
||||
Rule: **child table when you query/index/FK/aggregate it; JSONB when you read it whole and never filter on it.**
|
||||
|
||||
### 4.3 Self-referential FK — `Category`
|
||||
```ts
|
||||
export const categories = pgTable('categories', {
|
||||
id: uuid('id').primaryKey().defaultRandom(),
|
||||
legacyObjectId: text('legacy_object_id'),
|
||||
name: text('name').notNull(),
|
||||
nameEn: text('name_en'),
|
||||
parentId: uuid('parent_id'), // self-FK, see relations
|
||||
isActive: boolean('is_active').notNull().default(true),
|
||||
}, (t) => ({
|
||||
parentIdx: index('categories_parent_idx').on(t.parentId),
|
||||
activeIdx: index('categories_active_idx').on(t.isActive),
|
||||
}));
|
||||
// relations(): parentId → categories.id, ON DELETE SET NULL
|
||||
```
|
||||
`Category.parentId` is itself Mixed (ObjectId | string) in the model — verify all rows are ObjectIds during the pre-migration audit; treat stray strings as data errors to clean.
|
||||
|
||||
### 4.4 Sparse-unique → partial unique index — `User.email`, `User.referralCode`
|
||||
The runtime code in `connection.ts` rebuilds `users.email` as unique+sparse. In PG:
|
||||
```ts
|
||||
emailUq: uniqueIndex('users_email_uq').on(t.email).where(sql`email IS NOT NULL`),
|
||||
referralUq: uniqueIndex('users_referral_uq').on(t.referralCode).where(sql`referral_code IS NOT NULL`),
|
||||
```
|
||||
Reimplement `toJSON()` password/token stripping in the repository's read mapper (it deletes `refreshTokens`, `emailVerification*` before returning).
|
||||
|
||||
### 4.5 Atomic counter — `DerivedDestination.derivationIndex`
|
||||
Today allocation relies on Mongo atomicity. In PG use a real transaction with `SELECT … FOR UPDATE` on a per-(buyer,chain) counter row, or a dedicated sequence per chain. The `uniq_destination_by_buyer_seller_chain` unique index ports directly. `status` enum `('active','swept','sweeping','quarantined')` → `pgEnum`.
|
||||
|
||||
### 4.6 TTL → `pg_cron`
|
||||
`TempVerification` and `TelegramSession` stay on Mongo (ephemeral, recommended). If `Notification` (90-day TTL) ever moves: monthly range-partition + drop, or
|
||||
```sql
|
||||
SELECT cron.schedule('notifications_ttl', '0 3 * * *',
|
||||
$$DELETE FROM notifications WHERE created_at < now() - interval '90 days'$$);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. The dual-write seam (the mechanic that makes it safe)
|
||||
|
||||
```ts
|
||||
// repositories/dual/DualWritePaymentRepo.ts
|
||||
export class DualWritePaymentRepo implements IPaymentRepo {
|
||||
constructor(private mongo: IPaymentRepo, private pg: IPaymentRepo) {}
|
||||
|
||||
// READS: source of truth = Mongo until cutover
|
||||
findById(id) { return this.mongo.findById(id); }
|
||||
|
||||
// WRITES: both, idempotently. Mongo first (authoritative); PG must not break the request.
|
||||
async create(input) {
|
||||
const m = await this.mongo.create(input); // returns doc incl. _id
|
||||
try {
|
||||
await this.pg.upsertFromMongo(m); // keyed by legacyObjectId / idempotencyKey
|
||||
} catch (e) {
|
||||
metrics.dualWriteError('payments', 'create', e); // alert, do NOT throw
|
||||
}
|
||||
return m;
|
||||
}
|
||||
|
||||
async update(id, patch) {
|
||||
const m = await this.mongo.update(id, patch);
|
||||
try { await this.pg.upsertFromMongo(m); } catch (e) { metrics.dualWriteError('payments','update',e); }
|
||||
return m;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- **Mongo write is authoritative and must succeed**; PG write failures are logged + alerted, never surfaced to the user, during `dual` mode. (Once in `pg` mode, PG is authoritative and wrapped in real transactions.)
|
||||
- All PG writes are **idempotent upserts** keyed on `legacyObjectId` (or natural idempotency keys: `Payment` partial-unique set, `FundsLedgerEntry.idempotencyKey`). This lets backfill and live dual-write overlap without double-insert.
|
||||
- `$inc`/`$push` translate inside the repo: `$inc points` → `UPDATE … SET points = points + $1` in a transaction; `$push offers` → `INSERT INTO purchase_request_offers …`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Phased execution
|
||||
|
||||
Same phases as the guide §2, here with Drizzle-concrete entry/exit gates. Each phase ends with a collection in `pg` mode and dual-write removed only after the soak.
|
||||
|
||||
### Phase 0 — Foundations (2–5 wk) — *no data moves*
|
||||
- Stand up Postgres (per env), Drizzle, drizzle-kit, CI migrations. **Status 2026-05-31:** implemented in code and dev stack, but migrations must still be applied per target DB.
|
||||
- Build repository interfaces + `MongoRepo` wrappers for the relational-core domains (refactor services to call repos, not Mongoose directly). **Status 2026-05-31:** repo interfaces/implementations exist; service-layer wiring remains the bulk of the cutover risk.
|
||||
- Create `id_map`, the verification harness (§7), and the backfill batch runner skeleton.
|
||||
- **Exit:** all relational-core services call repositories; PG reachable everywhere; `id_map` + verify harness exist; CI runs migrations.
|
||||
|
||||
### Phase 1 — Address pilot (1–2 wk)
|
||||
- Smallest real domain; proves backfill → dual-write → verify → cutover end-to-end.
|
||||
- Reimplement the **one-primary-per-user** pre-save invariant as either a partial unique index `UNIQUE (user_id) WHERE primary = true` or a trigger.
|
||||
- **Exit:** `addresses` in `pg` mode in prod, invariant proven under concurrent writes, verify green, dual-write removed.
|
||||
|
||||
### Phase 2 — Reference/config (2–3 wk)
|
||||
- `Category` (self-FK, soft-delete), `LevelConfig`, `ConfigSetting`, `ConfigSettingHistory`, `ShopSettings`, `Review`.
|
||||
- Port seeds to run in dependency order. Enforce `ShopSettings.sellerId` unique, Category `parentId` ON DELETE SET NULL.
|
||||
- **Exit:** these read from PG; seeds run in PG.
|
||||
|
||||
### Phase 3 — User + auth core (3–5 wk)
|
||||
- `User` is the FK hub — **must precede the money core** so `id_map` for users is authoritative.
|
||||
- Normalize `profile`/`preferences`/`points`/`referralStats` into columns; extract `passkeys[]`, `refreshTokens[]` to child tables; partial-unique `email`/`referralCode`; reimplement `toJSON()` stripping; passkey `default: Date.now()` in app code.
|
||||
- Redis session/rate-limit + in-memory passkey challenge store stay as-is.
|
||||
- **Exit:** `users` in `pg` mode; referral self-FK intact; all auth flows pass; user uuids authoritative in `id_map`.
|
||||
|
||||
### Phase 4 — Money core (6–10 wk) — *the point of the project*
|
||||
- `PurchaseRequest`, `SellerOffer`, `Payment`, `FundsLedgerEntry`, `DerivedDestination`, `TrezorAccount`, `PointTransaction`.
|
||||
- Apply §4.1 (Mixed→discriminator+FK), §4.2 (offers/preferredSellers junctions, deliveryInfo/serviceInfo child tables), §4.5 (derivation counter).
|
||||
- **Wrap in real PG transactions the multi-doc writes that today have none:** `raiseDispute` (PurchaseRequest + Payment), payment confirm + `FundsLedgerEntry` AML-fee insert, referral reward (points + referralStats), PointsService flows (migrate its 2 `withTransaction` sites to PG `BEGIN/COMMIT`).
|
||||
- Preserve the `Payment` partial-unique idempotency index and `FundsLedgerEntry.idempotencyKey` uniqueness.
|
||||
- **Exit:** money core in `pg` mode; checksum equality on `funds_ledger_entries` sums & `payments` amounts across a full soak; idempotency + escrow-hold invariants pass concurrency tests.
|
||||
|
||||
### Phase 5 — Dispute + delivery (2–4 wk)
|
||||
- `Dispute.evidence[]`/`timeline[]` → child tables; pre-save timeline-append → explicit INSERT; delivery `$set/$push` nested updates → SQL.
|
||||
- `Dispute ↔ Chat` becomes a **cross-store call** (Chat stays on Mongo) — define the boundary API.
|
||||
- **Exit:** dispute lifecycle in `pg` mode; release-hold sync transactional.
|
||||
|
||||
### Phase 6 (deferred / optional) — `RequestTemplate`, `BlogPost`
|
||||
- Behind a search abstraction; `$regex` → PG trigram/FTS only if migrated. Otherwise leave on Mongo.
|
||||
|
||||
### Permanent on Mongo
|
||||
`Chat`, `Notification`, `TelegramSession`, `TempVerification`, `TelegramLink` link-state. Revisit only if dual-stack ops cost exceeds migration cost.
|
||||
|
||||
---
|
||||
|
||||
## 7. Verification (gate for every cutover)
|
||||
|
||||
Three layers, **all green before any read flip**:
|
||||
|
||||
1. **Row counts** — per collection and per FK relationship, Mongo vs PG. Catches dropped/dangling rows. Run continuously during dual-write.
|
||||
2. **Checksums** — column-level hashes; special attention to financial sums (`SUM(funds_ledger_entries.amount)`, `SUM(payments.amount)` grouped by status/provider) and the partial-unique idempotency set.
|
||||
3. **Shadow reads** — in prod, serve from Mongo, asynchronously read PG for the same key, diff, alert on mismatch. **A clean shadow-read window (e.g. 7 days, zero diffs on hot paths) is the exit criterion for cutover.**
|
||||
|
||||
```ts
|
||||
// verify/shadow.ts — wrap a repo read in dual mode
|
||||
async function shadowRead(key, mongoFn, pgFn) {
|
||||
const m = await mongoFn(key);
|
||||
pgFn(key).then(p => { if (!deepEqualNormalized(m, p)) metrics.shadowMismatch(key, diff(m, p)); })
|
||||
.catch(e => metrics.shadowError(key, e));
|
||||
return m; // user always gets Mongo result
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Cutover & rollback runbook (per collection)
|
||||
|
||||
1. **Backfill** in batches with checkpointing; allocate uuids → `id_map`; remap FKs from already-migrated parents. Re-runnable (idempotent upserts).
|
||||
2. **Enable `dual`** (flag) — writes go to both; shadow-read diffing on. Backfill the delta accumulated during step 1.
|
||||
3. **Soak** until row-count + checksum + shadow-read are clean for the agreed window.
|
||||
4. **Flip reads to `pg`** (flag). Keep dual-write on.
|
||||
5. **Soak again** (shorter). Rollback = flip reads back to `mongo`; data still mirrored, so rollback is instant.
|
||||
6. **Decommission**: stop writing Mongo for that collection; archive the collection.
|
||||
|
||||
> Near-zero downtime: there is no global write freeze except, optionally, a brief one during final ledger reconciliation for the money core.
|
||||
|
||||
---
|
||||
|
||||
## 9. First two weeks — concrete starter checklist
|
||||
|
||||
- [ ] Add `drizzle-orm`, `pg`, `drizzle-kit`; create `src/db/{schema,client.ts,migrations}` + `drizzle.config.ts`.
|
||||
- [x] Provision Postgres in dev (compose) + define `PG_URL`; keep Mongo running alongside. Use Postgres 18 volume mount `/var/lib/postgresql`, not `/var/lib/postgresql/data`.
|
||||
- [ ] Write `id_map` schema; generate + run the first migration in CI.
|
||||
- [ ] Define `IAddressRepo`; implement `MongoAddressRepo` by moving the existing Mongoose calls behind it; refactor address service to use the repo. **No behavior change** — prove the seam is invisible (existing tests pass).
|
||||
- [ ] Build the verification harness (row count + checksum) against `addresses`.
|
||||
- [ ] Author `addresses` Drizzle schema (incl. one-primary partial unique index) + `DrizzleAddressRepo` + `DualWriteAddressRepo`.
|
||||
- [ ] Write the batch backfill for `addresses`; run dev backfill; confirm verify is green.
|
||||
- [ ] Flip dev to `dual`, then `pg`; document the flag flips. This is the template for all later phases.
|
||||
|
||||
---
|
||||
|
||||
## 10. Effort recap (from the guide)
|
||||
|
||||
| Scope | Eng-weeks | Notes |
|
||||
|---|---|---|
|
||||
| **Partial — money/relational core (Phases 0–5 + cross-cutting)** | **~16–28** | Recommended stopping point; captures ~90% of value (ACID money + relational integrity). |
|
||||
| Full — all 23 collections | ~23–40 | Extra 7–12+ wks mostly buys Chat/Notification normalization the access patterns don't reward. |
|
||||
|
||||
Add ~20% contingency for data-audit surprises in the Mixed-id fields. One focused engineer assumed; parallelize to compress wall-clock, not effort.
|
||||
|
||||
---
|
||||
|
||||
> [!warning] Before trusting the code sketches
|
||||
> Drizzle schemas above use the real field names from `backend/src/models/` but are **first-pass sketches** — confirm the full `Payment.status` enum, the exact `amount` precision/scale your tokens need (USDT/USDC decimals), and audit which `Mixed` rows are actually strings vs ObjectIds **before** writing the money-core migration. See [[MongoDB to PostgreSQL Migration Guide]] §3/§5 for the authoritative per-field detail.
|
||||
@@ -10,6 +10,9 @@ aliases: [Payment Record, Escrow, IPayment]
|
||||
|
||||
Records every monetary movement in the marketplace: buyer pay-ins, seller payouts, and refunds. The current model is centered on Request Network pay-in, in-house checkout metadata, on-chain transaction verification, escrow state, and provider request IDs. The `provider` and `direction` discriminators let one collection hold incoming buyer payments, outgoing seller releases, refunds, and legacy/other provider records.
|
||||
|
||||
> [!warning] Runtime store
|
||||
> The `Payment` document is still created, read, and updated through Mongoose on normal request paths. Backend `2.6.80` can persist oracle quotes to Postgres `payment_quotes`, but that is conditional on `ORACLE_QUOTING_ENABLED=true` and does not make the whole payment domain PG-authoritative. See [[Postgres Runtime Cutover Status]].
|
||||
|
||||
> [!note] Source
|
||||
> `backend/src/models/Payment.ts:3` — schema definition
|
||||
> `backend/src/models/Payment.ts:257` — model export (default export)
|
||||
@@ -121,7 +124,7 @@ Defined at `backend/src/models/Payment.ts:174-188`:
|
||||
|
||||
## Postgres Quote Table
|
||||
|
||||
The Postgres money-core branch stores oracle quotes in `payment_quotes`, a 1:1 child table keyed by `payment_id → payments.id`. Amount/rate columns use `numeric(38,18)` and the route resolves the PG parent through `payments.legacy_object_id` or `id_map` during the Mongo/PG dual-write window. If the PG payment row is missing, the quote is mirrored to this Mongo `quote` subdocument and a `pg_dualwrite_gaps` row is recorded for reconciliation.
|
||||
The Postgres money-core branch can store oracle quotes in `payment_quotes`, a 1:1 child table keyed by `payment_id → payments.id`. Amount/rate columns use `numeric(38,18)` and the route resolves the PG parent through `payments.legacy_object_id` or `id_map` during the Mongo/PG dual-write window. If the PG payment row is missing, the quote is mirrored to this Mongo `quote` subdocument and a `pg_dualwrite_gaps` row is recorded for reconciliation. This table is quote/audit storage only until the payment service itself is wired through the PG repository path.
|
||||
|
||||
## Pre/Post Hooks
|
||||
|
||||
|
||||
68
02 - Data Models/Postgres Runtime Cutover Status.md
Normal file
68
02 - Data Models/Postgres Runtime Cutover Status.md
Normal file
@@ -0,0 +1,68 @@
|
||||
---
|
||||
title: Postgres Runtime Cutover Status
|
||||
tags: [data-model, postgres, migration, runtime-status]
|
||||
aliases: [Postgres Status, PG Cutover Status, Mongo vs Postgres Runtime]
|
||||
created: 2026-05-31
|
||||
source: backend origin/integrate-main-into-development@cab0719
|
||||
---
|
||||
|
||||
# Postgres Runtime Cutover Status
|
||||
|
||||
> **Current branch:** backend `origin/integrate-main-into-development` at `cab0719`, version `2.6.80`.
|
||||
>
|
||||
> **Bottom line:** this branch is **Postgres-capable**, not fully Postgres-backed. MongoDB remains the live source of truth for normal app traffic until services are actually rewired through the repository factory and cutover flags are verified in a running environment.
|
||||
|
||||
## What Uses Postgres Now
|
||||
|
||||
| Area | Runtime status | Notes |
|
||||
|---|---|---|
|
||||
| Postgres connection | Available when `PG_URL` is set | `src/db/client.ts` fail-fast requires an explicit DSN before importing PG code. |
|
||||
| Schema/migrations | Implemented through migration `0008` | Drizzle schemas cover users, categories, purchase requests, seller offers, payments, funds ledger, derived destinations, trezor accounts, points, `id_map`, `pg_dualwrite_gaps`, and `payment_quotes`. |
|
||||
| Repository implementations | Implemented but not broadly called | `src/db/repositories/{mongo,drizzle,dual}` and `factory.ts` exist for user, payment, points, and marketplace domains. |
|
||||
| Oracle quote persistence | Conditional runtime PG write | `/api/payment/request-network/intents` lazily imports `quoteRepo` only when `ORACLE_QUOTING_ENABLED=true`; it writes `payment_quotes` if the PG parent payment row exists, mirrors to Mongo `Payment.quote`, and records `pg_dualwrite_gaps` if PG is behind. |
|
||||
| Backfill/verify scripts | Available as operator tooling | `MIGRATION_PG_URL` drives backfill scripts; guards restrict allowed target hosts. These scripts are not run automatically by app startup. |
|
||||
|
||||
## What Is Still Mongo-Backed
|
||||
|
||||
The service layer still imports Mongoose models directly. A source scan on 2026-05-31 found no runtime calls to `createRepositories()` / `getPaymentRepo()` / `getMarketplaceRepo()` outside `src/db/repositories/factory.ts`; the only normal service import of `src/db/client.ts` is `priceOracle/quoteRepo.ts`.
|
||||
|
||||
| Domain | Current live store | Why not Postgres yet |
|
||||
|---|---|---|
|
||||
| Auth, users, passkeys, refresh tokens | MongoDB | Auth/user controllers call `User` and related Mongoose models directly. |
|
||||
| Marketplace requests/offers/templates/categories/shop settings | MongoDB | Marketplace, checkout, and seller-offer services call `PurchaseRequest`, `SellerOffer`, `RequestTemplate`, `Category`, and `ShopSettings` directly. |
|
||||
| Payments and escrow state | MongoDB primary | Request Network, AMN scanner, webhook, admin, release/refund, adapter, reconciliation, and legacy payment paths still create/update `Payment` Mongoose documents directly. |
|
||||
| Funds ledger | MongoDB primary | `FundsLedgerEntry` remains the ledger used by current services; PG ledger tables exist but are not the live write target. |
|
||||
| Derived destinations and sweeps | MongoDB | Wallet destination allocation and sweep services call `DerivedDestination` directly. |
|
||||
| Points/referrals/levels | MongoDB | Points service uses `User`, `PointTransaction`, `LevelConfig`, and Mongo transactions. |
|
||||
| Chat/messages | MongoDB | Chat intentionally remains document-shaped and is not part of the current PG cutover. |
|
||||
| Notifications | MongoDB | Notification TTL/read-state paths remain Mongo-backed. |
|
||||
| Disputes/reviews/blog/content/admin cleanup | MongoDB | These services still call their Mongoose models directly. |
|
||||
| Runtime config | MongoDB | `ConfigSetting` and `ConfigSettingHistory` remain Mongo-backed for admin-editable configuration. |
|
||||
| Telegram link/session/temp verification | MongoDB | These session/link/TTL records are not cut over to PG. |
|
||||
|
||||
## Env Flag Reality
|
||||
|
||||
| Flag | Current meaning |
|
||||
|---|---|
|
||||
| `PG_URL` | Makes PG code importable/reachable; does not cut over app domains by itself. |
|
||||
| `MIGRATION_PG_URL` | Used by backfill scripts and migration runbooks; not part of normal request handling. |
|
||||
| `REPO_USER`, `REPO_PAYMENT`, `REPO_POINTS`, `REPO_MARKETPLACE`, `REPO_DEFAULT` | Repository factory flags exist, but broad services are not yet wired through the factory. Treat them as migration controls that need integration verification before relying on them. |
|
||||
| `ORACLE_QUOTING_ENABLED` | Enables server-side quote computation and the only current PG write path in normal checkout: `payment_quotes`, when a PG parent row can be resolved. |
|
||||
|
||||
## Next Cutover Work
|
||||
|
||||
1. Apply Drizzle migrations to the target Postgres database.
|
||||
2. Run non-prod backfills in dependency order and record row-count/checksum results.
|
||||
3. Wire services to repository interfaces one domain at a time.
|
||||
4. Enable `dual` mode per domain only after wiring is proven by tests and smoke checks.
|
||||
5. Run shadow-read/reconcile during a soak window.
|
||||
6. Flip reads to `pg` per domain only after zero-diff shadow reads and a rollback plan are in place.
|
||||
|
||||
## Related Docs
|
||||
|
||||
- [[Database Strategy - Mongo vs Postgres Assessment]]
|
||||
- [[MongoDB to PostgreSQL Migration Plan (Drizzle)]]
|
||||
- [[Payment]]
|
||||
- [[Payment API]]
|
||||
- [[Environment Variables]]
|
||||
- [[Database Operations]]
|
||||
@@ -6,7 +6,7 @@ aliases: [Purchase Request, Buy Request, IPurchaseRequest]
|
||||
|
||||
# PurchaseRequest
|
||||
|
||||
> **Last updated:** 2026-05-30 — `budget.currency` locked to USDT; `categoryId` added to `IRequestTableItem`
|
||||
> **Last updated:** 2026-05-31 — `budget.currency` aligned with template/Postgres enum (`USD`, `EUR`, `IRR`, `USDT`, `USDC`); `categoryId` added to `IRequestTableItem`
|
||||
|
||||
The central buyer-side document. A `PurchaseRequest` captures what a buyer wants to acquire (physical product, digital product, service, or consultation), the budget envelope, urgency, delivery details, and the entire lifecycle from creation through payment, delivery, and completion. Sellers respond by attaching [[SellerOffer]] documents; the buyer accepts one, a [[Payment]] is opened, and delivery is verified by a 6-digit code.
|
||||
|
||||
@@ -31,7 +31,7 @@ The central buyer-side document. A `PurchaseRequest` captures what a buyer wants
|
||||
| `quantity` | Number | no | `1` | min 1 | — | Unit count. |
|
||||
| `budget.min` | Number | no | — | min 0 | — | Lower bound. |
|
||||
| `budget.max` | Number | no | — | min 0 | — | Upper bound. |
|
||||
| `budget.currency` | String | no | `USDT` | enum: `USDT` (escrow MVP — USD/EUR/IRR removed in commit 3aaa2fe) | — | Budget currency. |
|
||||
| `budget.currency` | String | no | `USDT` | enum: `USD` / `EUR` / `IRR` / `USDT` / `USDC` | — | Budget currency. Runtime Mongoose validation, request-template validation, and the PG `budget_currency` enum now share these values. |
|
||||
| `urgency` | String | no | `medium` | enum: `low` / `medium` / `high` / `urgent` | yes | Buyer urgency. |
|
||||
| `status` | String | no | `pending` | enum (13 values — see State Transitions below) | yes | Lifecycle state. |
|
||||
| `isPublic` | Boolean | no | `true` | — | — | Public marketplace listing vs. private request. |
|
||||
|
||||
@@ -28,7 +28,7 @@ A reusable template authored by a seller. When a buyer visits the template's `sh
|
||||
| `quantity` | Number | no | `1` | min 1 | — | Default unit count. |
|
||||
| `budget.min` | Number | no | — | min 0 | — | Lower bound. |
|
||||
| `budget.max` | Number | no | — | min 0 | — | Upper bound. |
|
||||
| `budget.currency` | String | no | `USD` | enum: `USD` / `EUR` / `IRR` | — | Currency. |
|
||||
| `budget.currency` | String | no | `USDT` | enum: `USD` / `EUR` / `IRR` / `USDT` / `USDC` | — | Currency. Shared with [[PurchaseRequest]] so a template can be converted without enum drift. |
|
||||
| `urgency` | String | no | `medium` | enum: `low` / `medium` / `high` / `urgent` | — | Urgency. |
|
||||
| `tags[]` | String[] | no | — | trim | — | Tags. |
|
||||
| `specifications[].key` | String | yes | — | trim | — | Spec key. |
|
||||
|
||||
Reference in New Issue
Block a user