From d072238fe8e5656debac36db615ee67544a950d9 Mon Sep 17 00:00:00 2001 From: Siavash Sameni Date: Wed, 3 Jun 2026 10:29:48 +0400 Subject: [PATCH] docs: update PG migration status, data models, architecture + add Telegram Mini App flow (v2.8.59) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Postgres Runtime Cutover Status: 17 migrations (0000–0017), dual-write repo matrix - Backend Architecture: dual-DB architecture, repo factory, MONGO_CONNECT_MODE modes - Data Model Overview: 23-model index with PG table names and migration status - User, PurchaseRequest, SellerOffer, Chat, Dispute: Drizzle schema + cutover status added - 04 - Flows/Telegram Mini App.md: new doc covering Mini App architecture and flows - mongo-to-pg-migration-prd.md: status block prepended with 2026-06-03 milestone tracking Co-Authored-By: Claude Sonnet 4.6 --- 01 - Architecture/Backend Architecture.md | 127 ++++- 02 - Data Models/Chat.md | 84 +++- 02 - Data Models/Data Model Overview.md | 193 ++++++-- 02 - Data Models/Dispute.md | 100 +++- .../Postgres Runtime Cutover Status.md | 195 ++++++-- 02 - Data Models/PurchaseRequest.md | 288 ++++++++++- 02 - Data Models/SellerOffer.md | 104 +++- 02 - Data Models/User.md | 188 ++++++-- 04 - Flows/Telegram Mini App.md | 441 +++++++++++++++++ mongo-to-pg-migration-prd.md | 449 ++++++++++++++++++ 10 files changed, 1998 insertions(+), 171 deletions(-) create mode 100644 04 - Flows/Telegram Mini App.md create mode 100644 mongo-to-pg-migration-prd.md diff --git a/01 - Architecture/Backend Architecture.md b/01 - Architecture/Backend Architecture.md index 0ac679e..ba6d0d1 100644 --- a/01 - Architecture/Backend Architecture.md +++ b/01 - Architecture/Backend Architecture.md @@ -2,14 +2,15 @@ title: Backend Architecture tags: [architecture, backend] created: 2026-05-23 +updated: 2026-06-03 --- # Backend Architecture -Module-level architecture of the Express 5 + TypeScript backend. MongoDB/Mongoose is still the primary runtime persistence layer; the `integrate-main-into-development` backend also contains the Drizzle/Postgres migration layer. +Module-level architecture of the Express 5 + TypeScript backend. The system is mid-migration: MongoDB/Mongoose remains the authoritative read store for most domains, with PostgreSQL (Drizzle ORM) running in dual-write mode across 17 landed migrations. The repository factory pattern (`src/db/repositories/factory.ts`) controls which backend each domain reads and writes through env flags. > [!info] -> Repo: `git@git.manko.yoga:222/nick/backend.git` · Active integration branch: `integrate-main-into-development` · Current baseline: backend `2.6.79` at `3a50dc4` +> Repo: `git@git.manko.yoga:222/nick/backend.git` · Current version: `2.8.56` · 17 Drizzle migrations landed · Dual-write active across all major domains --- @@ -24,10 +25,13 @@ backend/src/ │ ├── database/ # Mongoose connection, retries, graceful shutdown │ └── socket/socketService.ts # Socket.IO server, rooms, emit helpers ├── models/ # Mongoose models — see 02 - Data Models/ -├── db/ # Drizzle/Postgres migration layer: schemas, migrations, repos, backfill, verify +├── db/ # Drizzle/Postgres layer: schemas, migrations, repos, backfill, verify +│ ├── schema/ # Per-table Drizzle schema files + index.ts barrel +│ ├── migrations/ # 17 numbered SQL migration files +│ └── repositories/ # Drizzle repos, dual-write wrappers, factory.ts ├── routes/ # Express Router definitions (mounted in app.ts) ├── scripts/ # CLI utilities (seed:users, seed:categories, ...) -├── seeds/ # Seed data fixtures +├── seeds/ # Seed data fixtures (Postgres-capable as of v2.8.47) ├── services/ │ ├── ai/ # OpenAI integration (descriptions, moderation) │ ├── auth/ # JWT, OAuth, Passkey, password reset @@ -61,8 +65,8 @@ backend/src/ └── utils/ # Pure utility fns (logger, currencyUtils, etc.) ``` -> [!warning] Postgres is not the default runtime store yet -> `src/db/repositories/factory.ts` can select `mongo`, `dual`, or `pg` implementations for user, payment, points, and marketplace domains, but the broad service layer still imports Mongoose models directly. A code scan on 2026-05-31 found no runtime calls to `createRepositories()` / `getPaymentRepo()` / `getMarketplaceRepo()` outside the factory itself. See [[Postgres Runtime Cutover Status]] before assuming a `REPO_*` flag changes live behavior. +> [!warning] Reads still go to Mongo for all dual-write domains +> Even when `REPO_*=dual`, Mongo is the authoritative read source. The dual-write seam exists and is exercised in production, but no domain has been cut over to PG reads yet. See [[Postgres Runtime Cutover Status]] before assuming a `REPO_*` flag changes live read behavior. > [!tip] > Service folders are self-contained: each typically has `Service.ts`, `Controller.ts`, `Routes.ts`, `Validation.ts`. This makes each service movable to a microservice later with minimal coupling. @@ -75,22 +79,25 @@ The bootstrap is intentionally linear and easy to audit. Execution order: 1. **Imports & env load** — `dotenv` (if used), then `import { config } from './shared/config'`. 2. **Express app construction** — `const app = express();` -3. **Trust proxy** — `app.set('trust proxy', config.trustProxy)` so X-Forwarded-For works behind Nginx. +3. **Trust proxy** — `app.set('trust proxy', config.trustProxy)` so X-Forwarded-For works behind Traefik. 4. **Security headers** — `app.use(helmet({ ... }))`. 5. **CORS** — `cors({ origin: config.frontendUrl, credentials: true, methods: [...] })`. 6. **Body parsers** — `express.json({ limit: '10mb' })`, `express.urlencoded({ extended: true })`. 7. **Static uploads** — `app.use('/uploads', express.static(uploadDir))`. -8. **Health endpoint** — `GET /health` for Docker healthcheck and external monitors. +8. **Health endpoint** — `GET /health` for Docker healthcheck and external monitors. Now surfaces active Postgres store modes. 9. **Route mounting** — every `/api/*` route registered before the error handler. 10. **404 handler** — catches unmatched `/api/*`. 11. **Error handler** — central `errorHandler` middleware formats responses via `response-handler.ts`. 12. **HTTP server creation** — `const server = http.createServer(app)`. 13. **Socket.IO attach** — `initSocket(server, corsOptions)` (see [[Real-time Layer]]). -14. **DB connect** — `await connectDatabase()` for MongoDB/Mongoose. Postgres connects lazily only when PG modules are imported (for example oracle quote persistence with `ORACLE_QUOTING_ENABLED=true`) and requires `PG_URL`. +14. **DB connect** — controlled by `MONGO_CONNECT_MODE`: + - `always` (default) — connects Mongoose (Mongo) and PostgreSQL (via `PG_URL`) on boot. + - `never` — skips Mongo entirely; Postgres is the only persistence layer. Seeds are Postgres-capable in this mode. + - `optional` — connects Postgres; Mongo is attempted but failures are non-fatal. 15. **Redis connect** — `await connectRedis()`. 16. **Listen** — `server.listen(config.port, ...)`. 17. **Graceful shutdown** — SIGTERM/SIGINT handlers close server, drain sockets, close Mongoose, close Redis. -18. **Optional dev seeding** — when `NODE_ENV === 'development'` and `SEED_USERS !== 'false'`, the bootstrap calls the seed scripts to provision default test users. +18. **Optional dev seeding** — when `NODE_ENV === 'development'` and `SEED_USERS !== 'false'`, the bootstrap calls the seed scripts to provision default test users. Seeds are store-aware and run correctly against both Mongo and PG. --- @@ -104,14 +111,14 @@ The bootstrap is intentionally linear and easy to audit. Execution order: | 4 | `morgan` (dev only) | global | HTTP request log to stdout. | | 5 | `requestId` | global | Adds `X-Request-Id` for log correlation. | | 6 | `authMiddleware` | per-route | Verifies JWT, attaches `req.user`. Mounted only on protected routes. | -| 7 | `roleGuard('admin'|'seller'|...)` | per-route | RBAC check after auth. | +| 7 | `roleGuard('admin'\|'seller'\|'guard'\|...)` | per-route | RBAC check after auth. Roles: `admin`, `buyer`, `seller`, `resolver`, `guard`. | | 8 | `validate(schema)` | per-route | express-validator + zod inputs. | | 9 | `controllerFn` | per-route | Delegates to service layer. | | 10 | `notFound` | tail | Returns 404 envelope for unmatched routes. | | 11 | `errorHandler` | tail | Catches thrown errors, formats response. | > [!note] -> Rate-limit middleware is **active** as of 2026-05-24: auth 10 req/15 min, payment 30/15 min, AI 20/15 min, global 100/15 min. Request Network and Telegram webhooks are exempt from the global limiter. Counters are in-memory — a Redis adapter is planned for distributed deployments. +> Rate-limit middleware is **active**: auth 10 req/15 min, payment 30/15 min, AI 20/15 min, global 100/15 min. `GET /api/payment/:id` is exempt from the payment limiter (polling route). Request Network and Telegram webhooks are exempt from the global limiter. Counters are in-memory — a Redis adapter is planned for distributed deployments. --- @@ -128,7 +135,7 @@ The full route table mounted by `app.ts`: | `/api/marketplace/offers` | `services/marketplace/controllerRoutes.ts` | JWT (seller) | SellerOffer CRUD | | `/api/marketplace/templates` | `services/marketplace/controllerRoutes.ts` | JWT (seller) | RequestTemplate CRUD | | `/api/marketplace/categories` | `services/marketplace/controllerRoutes.ts` | public read | Category list | -| `/api/marketplace/shop-settings` | `services/marketplace/shopSettingsController.ts` | JWT (seller) | Shop profile | +| `/api/marketplace/shop-settings` | `services/marketplace/shopSettingsController.ts` | JWT (seller) | Shop profile; lookup tolerant of uuid/legacy id formats | | `/api/payment` | `services/payment/paymentControllerRoutes.ts` + `paymentRoutes.ts` | JWT | Payment CRUD, health, export | | `/api/payment/decentralized` | `services/payment/decentralizedPaymentRoutes.ts` | mixed | Legacy/manual Web3 save, verify, receiver | | `/api/payment/request-network` | `services/payment/requestNetwork/requestNetworkRoutes.ts` | mixed + HMAC sig on webhook | Request Network pay-in creation, in-house checkout rehydrate, webhooks | @@ -136,12 +143,12 @@ The full route table mounted by `app.ts`: | `/api/admin/rn/networks` | `services/payment/requestNetwork/networkRegistryRoutes.ts` | JWT (admin) | Supported RN chain/token registry | | `/api/admin/settings/confirmation-thresholds` | `services/admin/confirmationThresholdRoutes.ts` | JWT (admin) | Runtime min-confirmation thresholds | | `/api/admin/payments/awaiting-confirmation` | `services/admin/awaitingConfirmationRoutes.ts` | JWT (admin) | Payments blocked on safety confirmations | -| `/api/telegram` | `services/telegram/telegramRoutes.ts` | mixed (some JWT, webhook uses secret-token) | Mini App verify/session, identity link/unlink, bot webhook | +| `/api/telegram` | `services/telegram/telegramRoutes.ts` | mixed (some JWT, webhook uses secret-token) | Mini App verify/session, identity link/unlink, bot webhook; notifications delivered via Telegram as of v2.8.56 | | `/api/chat` | `services/chat/chatRoutes.ts` | JWT | Conversations, messages | | `/api/notification` | `services/notification/notificationRoutes.ts` + `notificationControllerRouter` | JWT | List, mark read | | `/api/disputes` | `routes/disputeRoutes.ts` + `services/dispute/disputeRoutes.ts` | JWT | Dispute CRUD plus release-hold helpers | | `/api/blog` | `services/blog/blogRoutes.ts` | mixed | Public reads, admin writes | -| `/api/admin/cleanup` | `services/admin/dataCleanupRoutes.ts` | JWT (admin) | Data cleanup operations | +| `/api/admin/cleanup` | `services/admin/dataCleanupRoutes.ts` | JWT (admin) | Data cleanup; scoped by provider to avoid wiping RN/multi-seller records | | `/api/points` | `services/points/pointsRoutes.ts` | JWT | Points, levels, referrals | | `/api/ai` | `services/ai/aiRoutes.ts` | JWT | OpenAI-backed helpers | | `/api/files` | `services/file/fileRoutes.ts` | JWT | Multipart upload | @@ -209,10 +216,11 @@ flowchart TB points -.-> notify notify --> socket notify --> email + notify --> telegram ``` > [!note] -> `socket` and `email` are leaf services — every notification path funnels through them. Mocking these two in tests covers most side-effect verification. +> `socket`, `email`, and `telegram` are leaf notification sinks — every notification path funnels through them. Mocking these three in tests covers most side-effect verification. Telegram notification delivery was added in v2.8.56. --- @@ -254,12 +262,26 @@ Full table in [[Environment Variables]]. Critical ones: | Key | Default | Notes | |---|---|---| | `PORT` | `5001` | Listen port | -| `MONGODB_URI` | `mongodb://localhost:27017/nickapp` | Includes db name | +| `MONGODB_URI` | `mongodb://localhost:27017/nickapp` | Includes db name; not required when `MONGO_CONNECT_MODE=never` | +| `MONGO_CONNECT_MODE` | `always` | `always` \| `never` \| `optional` — controls whether Mongoose connects on boot | +| `PG_URL` | required for PG | PostgreSQL connection string for Drizzle; required when any `REPO_*=pg\|dual` | | `REDIS_URI` | `redis://localhost:6379` | + `REDIS_PASSWORD` | | `JWT_SECRET` | required | ≥32 chars | | `JWT_EXPIRES_IN` | `7d` | | | `REFRESH_TOKEN_EXPIRES_IN` | `30d` | | | `FRONTEND_URL` | `http://localhost:3000` | CORS origin | +| `REPO_DEFAULT` | `mongo` | Global fallback store mode for all domains (`mongo` \| `dual` \| `pg`) | +| `REPO_USER` | inherits `REPO_DEFAULT` | Per-domain override for user store | +| `REPO_PAYMENT` | inherits `REPO_DEFAULT` | Per-domain override for payment store | +| `REPO_POINTS` | inherits `REPO_DEFAULT` | Per-domain override for points store | +| `REPO_MARKETPLACE` | inherits `REPO_DEFAULT` | Per-domain override for marketplace store | +| `REPO_TREZOR` | inherits `REPO_DEFAULT` | Per-domain override for trezor store | +| `REPO_DERIVED_DESTINATION` | inherits `REPO_DEFAULT` | Per-domain override for derived destination store | +| `REPO_BLOG` \| `BLOG_STORE` | inherits `REPO_DEFAULT` | Per-domain override for blog store | +| `REPO_NOTIFICATION` \| `NOTIFICATION_STORE` | inherits `REPO_DEFAULT` | Per-domain override for notification store | +| `REPO_DISPUTE` \| `DISPUTE_STORE` | inherits `REPO_DEFAULT` | Per-domain override for dispute store | +| `REPO_CHAT` \| `CHAT_STORE` | inherits `REPO_DEFAULT` | Chat dual-write not implemented; `dual` silently uses Mongo | +| `REPO_RELEASE_HOLD` \| `RELEASE_HOLD_STORE` | inherits `REPO_DEFAULT` | Release-hold dual-write not implemented; `dual` silently uses Mongo | | `REQUEST_NETWORK_API_BASE_URL` | `https://api.request.network` | Request Network API | | `REQUEST_NETWORK_API_KEY` | required | Request Network API credential | | `REQUEST_NETWORK_WEBHOOK_SECRET` | required | Webhook HMAC key | @@ -268,16 +290,77 @@ Full table in [[Environment Variables]]. Critical ones: | `DERIVED_DESTINATION_SWEEP_SIGNER` | `build-only` | Target hardware/Safe-backed signer | | `SMTP_*` | required | Nodemailer | | `OPENAI_API_KEY` | required | | +| `ORACLE_QUOTING_ENABLED` | `false` | Enables oracle-based depeg-protected payment quotes; requires `PG_URL` | --- ## 9. Database & connection management -- **Mongoose** is the ODM. Connection in `src/infrastructure/database/`. +The backend runs a **dual-database architecture** during the Mongo→Postgres migration. Both stores may be active simultaneously; which one serves each domain is controlled by `REPO_*` env flags. + +### MongoDB / Mongoose + +- ODM: Mongoose. Connection in `src/infrastructure/database/`. - Connection options enable retryable writes, exponential backoff on reconnect. -- Indexes are defined on each model and auto-created on connect (Mongoose `autoIndex: true` in dev, recommend `false` in prod with explicit migration). +- Indexes defined on each model and auto-created on connect (`autoIndex: true` in dev; recommend `false` in prod with explicit migration scripts). +- Remains the **authoritative read store** for all dual-write domains until read cutover is explicitly executed per domain. - See [[Data Model Overview]] for the relational map and per-model docs. +### PostgreSQL / Drizzle + +- ORM: Drizzle. Schemas in `src/db/schema/`, migrations in `src/db/migrations/` (17 migrations landed as of 2026-06-03). +- Managed via `drizzle-kit migrate` — never edit migration files manually. +- Connects lazily when any PG-capable store is imported, or eagerly on boot when `MONGO_CONNECT_MODE=never`. +- Every migrated table carries a `legacy_object_id text` column with a partial-unique index for idempotent backfill upserts. +- Money columns use `numeric(38,18)` (except `seller_offers`: `numeric(18,8)`). Blockchain balance columns use `numeric(78,0)` to hold uint256 without overflow. +- See [[Drizzle Schema Reference]] for the full per-table breakdown. + +### Repository factory — `src/db/repositories/factory.ts` + +The factory is the single routing layer between service code and the underlying store. It exposes per-domain getters and resolves the mode (`mongo` | `dual` | `pg`) in this order: + +1. Per-domain env flag (e.g. `REPO_PAYMENT`) +2. `REPO_DEFAULT` (global staging-wide fallback) +3. Hardcoded default: `mongo` + +Unrecognized values silently fall back to `mongo` — intentional safety net against typos on money writes. + +| Domain | Getter | Dual-write | PG-only | +|---|---|---|---| +| user | `getUserRepo` | Yes (full trio) | Yes | +| payment | `getPaymentRepo` | Yes (full trio) | Yes | +| points | `getPointsRepo` | Yes (full trio) | Yes | +| marketplace | `getMarketplaceRepo` | Yes (full trio) | Yes | +| trezor | `getTrezorRepo` | Yes (full trio) | Yes | +| derivedDestination | `getDerivedDestinationRepo` | Yes (full trio) | Yes | +| blog | `getBlogRepo` | Yes (full trio) | Yes | +| notification | `getNotificationRepo` | Yes (full trio) | Yes | +| dispute | `getDisputeRepo` | Yes (full trio) | Yes | +| releaseHold | `getReleaseHoldRepo` | No — `dual` silently uses Mongo | Yes | +| chat | `getChatRepo` | No — `dual` silently uses Mongo | Yes | + +> [!warning] `MONGO_CONNECT_MODE` is not handled by the factory +> `MONGO_CONNECT_MODE` is consumed by the Mongoose connection module, not by `factory.ts`. The factory only reads `REPO_*` flags. These two controls are orthogonal: `MONGO_CONNECT_MODE=never` prevents Mongoose from connecting, while `REPO_*=pg` prevents the factory from routing to Mongo. For a full PG-only boot, set **both**. + +### Migration phase status (as of 2026-06-03) + +| Phase | Status | +|---|---| +| Schema / migrations | Done — 17 migrations landed, all domain tables exist in PG | +| Dual-write seam | Done — active for all major domains via factory | +| Backfill tooling | Done — backfill + verification harness in `src/db/` | +| Reads cutover | Not started — all reads still served from Mongo | +| Chat normalization | Blocked — Chat stored as JSONB blobs; normalization required before PG read cutover | +| Mongo retirement | Future — blocked on per-domain read cutover completion | + +### Infrastructure / bridge tables (PG-only) + +- **`id_map`** — ObjectId → UUID bridge; every migrated entity upserts here during backfill/dual-write. +- **`pg_dualwrite_gaps`** — Append-only reconciliation log for failed PG dual-writes; includes severity, resolver notes, and error stack. +- **`payment_quotes`** — Oracle-based depeg-protected quote snapshots (1:1 with payments); PG-only, no Mongo equivalent. Only active when `ORACLE_QUOTING_ENABLED=true`. + +### Redis + Redis client (in `src/services/redis/`) provides: - Session caching (login attempts, lockout counters) - Rate-limit counters (when middleware is enabled) @@ -323,6 +406,8 @@ Run with `npm run test:all`. CI runs the same. Reach for `npm run test:models`, | `src/shared/utils/response-handler.ts` | Standard response shape | | `src/shared/middleware/auth.ts` | JWT verify + RBAC | | `src/infrastructure/socket/socketService.ts` | All socket plumbing | +| `src/db/repositories/factory.ts` | Store routing — which backend each domain uses | +| `src/db/schema/index.ts` | Drizzle schema barrel — all 25+ PG tables | | `src/services/payment/requestNetwork/requestNetworkRoutes.ts` | Request Network checkout and webhook route | | `src/services/payment/ledger/fundsLedgerService.ts` | Immutable payment ledger writes | | `src/services/marketplace/PurchaseRequestService.ts` | Core marketplace state machine | @@ -338,5 +423,7 @@ Run with `npm run test:all`. CI runs the same. Reach for `npm run test:models`, - [[Frontend Architecture]] — how the FE talks to this BE - [[Real-time Layer]] — Socket.IO room model - [[Security Architecture]] — JWT, passkeys, webhook HMAC -- [[Data Model Overview]] — entity-relationship map +- [[Data Model Overview]] — entity-relationship map (Mongoose) +- [[Drizzle Schema Reference]] — PostgreSQL table definitions, enums, migration status +- [[Postgres Runtime Cutover Status]] — per-domain read cutover tracker - [[Authentication Flow]] · [[Escrow Flow]] · [[Dispute Flow]] diff --git a/02 - Data Models/Chat.md b/02 - Data Models/Chat.md index 9006b48..a8c27e1 100644 --- a/02 - Data Models/Chat.md +++ b/02 - Data Models/Chat.md @@ -1,11 +1,11 @@ --- title: Chat -tags: [data-model, mongoose] +tags: [data-model, mongoose, postgres] aliases: [Conversation, IChat, IMessage] --- # Chat -> **Last updated:** 2026-05-29 — aligned with code (see [Doc vs Code Audit Report](../09%20-%20Audits/Doc%20vs%20Code%20Audit%20Report%20-%202026-05-29.md)) +> **Last updated:** 2026-06-03 — added Postgres/Drizzle schema section; migration status clarified. Conversation container with embedded messages. Used for buyer-seller direct chats, group chats, and support tickets. Each chat carries a list of participants, an embedded `messages[]` array (with reactions, replies, edit history), a denormalised `lastMessage` snapshot for list views, and per-user `unreadCounts`. A chat can be linked to any other entity through the `relatedTo` discriminator (currently `PurchaseRequest`, `SellerOffer`, or `Transaction`). @@ -13,6 +13,7 @@ Conversation container with embedded messages. Used for buyer-seller direct chat > `backend/src/models/Chat.ts:130` — chat schema definition > `backend/src/models/Chat.ts:69` — message subdocument schema > `backend/src/models/Chat.ts:348` — model export +> `backend/src/db/schema/chat.ts` — Drizzle/Postgres schema > [!warning] Embedded messages > Messages live inside the chat document. Very long-running chats can grow past the 16 MB document limit. Treat this as a known constraint of the current schema. @@ -20,7 +21,10 @@ Conversation container with embedded messages. Used for buyer-seller direct chat > [!warning] `relatedTo` is NOT set via `POST /api/chat` > Although `relatedTo` exists in the schema, it is **not accepted** by the `POST /api/chat` create endpoint. Purchase-request linkage is established server-side through the dedicated `POST /api/chat/purchase-request`, not by passing `relatedTo` to the generic create endpoint. -## Schema — Chat +> [!danger] Migration status — DUAL-WRITE, reads still on Mongo +> Chat writes go to **both** MongoDB and Postgres (via `DrizzleChatRepo`). However, **all reads still come from MongoDB**. The Postgres `chats` table is a conservative shim: `participants` and `messages` are stored as JSONB blobs, not normalised child tables. Full normalisation (splitting messages into a separate table with proper threading) is a **known open blocker** for the Mongo → PG read cutover. Do not assume PG data is queryable relationally until that work is complete. + +## Schema — Chat (MongoDB / Mongoose) | Field | Type | Required | Default | Validation | Index | Description | | --- | --- | --- | --- | --- | --- | --- | @@ -57,7 +61,7 @@ Conversation container with embedded messages. Used for buyer-seller direct chat > [!note] Soft removal of participants > Removing a participant (via `DELETE /api/chat/:id/participants/:participantId`) does **not** delete the subdocument. It is a soft removal: `isActive` is set to `false` and `leftAt` is timestamped, preserving message attribution and history. -## Schema — Message (embedded) +## Schema — Message (embedded, MongoDB) | Field | Type | Required | Default | Validation | Description | | --- | --- | --- | --- | --- | --- | @@ -80,13 +84,65 @@ Conversation container with embedded messages. Used for buyer-seller direct chat > [!note] Messages are soft-deleted > Deleting a message sets `deletedAt` and clears `content` (the body becomes empty). The message subdocument is **not** physically removed from `messages[]`, and a `message-deleted` socket event is emitted. +## Schema — `chats` table (Postgres / Drizzle) + +> Source: `backend/src/db/schema/chat.ts` + +> [!warning] Conservative JSONB shim — not normalised +> Unlike most other migrated tables, participants and messages are stored as **JSONB blobs** (`ChatParticipant[]` and `ChatMessage[]`), not as separate relational child tables. This was a deliberate trade-off to unblock dual-write without committing to a normalisation design. The normalised schema (separate `chat_messages` and `chat_participants` tables with proper FKs and threading support) is the **primary blocker** for cutting reads over to Postgres. + +### Enums (declared in `_enums.ts`) + +| Enum | Values | +| --- | --- | +| `chat_type` | `direct`, `group`, `support` | +| `chat_participant_role` | `member`, `admin`, `owner` | +| `chat_message_type` | `text`, `image`, `file`, `system` | +| `chat_related_to_type` | `PurchaseRequest`, `SellerOffer`, `Transaction` | + +### Table: `chats` + +| Column | PG type | Nullable | Default | Notes | +| --- | --- | --- | --- | --- | +| `id` | `uuid` | NOT NULL | `gen_random_uuid()` | Primary key | +| `legacy_object_id` | `text` | nullable | — | Mongo ObjectId bridge; partial-unique index WHERE NOT NULL | +| `type` | `chat_type` enum | NOT NULL | `'direct'` | | +| `name` | `text` | nullable | — | Group chat display name | +| `description` | `text` | nullable | — | | +| `participants` | `jsonb` | nullable | — | `ChatParticipant[]` blob — **not normalised** | +| `messages` | `jsonb` | nullable | — | `ChatMessage[]` blob — **not normalised** | +| `related_to` | `jsonb` | nullable | — | `{ type: chat_related_to_type, id: string }` blob | +| `last_message` | `jsonb` | nullable | — | Denormalised snapshot | +| `unread_counts` | `jsonb` | nullable | — | `{ userId, count }[]` blob | +| `settings_is_archived` | `boolean` | nullable | `false` | | +| `settings_is_muted` | `boolean` | nullable | `false` | | +| `settings_muted_until` | `timestamp with time zone` | nullable | — | | +| `settings_notifications` | `boolean` | nullable | `true` | | +| `created_by` | `text` | nullable | — | Mongo ObjectId or UUID string of creator | +| `created_at` | `timestamp with time zone` | NOT NULL | `now()` | | +| `updated_at` | `timestamp with time zone` | NOT NULL | `now()` | | +| `last_activity` | `timestamp with time zone` | nullable | `now()` | Sort key for chat lists | + +### Indexes on `chats` + +| Index | Definition | Notes | +| --- | --- | --- | +| PK | `id` | | +| partial-unique | `legacy_object_id` WHERE NOT NULL | Idempotent backfill upsert | +| regular | `type` | | +| regular | `created_by` | | +| regular | `last_activity` | | + +> [!note] No FK to `users` +> `created_by` is stored as `text` (not `uuid` FK) to accommodate both Mongo ObjectIds and PG UUIDs during the transition period. + ## Virtuals | Virtual | Returns | Definition | | --- | --- | --- | | `participantsCount` | Count of active participants | `backend/src/models/Chat.ts:259` | -## Indexes +## Indexes (MongoDB) Defined at `backend/src/models/Chat.ts:243-247`: @@ -119,6 +175,24 @@ None defined. - **References**: [[User]] (`participants[].userId`, `messages[].senderId`, `messages[].reactions[].userId`, `lastMessage.senderId`, `unreadCounts[].userId`, `metadata.createdBy`). - **Referenced by**: [[Dispute]] (`chatId`), and any [[PurchaseRequest]] / [[SellerOffer]] indirectly through `relatedTo`. +## Migration Status + +| Dimension | Status | +| --- | --- | +| Dual-write repo | `DrizzleChatRepo` — active | +| Writes | Both MongoDB and Postgres receive writes | +| Reads | **MongoDB only** — not yet cut over | +| Postgres schema style | JSONB shim (participants + messages as blobs) | +| Normalisation blocker | Chat message threading design not finalised — blocks PG read cutover | + +The normalisation work required before reads can be cut to PG: +1. Design a `chat_messages` table with proper threading/reply support (currently `replyTo` is an ObjectId embedded in a JSONB blob) +2. Design a `chat_participants` table (currently a JSONB blob with soft-removal semantics) +3. Migrate reactions, edit history, and read tracking to relational rows +4. Align unread counts with the new structure + +Until that work is complete, the Postgres `chats` table is treated as a write-ahead log / backup, not the source of truth for reads. + ## State Transitions No top-level status. Chat-level archival is a boolean flag (`settings.isArchived`): diff --git a/02 - Data Models/Data Model Overview.md b/02 - Data Models/Data Model Overview.md index bac8cd3..1cfb87b 100644 --- a/02 - Data Models/Data Model Overview.md +++ b/02 - Data Models/Data Model Overview.md @@ -1,47 +1,57 @@ --- title: Data Model Overview -tags: [data-model, mongoose, overview] +tags: [data-model, mongoose, postgres, drizzle, overview] aliases: [Models Index, Schema Overview] --- # Data Model Overview -This section documents every Mongoose model that backs the marketplace. On backend `integrate-main-into-development@cab0719`, these Mongoose models are still the live application persistence layer. The repo also contains a Drizzle/Postgres migration layer, but most services still call `backend/src/models/*` directly. +This section documents every Mongoose model that backs the marketplace and the parallel Drizzle/Postgres schema that is progressively replacing it. On backend `integrate-main-into-development@cab0719`, Mongoose models are still the live read path for most domains. The Drizzle layer has 17 applied migrations (0000–0017) and active dual-write repos for the majority of tables. > [!note] Scope > Twenty-two models are present in `backend/src/models/`. The "File" concept exists only at the service layer (`backend/src/services/file/`) and is not persisted as its own Mongoose collection, so it is not listed below. > > [!note] Documentation freshness -> The 2026-05-24 audit note that marked `Dispute`, `BlogPost`, `Review`, `PointTransaction`, `LevelConfig`, and `ShopSettings` as missing is now stale: schema files exist for those models. Newer operational models such as [[ConfigSetting]], [[DerivedDestination]], [[FundsLedgerEntry]], and [[TrezorAccount]] should be expanded into dedicated model pages when the docs are next deepened. +> As of 2026-06-03 the Postgres migration inventory reflects migrations 0000–0017. The dual-write summary table at the bottom of this page is the authoritative migration-status reference. Individual model pages should be updated to note their PG table name and dual-write repo when they are deepened. > [!warning] Mongo vs Postgres runtime status -> Postgres schemas and repositories exist for the money/relational core, but normal app traffic is not fully cut over. Payment quote rows are the only current conditional PG write in checkout, and even that requires `ORACLE_QUOTING_ENABLED=true` plus a resolvable PG payment row. See [[Postgres Runtime Cutover Status]]. +> Dual-write repos exist for the majority of domain tables, but **reads are still served from Mongo** for all dual-write tables. Postgres is the sole store only for infra/bridge tables (`id_map`, `pg_dualwrite_gaps`), oracle quote rows (`payment_quotes`), and `config_setting_history`. Full read cutover is human-gated. See [[Postgres Runtime Cutover Status]]. ## Index of Models -- [[User]] — Core identity. Stores credentials, profile, preferences, referral data, points, and WebAuthn passkeys. Every other model that records "who did what" points back at a `User._id`. Buyers, sellers, and admins all live in this collection, differentiated by a `role` enum. -- [[PurchaseRequest]] — The buyer-side document at the heart of the marketplace. Captures what a buyer wants, the budget, urgency, delivery preferences, and the full lifecycle status (`pending_payment` → `seller_paid`). Aggregates [[SellerOffer]] references and tracks delivery codes. -- [[SellerOffer]] — A seller's bid against a [[PurchaseRequest]]. Holds price, delivery ETA, attachments, and a small status machine (`pending` / `accepted` / `rejected` / `withdrawn`). -- [[Payment]] — Records monetary movement intent and state: buyer pay-in, seller release, and refund. The current primary provider path is Request Network plus in-house checkout, derived destinations, funds ledger entries, and Transaction Safety Provider metadata. -- [[Chat]] — Conversation container with embedded messages, participants, unread counters, and reactions. Used for direct buyer-seller chats, group chats, and support tickets. Can be linked to a [[PurchaseRequest]] or [[SellerOffer]]. -- [[Notification]] — Per-user notification with category, type, and 90-day TTL for automatic cleanup. References any related entity by stringified id. -- [[RequestTemplate]] — A seller-authored, sharable template that pre-fills a [[PurchaseRequest]]. Carries a public shareable link, usage counter, and an optional default proposal. -- [[Dispute]] — Buyer-raised complaint tied to a [[PurchaseRequest]]. Captures evidence uploads, a timeline of admin actions, deadlines, and a structured resolution. -- [[BlogPost]] — Editorial content: title, slug, rich content, media, SEO metadata, view/like counters, and a draft/published/archived workflow. -- [[Address]] — User shipping address book entry. Enforces a single primary address per user via a pre-save hook. -- [[Category]] — Hierarchical product/service taxonomy referenced by [[PurchaseRequest]] and [[RequestTemplate]]. Supports parent/child via `parentId` and bilingual `name` / `nameEn`. -- [[Review]] — Polymorphic 1-5 star review against either a seller or a [[RequestTemplate]] (`subjectType` discriminator). One review per reviewer per subject (compound unique index). -- [[PointTransaction]] — Ledger of point grants and spends per user. Sources include purchase, referral, bonus, admin grant, and redemption. -- [[LevelConfig]] — Static configuration of loyalty tiers (level number, point thresholds, benefits, icon, color). Driven by admins; consumed by the [[User]].points.level field. -- [[ShopSettings]] — One-to-one storefront branding for a seller: name, description, avatar, cover image, review toggles, and social links. -- [[TempVerification]] — Short-lived signup record that holds candidate user data and a verification code. Auto-purges via TTL when `emailVerificationCodeExpires` passes. -- [[TelegramLink]] — Permanent auditable association between a Telegram user ID and an Amanat [[User]]. Stores Telegram profile metadata, link source (`miniapp` / `bot` / `login_widget`), status (`active` / `blocked`), and last-seen timestamp. One per Telegram user (unique on both `userId` and `telegramUserId`). -- [[TelegramSession]] — Short-lived Telegram Mini App session token issued when `initData` is verified. Carries the `initDataFingerprint` for replay protection and auto-expires via a MongoDB TTL index on `expiresAt`. -- [[ConfigSetting]] — Runtime configuration persisted in MongoDB for operational knobs that need an admin surface rather than a deploy. -- [[DerivedDestination]] — Per-payment derived wallet destination records used to reduce address reuse and reconcile on-chain pay-ins. -- [[FundsLedgerEntry]] — Immutable accounting ledger rows for pay-in, hold, release, refund, fee, adjustment, and reversal events. -- [[TrezorAccount]] — Hardware-wallet/safekeeping account metadata for custody operations and staged signer hardening. -- [[ConfigSettingHistory]] — Immutable audit trail of numeric runtime-config changes. Currently used for per-chain confirmation threshold change events, keyed as `confirmation_threshold:`. Added in commit `27fb15a`. +### Mongo Models (still live read path) + +- [[User]] — Core identity. Stores credentials, profile, preferences, referral data, points, and WebAuthn passkeys. Every other model that records "who did what" points back at a `User._id`. Buyers, sellers, admins, resolvers, and guards all live in this collection, differentiated by a `role` enum. PG table: `users` (dual-write active). +- [[PurchaseRequest]] — The buyer-side document at the heart of the marketplace. Captures what a buyer wants, the budget, urgency, delivery preferences, and the full lifecycle status (`pending_payment` → `seller_paid`). Aggregates [[SellerOffer]] references and tracks delivery codes. PG table: `purchase_requests` + 6 child tables (dual-write active). +- [[SellerOffer]] — A seller's bid against a [[PurchaseRequest]]. Holds price, delivery ETA, attachments, and a small status machine (`pending` / `accepted` / `rejected` / `withdrawn`). PG table: `seller_offers` (dual-write active). +- [[Payment]] — Records monetary movement intent and state: buyer pay-in, seller release, and refund. The current primary provider path is Request Network plus in-house checkout, derived destinations, funds ledger entries, and Transaction Safety Provider metadata. PG table: `payments` (dual-write active). +- [[Chat]] — Conversation container with embedded messages, participants, unread counters, and reactions. Used for direct buyer-seller chats, group chats, and support tickets. Can be linked to a [[PurchaseRequest]] or [[SellerOffer]]. PG table: `chats` (conservative JSONB shim; Chat normalization is an open blocker). +- [[Notification]] — Per-user notification with category, type, and 90-day TTL for automatic cleanup. References any related entity by stringified id. PG table: `notifications` (dual-write active; `user_id` stored as `text`, no hard FK). +- [[RequestTemplate]] — A seller-authored, sharable template that pre-fills a [[PurchaseRequest]]. Carries a public shareable link, usage counter, and an optional default proposal. PG table: `request_templates` (dual-write active). +- [[Dispute]] — Buyer-raised complaint tied to a [[PurchaseRequest]]. Captures evidence uploads, a timeline of admin actions, deadlines, and a structured resolution. PG table: `disputes` (dual-write active; all IDs as `text` for ObjectId/UUID coexistence). +- [[BlogPost]] — Editorial content: title, slug, rich content, media, SEO metadata, view/like counters, and a draft/published/archived workflow. PG table: `blog_posts` (dual-write active). +- [[Address]] — User shipping address book entry. Enforces a single primary address per user via a pre-save hook. PG table: `addresses` (schema scaffolded, migration 0016; `addressStore.ts` reads PG directly). +- [[Category]] — Hierarchical product/service taxonomy referenced by [[PurchaseRequest]] and [[RequestTemplate]]. Supports parent/child via `parentId` and bilingual `name` / `nameEn`. PG table: `categories` (dual-write active). +- [[Review]] — Polymorphic 1-5 star review against either a seller or a [[RequestTemplate]] (`subjectType` discriminator). One review per reviewer per subject (compound unique index). PG table: `reviews` (schema scaffolded, no dual-write repo yet). +- [[PointTransaction]] — Ledger of point grants and spends per user. Sources include purchase, referral, bonus, admin grant, and redemption. PG table: `point_transactions` (dual-write active). +- [[LevelConfig]] — Static configuration of loyalty tiers (level number, point thresholds, benefits, icon, color). Driven by admins; consumed by the [[User]].points.level field. No PG table (read-only config; not yet migrated). +- [[ShopSettings]] — One-to-one storefront branding for a seller: name, description, avatar, cover image, review toggles, and social links. PG table: `shop_settings` (schema scaffolded, no dual-write repo yet). +- [[TempVerification]] — Short-lived signup record that holds candidate user data and a verification code. Auto-purges via TTL when `emailVerificationCodeExpires` passes. No PG table (TTL-only; not yet migrated). +- [[TelegramLink]] — Permanent auditable association between a Telegram user ID and an Amanat [[User]]. Stores Telegram profile metadata, link source (`miniapp` / `bot` / `login_widget`), status (`active` / `blocked`), and last-seen timestamp. One per Telegram user (unique on both `userId` and `telegramUserId`). PG table: `telegram_links` (schema scaffolded, no dual-write repo yet). +- [[TelegramSession]] — Short-lived Telegram Mini App session token issued when `initData` is verified. Carries the `initDataFingerprint` for replay protection and auto-expires via a MongoDB TTL index on `expiresAt`. PG table: `telegram_sessions` (schema scaffolded, no dual-write repo yet). +- [[ConfigSetting]] — Runtime configuration persisted in MongoDB for operational knobs that need an admin surface rather than a deploy. PG table: `config_settings` (schema scaffolded, no dual-write repo yet). +- [[DerivedDestination]] — Per-payment derived wallet destination records used to reduce address reuse and reconcile on-chain pay-ins. PG table: `derived_destinations` + `derived_destination_sweeps` (dual-write active). +- [[FundsLedgerEntry]] — Immutable accounting ledger rows for pay-in, hold, release, refund, fee, adjustment, and reversal events. PG table: `funds_ledger_entries` (dual-write active; immutability enforced by DB trigger since migration 0015). +- [[TrezorAccount]] — Hardware-wallet/safekeeping account metadata for custody operations and staged signer hardening. PG table: `trezor_accounts` + `trezor_derived_addresses` (dual-write active). +- [[ConfigSettingHistory]] — Immutable audit trail of numeric runtime-config changes. Currently used for per-chain confirmation threshold change events, keyed as `confirmation_threshold:`. Added in commit `27fb15a`. PG table: `config_setting_history` (PG-only; no Mongo equivalent). + +### PG-Only Tables (no Mongo equivalent) + +- `id_map` — ObjectId → UUID bridge. Every migrated table upserts here during backfill/dual-write. Composite PK on `(collection, legacy_object_id)`, unique on `new_id`. +- `pg_dualwrite_gaps` — Append-only reconciliation gap log for failed PG dual-writes. Tracks collection, op, payload, severity, and resolution metadata. +- `payment_quotes` — Oracle pricing quotes per payment (oracle depeg-protection feature). Stores `fx_rate`, `token_price_usd`, `depeg_adjustment_bps`, `settle_amount`, chain/token, and expiry. Requires `ORACLE_QUOTING_ENABLED=true`. 1:1 to `payments`. +- `user_passkeys` — WebAuthn credential store (child of `users`). Columns: credential id (text PK), `user_id FK→users CASCADE`, `public_key`, `counter`, `device_type`, `device_name`. +- `user_refresh_tokens` — Refresh token store (child of `users`). Columns: `token text PK`, `user_id FK→users CASCADE`. ## Relationship Diagram @@ -59,6 +69,10 @@ erDiagram USER ||--o{ DISPUTE : "raises as buyer" USER ||--o{ USER : "referred by" USER ||--o{ TREZOR_ACCOUNT : "controls custody account" + USER ||--o{ USER_PASSKEY : "authenticates with" + USER ||--o{ USER_REFRESH_TOKEN : "sessions via" + USER ||--o| TELEGRAM_LINK : "links identity" + USER ||--o{ TELEGRAM_SESSION : "session for" PURCHASE_REQUEST }o--|| CATEGORY : "belongs to" PURCHASE_REQUEST ||--o{ SELLER_OFFER : "receives" @@ -74,6 +88,7 @@ erDiagram PAYMENT }o--|| USER : "seller" PAYMENT ||--o{ FUNDS_LEDGER_ENTRY : "accounted by" PAYMENT ||--o| DERIVED_DESTINATION : "collects into" + PAYMENT ||--o| PAYMENT_QUOTE : "oracle-priced by" CHAT }o--o{ USER : "participants" CHAT ||--o{ DISPUTE : "support channel" @@ -91,10 +106,17 @@ erDiagram TELEGRAM_LINK }o--|| USER : "links identity" TELEGRAM_SESSION }o--o| USER : "session for" TELEGRAM_SESSION }o--|| TELEGRAM_LINK : "matches" + + TREZOR_ACCOUNT ||--o{ TREZOR_DERIVED_ADDRESS : "issues" + DERIVED_DESTINATION ||--o{ DERIVED_DESTINATION_SWEEP : "swept by" + + ID_MAP ||..|| USER : "bridges ObjectId" ``` ## Conventions Across All Models +### Mongoose Conventions + > [!note] Shared schema patterns > - **Timestamps**: every model declares `{ timestamps: true }`, so `createdAt` and `updatedAt` are always present. > - **ObjectId references**: foreign keys use `Schema.Types.ObjectId` with an explicit `ref` (e.g. `ref: 'User'`). The two exceptions are [[Notification]] and [[Payment]] which use string-typed or `Mixed` identifiers in places to support template-flow payments. @@ -105,15 +127,126 @@ erDiagram > [!warning] Index discipline > Several schemas leave a comment noting that `unique: true` already creates an index — adding `schema.index({ field: 1 })` on top would produce a duplicate-index warning at startup. When introducing new indexes, search for `unique: true` first. +### Drizzle/Postgres Conventions + +> [!note] PG schema patterns +> - **Legacy bridge**: every migrated table carries `legacy_object_id text` with a partial-unique index `WHERE legacy_object_id IS NOT NULL` for idempotent backfill upserts. The `id_map` table records the ObjectId → UUID mapping centrally. +> - **Money columns**: `numeric(38,18)` for fiat/crypto amounts throughout, except `seller_offers` which uses `numeric(18,8)` per the Migration Guide. Blockchain balance columns use `numeric(78,0)` to hold uint256 without overflow. +> - **Polymorphic triples**: the `ref_kind` enum (`entity` | `template`) discriminator is expanded into three columns (`_ref_kind`, `_id`, `_external_ref`) with a CHECK constraint to enforce discriminator integrity. Used by `payments`, `funds_ledger_entries`, and `derived_destinations`. +> - **Soft delete**: `addresses` uses `deleted_at timestamptz` (nullable) with partial-unique indexes scoped to `WHERE deleted_at IS NULL`. Most other tables retain the Mongo `status` flag approach. +> - **Timestamps**: all timestamp columns declare `withTimezone: true`. +> - **Immutability**: `funds_ledger_entries` has both an UPDATE-blocking and a DELETE-blocking trigger installed at the DB level (migrations 0004, 0015). A TRUNCATE trigger was added in migration 0013. +> - **user_role enum**: values are `admin`, `buyer`, `seller`, `resolver`, `guard`. The `guard` value was added in migration 0017. + +## Postgres Migration Inventory + +Schema entry point: `backend/src/db/schema/index.ts` + +| Migration | File | Summary | +|---|---|---| +| 0000 | `0000_slimy_veda.sql` | Initial: core enums + `id_map` + `categories` | +| 0001 | `0001_wild_cargill.sql` | `trezor_accounts` + `trezor_derived_addresses` (later reset) | +| 0002 | `0002_motionless_grey_gargoyle.sql` | Schema reset: drops 0000/0001 tables to be rebuilt in 0003; adds `categories.parent_id` self-FK | +| 0003 | `0003_remarkable_retro_girl.sql` | Comprehensive rebuild: all enums + full core domain (`users`, `payments`, `funds_ledger_entries`, `derived_destinations`, `purchase_requests` + 6 children, `seller_offers`, `point_transactions`, `trezor_*`) | +| 0004a | `0004_funds_ledger_entries.sql` | UPDATE-blocking immutability trigger on `funds_ledger_entries` | +| 0004b | `0004_seller_offer.sql` | Physical FKs on `seller_offers` → `users` and `purchase_requests` (CASCADE) | +| 0005 | `0005_simple_champions.sql` | `pg_dualwrite_gaps`; FKs on `payments`; `legacy_object_id` unique indexes; refined pending-RN payment unique index | +| 0006 | `0006_normal_madame_hydra.sql` | CHECK: `purchase_requests.budget_currency` restricted to crypto (USDT, USDC) | +| 0007 | `0007_woozy_shaman.sql` | Drops 0006 constraint; sets `budget_currency` default to `'USDT'` | +| 0008 | `0008_giant_winter_soldier.sql` | Adds `'TRY'` to `offer_currency` enum; creates `payment_quotes` table | +| 0009 | `0009_unique_active_categories.sql` | Category deduplication; partial unique index on normalized active category name | +| 0010 | `0010_request_templates.sql` | Creates `request_templates`; deduplicates `purchase_request_specifications`; adds unique key constraint | +| 0011 | `0011_chats.sql` | Creates `chats` with JSONB participant/message storage + chat-related enums | +| 0012 | `0012_disputes.sql` | Creates `disputes` (text IDs, JSONB evidence/timeline/resolution) | +| 0013 | `0013_money_constraints.sql` | Money-integrity CHECKs on `payments`, `payment_quotes`, `point_transactions`, `users`; TRUNCATE trigger on `funds_ledger_entries`; composite PK + unique on `id_map` | +| 0014 | `0014_physical_fks.sql` | NOT VALID FKs across all major tables (validated immediately); composite indexes on `payments`, `purchase_requests`, `seller_offers` | +| 0015 | `0015_funds_ledger_immutable_trigger.sql` | Replaces/extends ledger triggers: UPDATE-block + new DELETE-block on `funds_ledger_entries` | +| 0016 | `0016_addresses_table.sql` | `address_type` enum + `addresses` table; partial-unique primary-address-per-user index | +| 0017 | `0017_user_role_guard.sql` | Adds `'guard'` to `user_role` enum (idempotent `ADD VALUE IF NOT EXISTS`) | + +## Drizzle Table Inventory and Migration Status + +### Infrastructure / Bridge + +| PG Table | Schema File | Status | Notes | +|---|---|---|---| +| `id_map` | `idMap.ts` | PG-only | ObjectId → UUID bridge; composite PK + unique on `new_id` | +| `pg_dualwrite_gaps` | `pgDualwriteGaps.ts` | PG-only | Append-only reconciliation gap log for failed dual-writes | + +### Core Domain + +| PG Table | Schema File | Status | Dual-Write Repo | +|---|---|---|---| +| `users` | `users.ts` | Dual-write active | `DualWriteUserRepo` + `DrizzleUserRepo` + `MongoUserRepo` | +| `user_passkeys` | `users.ts` | Dual-write active (child of users) | — | +| `user_refresh_tokens` | `users.ts` | Dual-write active (child of users) | — | +| `categories` | `category.ts` | Dual-write active | `DualWriteMarketplaceRepo` | +| `purchase_requests` | `purchaseRequest.ts` | Dual-write active | `DualWriteMarketplaceRepo` | +| `purchase_request_delivery_info` | `purchaseRequest.ts` | Dual-write active (1:1 child) | — | +| `purchase_request_delivery_address` | `purchaseRequest.ts` | Dual-write active (1:1 child) | — | +| `purchase_request_seller_delivery_info` | `purchaseRequest.ts` | Dual-write active (1:1 child) | — | +| `delivery_attempts` | `purchaseRequest.ts` | Dual-write active (1:N child) | — | +| `purchase_request_service_info` | `purchaseRequest.ts` | Dual-write active (1:1 child) | — | +| `purchase_request_specifications` | `purchaseRequest.ts` | Dual-write active (1:N child) | — | +| `purchase_request_preferred_sellers` | `purchaseRequest.ts` | Dual-write active (N:M junction) | — | +| `seller_offers` | `sellerOffer.ts` | Dual-write active | `DualWriteMarketplaceRepo` | +| `payments` | `payment.ts` | Dual-write active | `DualWritePaymentRepo` + `DrizzlePaymentRepo` + `MongoPaymentRepo` | +| `payment_quotes` | `paymentQuote.ts` | PG-only | No Mongo equivalent; oracle depeg-protection feature | +| `funds_ledger_entries` | `fundsLedgerEntry.ts` | Dual-write active | `DrizzlePaymentRepo` / `DualWritePaymentRepo` | +| `derived_destinations` | `derivedDestination.ts` | Dual-write active | `DualWriteDerivedDestinationRepo` + `DrizzleDerivedDestinationRepo` | +| `derived_destination_sweeps` | `derivedDestination.ts` | Dual-write active (append-only child) | — | +| `trezor_accounts` | `trezorAccount.ts` | Dual-write active | `DualWriteTrezorAccountRepo` + `DrizzleTrezorAccountRepo` | +| `trezor_derived_addresses` | `trezorAccount.ts` | Dual-write active (child of trezor_accounts) | — | +| `point_transactions` | `pointTransaction.ts` | Dual-write active | `DualWritePointsRepo` + `DrizzlePointsRepo` | +| `request_templates` | `requestTemplate.ts` | Dual-write active | `DualWriteMarketplaceRepo` | +| `chats` | `chat.ts` | Dual-write active | `DrizzleChatRepo` | +| `blog_posts` | `blogPost.ts` | Dual-write active | `DualWriteBlogRepo` + `DrizzleBlogRepo` | +| `notifications` | `notification.ts` | Dual-write active | `DualWriteNotificationRepo` + `DrizzleNotificationRepo` | +| `disputes` | `dispute.ts` | Dual-write active | `DualWriteDisputeRepo` + `DrizzleDisputeRepo` | +| `addresses` | `address.ts` | Schema scaffolded | No dual-write repo; `addressStore.ts` reads PG directly (migration 0016) | +| `shop_settings` | `shopSettings.ts` | Schema scaffolded | No dual-write repo | +| `config_settings` | `configSetting.ts` | Schema scaffolded | No dual-write repo | +| `config_setting_history` | `configSetting.ts` | PG-only | No Mongo equivalent; child of `config_settings` | +| `telegram_links` | `telegramLink.ts` | Schema scaffolded | No dual-write repo | +| `telegram_sessions` | `telegramSession.ts` | Schema scaffolded | No dual-write repo | +| `reviews` | `review.ts` | Schema scaffolded | No dual-write repo | + +> [!note] Read cutover status +> **Dual-write active** means writes go to both Mongo and PG; reads still come from Mongo (per MEMORY.md as of 2026-06-03). **Schema scaffolded** means the Drizzle table exists but no DualWriteRepo plumbs it. **PG-only** means there is no Mongo model for that data. + +## Shared Enum Reference + +Enums live in `backend/src/db/schema/_enums.ts` (shared) and individual schema files. Key enums: + +| Enum | Values | +|---|---| +| `user_role` | admin, buyer, seller, resolver, guard | +| `auth_provider` | email, google, telegram | +| `user_status` | active, suspended, deleted | +| `purchase_request_status` | pending_payment, pending, received_offers, in_negotiation, payment_pending, payment_confirmed, in_progress, delivery, delivered, completed, disputed, refunded, seller_paid | +| `offer_status` | pending, accepted, rejected, withdrawn, active | +| `offer_currency` | USD, EUR, IRR, USDT, USDC, TRY | +| `payment_provider` | request.network, amn.scanner, shkeeper, other | +| `payment_status` | pending, processing, confirmed, completed, failed, cancelled, refunded | +| `escrow_state` | funded, releasable, released, refunded, releasing, failed, cancelled, partial | +| `funds_ledger_entry_type` | payment_detected, provider_fee, platform_fee, hold, release, refund, dispute_hold, adjustment | +| `derived_destination_status` | active, swept, sweeping, quarantined | +| `ref_kind` | entity, template | +| `chat_type` | direct, group, support | +| `review_subject_kind` | seller, template | +| `address_type` | Home, Office, Other | +| `telegram_link_source` | miniapp, bot, login_widget | +| `telegram_link_status` | active, blocked | + ## Lifecycle View The dominant happy-path flow exercises five collections in order: 1. A buyer (`User`) creates a `PurchaseRequest` with `status: 'pending'`. 2. Sellers (other `User`s) attach `SellerOffer` documents; the request transitions through `received_offers` → `in_negotiation` as the parties chat in a `Chat`. -3. The buyer accepts an offer; a `Payment` is opened against the Request Network provider and, once verified by webhook/reconciliation and safety checks, advances to a funded escrow state. +3. The buyer accepts an offer; a `Payment` is opened against the Request Network provider and, once verified by webhook/reconciliation and safety checks, advances to a funded escrow state. If `ORACLE_QUOTING_ENABLED=true`, a `payment_quote` row is written to PG at this point. 4. The seller marks the request `delivery` → `delivered`; the buyer confirms with the 6-digit `deliveryCode` and the request becomes `completed`. -5. The escrow `Payment` flips to `released` after a ledger-gated custody transfer instruction. Optionally the buyer writes a `Review` and earns a `PointTransaction`. +5. The escrow `Payment` flips to `released` after a ledger-gated custody transfer instruction. Each ledger event appends an immutable `FundsLedgerEntry` row (Mongo + PG). Optionally the buyer writes a `Review` and earns a `PointTransaction`. If anything goes sideways, the buyer can open a `Dispute`, which freezes release until an admin resolves it (refund, replacement, compensation, or no-action). @@ -122,4 +255,4 @@ If anything goes sideways, the buyer can open a `Dispute`, which freezes release Each model has its own note in this folder. Cross-references use `[[wikilinks]]` so backlinks work in Obsidian's graph view. Schemas are documented at field-level granularity — every field is listed with its type, default, validation, and indexing decisions. Where a model carries a meaningful state machine, a Mermaid `stateDiagram-v2` accompanies the schema table. > [!note] Source of truth -> The information below is mirrored from the TypeScript schema definitions. If a field listed here disagrees with the code, the code wins — please update the note. All source citations use the form `backend/src/models/.ts:`. +> The information below is mirrored from the TypeScript schema definitions. If a field listed here disagrees with the code, the code wins — please update the note. All source citations use the form `backend/src/models/.ts:` for Mongo and `backend/src/db/schema/.ts:` for Drizzle/PG. diff --git a/02 - Data Models/Dispute.md b/02 - Data Models/Dispute.md index a7ef223..2db61ab 100644 --- a/02 - Data Models/Dispute.md +++ b/02 - Data Models/Dispute.md @@ -1,23 +1,27 @@ --- title: Dispute -tags: [data-model, mongoose] +tags: [data-model, mongoose, postgres] aliases: [Complaint, IDispute] --- # Dispute -> **Last updated:** 2026-05-29 — aligned with code (see [Doc vs Code Audit Report](../09%20-%20Audits/Doc%20vs%20Code%20Audit%20Report%20-%202026-05-29.md)) +> **Last updated:** 2026-06-03 — added Postgres / Drizzle schema and migration status (see [Doc vs Code Audit Report](../09%20-%20Audits/Doc%20vs%20Code%20Audit%20Report%20-%202026-05-29.md)) Buyer-raised complaint tied to a [[PurchaseRequest]]. Captures the reason, priority, category, an array of evidence uploads, a chronological `timeline` of actions, an optional resolution, and SLA deadlines. An admin (`adminId`) is assigned during triage and resolves the dispute with a structured action (`refund`, `replacement`, `compensation`, `warning_seller`, `ban_seller`, or `no_action`). > [!note] Implementation status > `backend/src/models/Dispute.ts`, `backend/src/services/dispute/DisputeService.ts`, `backend/src/routes/disputeRoutes.ts`, and release-hold helper routes now exist. The remaining gap is canonical state alignment between the full dispute document and the lighter `PurchaseRequest`/`Payment` hold flags used by release gating. > -> Source: `backend/src/models/Dispute.ts` — schema definition and model export. +> Sources: `backend/src/models/Dispute.ts` (Mongoose schema), `backend/src/db/schema/dispute.ts` (Drizzle/Postgres schema). -> ⚠️ **SECURITY** — The dispute `status` update endpoint and the `resolve` endpoint currently have **no role guards**. Any authenticated user (not just admins) can modify dispute status or submit a resolution. This is a known gap pending a role-guard audit. +> WARNING — The dispute `status` update endpoint and the `resolve` endpoint currently have **no role guards**. Any authenticated user (not just admins) can modify dispute status or submit a resolution. This is a known gap pending a role-guard audit. -## Schema +## Migration Status + +**DUAL-WRITE** — `DualWriteDisputeRepo` + `DrizzleDisputeRepo` + `MongoDisputeRepo`. Writes go to both Mongo and Postgres. Reads still come from Mongo (cutover not yet executed). + +## Mongo Schema | Field | Type | Required | Default | Validation | Index | Description | | --- | --- | --- | --- | --- | --- | --- | @@ -57,26 +61,80 @@ Buyer-raised complaint tied to a [[PurchaseRequest]]. Captures the reason, prior Valid values: `product_quality` · `delivery_delay` · `wrong_item` · `payment_issue` · `seller_behavior` · `other` -**Note:** `fraud` is **not** a valid category value. Use `seller_behavior` or `other` for fraud-related complaints. +Note: `fraud` is NOT a valid category value. Use `seller_behavior` or `other` for fraud-related complaints. ### Status enum Valid values: `pending` · `in_progress` · `waiting_response` · `resolved` · `rejected` · `closed` -**Note:** `under_review` does **not** exist in the schema. The equivalent lifecycle state is `in_progress`. +Note: `under_review` does NOT exist in the schema. The equivalent lifecycle state is `in_progress`. ### Resolution action enum Valid values: `refund` · `replacement` · `compensation` · `warning_seller` · `ban_seller` · `no_action` -> [!note] `messages` in the interface -> The TypeScript interface mentions an optional embedded `messages[]` array, but the actual Mongoose schema does not declare it — messages live in [[Chat]] via `chatId`. +Note: The TypeScript interface mentions an optional embedded `messages[]` array, but the actual Mongoose schema does not declare it — messages live in [[Chat]] via `chatId`. + +## Postgres / Drizzle Schema + +Source: `backend/src/db/schema/dispute.ts` — migration 0012. + +### `disputes` table + +| Column | PG Type | Notes | +| --- | --- | --- | +| `id` | `uuid` PK | Generated UUID primary key. | +| `legacy_object_id` | `text` | Mongo ObjectId bridge; partial-unique WHERE NOT NULL. | +| `purchase_request_id` | `text` | Stored as text (not uuid FK) to accommodate Mongo ObjectIds and PG UUIDs during cutover. No hard FK. | +| `buyer_id` | `text` | Same cutover reason — text, no hard FK. | +| `seller_id` | `text` | Optional; text, no hard FK. | +| `admin_id` | `text` | Optional; text, no hard FK. | +| `reason` | `text` | Short reason. | +| `description` | `text` | Detailed description. | +| `priority` | `text` | No DB-level enum; app-layer validated. | +| `category` | `text` | No DB-level enum; app-layer validated. | +| `status` | `text` | No DB-level enum; app-layer validated. | +| `evidence` | `jsonb` | Array of evidence objects (serialized). | +| `chat_id` | `text` | Optional; text reference to Chat. | +| `messages` | `jsonb` | Embedded messages blob (conservative shim; normalization pending). | +| `timeline` | `jsonb` | Array of timeline action objects. | +| `resolution` | `jsonb` | Resolution object when resolved. | +| `deadline` | `timestamptz` | Overall SLA deadline. | +| `response_deadline` | `timestamptz` | Response SLA. | +| `tags` | `jsonb` | Array of tag strings. | +| `created_at` | `timestamptz` | Auto-managed. | +| `updated_at` | `timestamptz` | Auto-managed. | +| `closed_at` | `timestamptz` | Set when status reaches `closed`. | + +> [!note] ID columns as `text` +> `purchase_request_id`, `buyer_id`, `seller_id`, and `admin_id` are all stored as `text` (not `uuid` with a FK) to accommodate both legacy Mongo ObjectIds and PG UUIDs transparently during the cutover window. No referential integrity constraints exist at the DB layer for these columns. + +> [!note] `messages` jsonb column +> The Postgres schema includes a `messages jsonb` column that is absent from the Mongo schema (where messages live in Chat via `chatId`). This is a conservative shim added during migration scaffolding. Full normalization of chat/messages is flagged as an open blocker. + +### Postgres Indexes + +| Index | Type | Notes | +| --- | --- | --- | +| `(legacy_object_id)` WHERE NOT NULL | partial-unique | Idempotent backfill upserts. | +| `(purchase_request_id)` | regular | Lookup by request. | +| `(buyer_id)` | regular | Buyer's disputes. | +| `(seller_id)` | regular | Seller's disputes. | +| `(admin_id)` | regular | Admin workload. | +| `(status)` | regular | Lifecycle filtering. | +| `(priority)` | regular | Priority filtering. | +| `(category)` | regular | Category filtering. | +| `(created_at)` | regular | Time-ordered listing. | +| `(status, priority)` | compound | Admin queue sort. | +| `(admin_id, status)` | compound | Per-admin workload view. | + +Mirrors the Mongo index set exactly. ## Virtuals None defined. -## Indexes +## Mongo Indexes Defined at `backend/src/models/Dispute.ts`: @@ -128,21 +186,35 @@ stateDiagram-v2 ## Common Queries ```ts -// Admin queue +// Admin queue (Mongo) Dispute.find({ status: { $in: ['pending', 'in_progress', 'waiting_response'] } }) .sort({ priority: -1, createdAt: 1 }); -// Buyer's disputes +// Buyer's disputes (Mongo) Dispute.find({ buyerId }).sort({ createdAt: -1 }); -// Seller's open disputes +// Seller's open disputes (Mongo) Dispute.find({ sellerId, status: { $nin: ['resolved', 'rejected', 'closed'] } }); -// Append timeline entry atomically +// Append timeline entry atomically (Mongo) Dispute.updateOne( { _id }, { $push: { timeline: { action, performedBy: adminId, performedAt: new Date(), details } } } ); ``` +```sql +-- Admin queue (Postgres) +SELECT * FROM disputes +WHERE status IN ('pending', 'in_progress', 'waiting_response') +ORDER BY priority DESC, created_at ASC; + +-- Buyer's disputes (Postgres) +SELECT * FROM disputes WHERE buyer_id = $1 ORDER BY created_at DESC; + +-- Seller's open disputes (Postgres) +SELECT * FROM disputes +WHERE seller_id = $1 AND status NOT IN ('resolved', 'rejected', 'closed'); +``` + Related: [[PurchaseRequest]], [[User]], [[Chat]], [[Payment]]. diff --git a/02 - Data Models/Postgres Runtime Cutover Status.md b/02 - Data Models/Postgres Runtime Cutover Status.md index 85a3ac0..5148c7a 100644 --- a/02 - Data Models/Postgres Runtime Cutover Status.md +++ b/02 - Data Models/Postgres Runtime Cutover Status.md @@ -3,56 +3,125 @@ title: Postgres Runtime Cutover Status tags: [data-model, postgres, migration, runtime-status] aliases: [Postgres Status, PG Cutover Status, Mongo vs Postgres Runtime] created: 2026-05-31 -source: backend integrate-main-into-development@cf59726 + frontend integrate-main-into-development@a2b972b + deployment main@8764fdf +updated: 2026-06-03 +source: backend integrate-main-into-development@14d164c + deployment main@8764fdf --- # Postgres Runtime Cutover Status -> **Current branch:** backend `integrate-main-into-development` at `cf59726`, version `2.8.37`; frontend `integrate-main-into-development` at `a2b972b`, version `2.8.37`; dev deployment `main` at `8764fdf`. +> **Current branch:** backend `integrate-main-into-development` at `14d164c`, version `2.8.56`; dev deployment `main` at `8764fdf`. > -> **Bottom line:** this branch is **Postgres-capable**, not fully Postgres-backed. Dev deployment now defaults eight existing PG-capable runtime stores to Postgres: auth-owned users/Telegram auth, confirmation-threshold config/history, user addresses, categories, level config, shop settings, reviews, and notifications. Code-level defaults remain Mongo outside that deployment override, and Mongo remains the compatibility store for still-Mongo domains. The category PG path enforces one active visible category per normalized name. As of backend `2.8.37`, the active startup/health/admin/report import surface no longer has non-type top-level `mongoose` or `models/*` imports, repository factory flags accept `postgres` as an alias for `pg`, and the unmounted legacy marketplace router is no longer re-exported from the marketplace service index; legacy Mongo models are lazy-loaded only when fallback/backfill/maintenance actions run. All PG-backed stores require `PG_URL`. +> **Bottom line:** the codebase is in **active dual-write phase**. All 11 repository domains in the factory now have Drizzle schemas, Drizzle repos, and dual-write wrappers (except Chat and ReleaseHold, which have Drizzle repos but no dual-write counterpart). 18 Drizzle migrations (0000–0017) have landed, covering every table in scope. Dev deployment defaults nine PG-capable stores to Postgres: auth-owned users/Telegram auth, confirmation-threshold config/history, user addresses, categories, level config, shop settings, reviews, notifications, and the oracle payment_quotes path. Code-level defaults remain `mongo` outside those deployment overrides. Repository factory normalizes `postgres` and `pg` as equivalent mode tokens. The unmounted legacy marketplace router is detached. As of `2.8.54–2.8.56`, the `guard` user role is in PG schema, chat routes are fixed, notifications deliver in real time, and PG response serialization/id resolution in marketplace is corrected. Reads are still Mongo-authoritative across all dual-write domains — read cutover is the remaining gate for each domain. Chat normalization (participants/messages stored as JSONB blobs, not relational child tables) remains an open blocker for full Chat cutover. + +## Schema and Repository Coverage + +### Tables with Full Drizzle Schema + +All tables below have a `.ts` schema file in `src/db/schema/` and are covered by at least one migration: + +**Infrastructure:** `id_map`, `pg_dualwrite_gaps` + +**Auth/Users:** `users`, `user_passkeys`, `user_refresh_tokens`, `telegram_links`, `telegram_sessions` + +**Marketplace:** `categories`, `purchase_requests`, `purchase_request_delivery_info`, `purchase_request_delivery_address`, `purchase_request_seller_delivery_info`, `purchase_request_service_info`, `purchase_request_specifications`, `purchase_request_preferred_sellers`, `delivery_attempts`, `seller_offers`, `request_templates` + +**Payments:** `payments`, `payment_quotes`, `funds_ledger_entries`, `derived_destinations`, `derived_destination_sweeps` + +**Points/Wallet:** `point_transactions`, `trezor_accounts`, `trezor_derived_addresses` + +**Config/Ops:** `config_settings`, `config_setting_history`, `shop_settings`, `addresses`, `reviews` + +**Content/Social:** `blog_posts`, `notifications`, `disputes`, `chats` + +Total: **32 tables** across 18 migrations (0000–0017). + +### Tables with a Drizzle Repository + +| Drizzle Repo | Dual-Write Repo | Domain | +|---|---|---| +| `DrizzleUserRepo` | `DualWriteUserRepo` | Users, passkeys, refresh tokens | +| `DrizzlePaymentRepo` | `DualWritePaymentRepo` | Payments, funds ledger | +| `DrizzleMarketplaceRepo` | `DualWriteMarketplaceRepo` | Categories, purchase requests, seller offers, request templates | +| `DrizzleDerivedDestinationRepo` | `DualWriteDerivedDestinationRepo` | Derived destinations, sweeps | +| `DrizzleTrezorAccountRepo` | `DualWriteTrezorAccountRepo` | Trezor accounts, derived addresses | +| `DrizzlePointsRepo` | `DualWritePointsRepo` | Point transactions | +| `DrizzleNotificationRepo` | `DualWriteNotificationRepo` | Notifications | +| `DrizzleDisputeRepo` | `DualWriteDisputeRepo` | Disputes | +| `DrizzleBlogRepo` | `DualWriteBlogRepo` | Blog posts | +| `DrizzleChatRepo` | _(none — no dual-write wrapper)_ | Chats (JSONB shim; Chat normalization is a blocker) | +| `DrizzleReleaseHoldRepo` | _(none — no dual-write wrapper)_ | Release holds (bridges payments + purchase_requests) | + +Tables with schema but no dedicated Drizzle repo yet: `addresses` (handled via addressStore facade), `shop_settings` (handled via shopSettings facade), `config_settings` / `config_setting_history` (handled via config-store facade), `telegram_links` / `telegram_sessions` (handled via auth-store facade), `reviews` (handled via review-store facade). + +### Migration Count + +18 migrations landed: **0000 through 0017**. + +| Migration | Key change | +|---|---| +| 0000 | Core enums + `id_map` + `categories` | +| 0001 | `trezor_accounts` + `trezor_derived_addresses` | +| 0002 | Schema reset (drops 0000/0001 tables, adds category self-FK) | +| 0003 | Full rebuild: all core domain tables (users, payments, marketplace, funds ledger, derived destinations, points, trezor) | +| 0004 (×2) | Funds ledger immutability trigger; seller_offer physical FKs | +| 0005 | `pg_dualwrite_gaps`; payment FKs; legacy_object_id uniques; pending payment index fix | +| 0006 | budget_currency crypto-only CHECK on purchase_requests | +| 0007 | Drops 0006 constraint; sets USDT default | +| 0008 | `offer_currency` adds TRY; creates `payment_quotes` | +| 0009 | Active category deduplication; `categories_active_name_norm_uq` | +| 0010 | `request_templates`; purchase_request_specifications unique constraint | +| 0011 | `chats` + chat enums | +| 0012 | `disputes` | +| 0013 | Money-integrity CHECK constraints; ledger TRUNCATE guard; id_map composite PK | +| 0014 | Physical NOT VALID FKs across schema; validates all | +| 0015 | Ledger immutability extended: UPDATE + DELETE triggers | +| 0016 | `address_type` enum + `addresses` table | +| 0017 | `guard` value added to `user_role` enum | ## What Uses Postgres Now | Area | Runtime status | Notes | |---|---|---| -| Postgres connection | Available when `PG_URL` is set | Current store facades use `src/infrastructure/postgres/client.ts`; the broader `src/db/` Drizzle layer and repository factory exist, but most live services are not wired through that factory yet. | -| Runtime schema bootstrap | Implemented for auth, config, address, and reference stores | Auth tables are bootstrapped from `src/services/auth/postgresAuthSchema.ts`; store facades bootstrap their own tables at startup when their `*_STORE=postgres` flag is enabled. | -| Health observability | Implemented in `/api/health` | `checks.postgres` reports `configured`, `required`, `storeModes`, `enabledStores`, and `enabledStoreCount`, so Gatus/operators can verify both PG reachability and which runtime stores are actively PG-backed. Dev Gatus now asserts all eight dev PG-backed store modes are `postgres`, including notifications. Backend `2.8.33` lazy-loads Mongoose for the legacy Mongo health check and skips it when Mongo is optional under `MONGO_CONNECT_MODE=auto/never`. | -| Auth-owned user store | PG-backed in dev deployment; code opt-in with `AUTH_STORE=postgres` | Auth, passkey, Telegram auth/link/session/temp-verification, and `/api/user` profile paths use an auth-store facade. In PG mode, users are stored in Postgres and mirrored back to Mongo through `legacy_object_id` for compatibility with still-Mongo services. | -| Confirmation-threshold runtime config | PG-backed in dev deployment; code opt-in with `CONFIG_STORE=postgres` | `ConfigSetting` / `ConfigSettingHistory` access for `/api/admin/settings/confirmation-thresholds` and transaction-safety confirmation thresholds uses a config-store facade. PG-mode writes mirror back to Mongo for rollback. Backend `2.8.32` removed top-level Mongo model imports from this facade; legacy models load only for Mongo fallback/backfill/mirror paths. | -| User addresses | PG-backed in dev deployment; code opt-in with `ADDRESS_STORE=postgres` | `/api/addresses` CRUD uses an address-store facade. PG mode enforces one primary address per user with a partial unique index and mirrors writes/deletes back to Mongo for rollback. | -| Marketplace categories | PG-backed in dev deployment; code opt-in with `CATEGORY_STORE=postgres` | `CategoryService` and the default `General` category path use a category-store facade. PG-mode writes mirror back to Mongo for rollback and still-Mongo request/template references. PG schema bootstrap/migration deactivates duplicate active category labels, repoints existing category references to the kept row, and enforces `categories_active_name_norm_uq` on `lower(btrim(name)) WHERE is_active = true`. List/cache reads also dedupe by normalized name. | -| Level configuration | PG-backed in dev deployment; code opt-in with `LEVEL_CONFIG_STORE=postgres` | `PointsService` level reads use a level-config facade. `PointTransaction` and user points remain Mongo-backed. | -| Shop settings | PG-backed in dev deployment; code opt-in with `SHOP_SETTINGS_STORE=postgres` | Shop settings controller, seller payment rail resolution, and review enable/disable checks use a shop-settings facade. PG-mode writes mirror back to Mongo. Backend `2.8.32` removed top-level Mongo model imports from this facade; legacy models load only for Mongo fallback/backfill/mirror paths. | -| Marketplace reviews | PG-backed in dev deployment; code opt-in with `REVIEW_STORE=postgres` | Review list/summary/create routes use a review-store facade. PG-mode list responses still hydrate `reviewerId` from the user mirror to preserve frontend shape. Backend `2.8.32` removed top-level Mongo model imports from this facade; legacy models load only for Mongo fallback/backfill/mirror paths. | -| Notifications | PG-backed in dev deployment; code opt-in with `NOTIFICATION_STORE=postgres` or `REPO_NOTIFICATION=pg` | `NotificationService` uses `getNotificationRepo()` for create/list/read/delete/count paths. Backend `2.8.34` adds Mongo→Postgres notification backfill tooling, ordered-runner support, a dry-run path, and `scripts/smoke/notifications-postgres.sh`. Deployment `8764fdf` defaults `NOTIFICATION_STORE=postgres` in dev and Gatus requires the notification store mode. Backend `2.8.37` fixes repository mode aliasing so this `postgres` store flag resolves to the Drizzle notification repo rather than Mongo. | -| Repository implementations | Present with first payment-ledger runtime seam | `src/db/repositories/*` and Drizzle schemas exist for the target architecture. Backend `2.8.20` wires `fundsLedgerService` appends/balance reads through `getPaymentRepo()`, making that ledger slice controllable by `REPO_PAYMENT=mongo|dual|pg`. The broader payment, marketplace, and points services still need method-by-method service wiring before their repo flags are safe runtime cutovers. | +| Postgres connection | Available when `PG_URL` is set | Store facades use `src/infrastructure/postgres/client.ts`; the broader `src/db/` Drizzle layer and repository factory are fully populated. | +| Runtime schema bootstrap | Implemented for auth, config, address, and reference stores | Auth tables bootstrapped from `src/services/auth/postgresAuthSchema.ts`; store facades bootstrap their own tables at startup when their `*_STORE=postgres` flag is enabled. | +| Health observability | Implemented in `/api/health` | `checks.postgres` reports `configured`, `required`, `storeModes`, `enabledStores`, and `enabledStoreCount`. Dev Gatus asserts all dev PG-backed store modes are `postgres`, including notifications. Mongoose health check is lazy-loaded and skipped when Mongo is optional under `MONGO_CONNECT_MODE=auto/never`. | +| Auth-owned user store | PG-backed in dev deployment; code opt-in with `AUTH_STORE=postgres` | Auth, passkey, Telegram auth/link/session/temp-verification, and `/api/user` profile paths use an auth-store facade. PG-mode users are mirrored back to Mongo through `legacy_object_id` for compatibility with still-Mongo services. | +| Confirmation-threshold runtime config | PG-backed in dev deployment; code opt-in with `CONFIG_STORE=postgres` | `ConfigSetting` / `ConfigSettingHistory` access for `/api/admin/settings/confirmation-thresholds` and transaction-safety confirmation thresholds uses a config-store facade. PG-mode writes mirror back to Mongo. Legacy models load only for Mongo fallback/backfill/mirror paths. | +| User addresses | PG-backed in dev deployment; code opt-in with `ADDRESS_STORE=postgres` | `/api/addresses` CRUD uses an address-store facade. PG mode enforces one primary address per user with a partial unique index and mirrors writes/deletes back to Mongo. | +| Marketplace categories | PG-backed in dev deployment; code opt-in with `CATEGORY_STORE=postgres` | `CategoryService` and the default `General` category path use a category-store facade. PG-mode writes mirror back to Mongo. Migration 0009 deactivated duplicate active category labels and enforces `categories_active_name_norm_uq` on `lower(btrim(name)) WHERE is_active = true`. | +| Level configuration | PG-backed in dev deployment; code opt-in with `LEVEL_CONFIG_STORE=postgres` | `PointsService` level reads use a level-config facade. `PointTransaction` and user points remain Mongo-backed. `LEVEL_STORE=postgres` accepted as compatibility alias. | +| Shop settings | PG-backed in dev deployment; code opt-in with `SHOP_SETTINGS_STORE=postgres` | Shop settings controller, seller payment rail resolution (`2.8.56` fixes seller shop lookup to handle both uuid and legacy id formats), and review enable/disable checks use a shop-settings facade. PG-mode writes mirror back to Mongo. | +| Marketplace reviews | PG-backed in dev deployment; code opt-in with `REVIEW_STORE=postgres` | Review list/summary/create routes use a review-store facade. PG-mode list responses still hydrate `reviewerId` from the user mirror to preserve frontend shape. | +| Notifications | PG-backed in dev deployment; code opt-in with `NOTIFICATION_STORE=postgres` or `REPO_NOTIFICATION=pg` | `NotificationService` uses `getNotificationRepo()` for create/list/read/delete/count paths. `2.8.55` fixes chat routes and delivers notifications in real time. `2.8.37` fixes repository mode aliasing so `postgres` resolves to the Drizzle notification repo. Backfill script (`npm run backfill:notification:postgres`) and smoke script (`scripts/smoke/notifications-postgres.sh`) are available. | | Oracle quote persistence | Conditional runtime PG write | `/api/payment/request-network/intents` lazily imports `quoteRepo` only when `ORACLE_QUOTING_ENABLED=true`; it writes `payment_quotes` if the PG parent payment row exists, mirrors to Mongo `Payment.quote`, and records `pg_dualwrite_gaps` if PG is behind. | -| Backfill/verify scripts | Available as operator tooling | `MIGRATION_PG_URL` drives backfill scripts; guards restrict allowed target hosts. The marketplace-core runner group now backfills users/categories, request templates, purchase requests, seller offers, and the post-offer `selectedOfferId` remap in dependency order. These scripts are not run automatically by app startup. | +| Funds ledger | Repository-backed, default Mongo | `appendFundsLedgerEntry` and `getFundsBalanceBy*` call `getPaymentRepo()`. Default is `MongoPaymentRepo`; `REPO_PAYMENT=dual`/`pg` exercises the Drizzle ledger after backfill/soak. | +| Backfill/verify scripts | Available as operator tooling | `MIGRATION_PG_URL` drives all backfill scripts; guards restrict allowed target hosts. Marketplace-core runner backfills users/categories, request templates, purchase requests, seller offers, and `selectedOfferId` remap in dependency order. Not run automatically at startup. | +| Guard user role | PG schema-ready | Migration 0017 adds `guard` to the `user_role` enum. `2.8.54` adds guard role support across auth and user management. | +| PG response serialization | Fixed in `2.8.51–2.8.53` | PG response serialization and id resolution in marketplace purchase-request paths corrected; user creation and purchase request unblocked from a PG FK constraint error. | +| Admin user management | PG-capable as of `2.8.50` | Admin user count queries route through postgres-capable stores; admin user management works end-to-end under PG. | +| Seeds | Postgres-capable as of `2.8.47` | Seeds in `src/seeds/*` are store-aware and idempotent; can seed fresh PG under `MONGO_CONNECT_MODE=never`. | ## What Is Still Mongo-Backed -Active app startup, health, and the PG-capable store facades no longer top-load Mongoose models. Auth-owned paths now have an auth-store boundary; confirmation-threshold config, user addresses, categories, level config, shop settings, reviews, and notifications have store/repository boundaries, and their PG-capable facades lazy-load legacy Mongo fallbacks instead of top-loading them. Funds ledger appends/balance reads now use the payment repository seam, but default to Mongo unless `REPO_PAYMENT` is flipped. Broad marketplace requests/offers/templates, most payment paths, points transactions, chat, and admin maintenance actions remain Mongo-first when exercised. +Writes across all dual-write domains go to both Mongo and Postgres. Reads remain Mongo-authoritative for every dual-write domain — read cutover has not been performed for any domain. Chat is repository-backed (Drizzle repo exists) but participants/messages are stored as JSONB blobs rather than normalized child tables; Chat normalization is the primary structural blocker for Chat read cutover. | Domain | Current live store | Why not Postgres yet | |---|---|---| -| Legacy/broad user consumers | MongoDB mirror | Auth-owned users can be PG-backed, but still-Mongo domains expect Mongo ObjectId user references. PG-mode writes therefore maintain a Mongo mirror until those domains are cut over. | -| Admin cleanup / seed address tooling | MongoDB | User-facing address CRUD is PG-capable, but admin cleanup and seed scripts still operate on Mongo first. Backend `2.8.33` makes admin cleanup lazy-load its Mongo models only when cleanup/stat/user-data maintenance actions run. Seed scripts backfill addresses to PG when `ADDRESS_STORE=postgres`. | -| Marketplace requests/offers/templates | Repository-backed, default Mongo | Controller/service paths route through `getMarketplaceRepo()`, and PurchaseRequest/SellerOffer/RequestTemplate backfill tooling is operator-ready. `REPO_MARKETPLACE` still defaults to Mongo, the old unmounted route file remains present for now, and full PG/dual marketplace runtime cutover still needs route/service smoke coverage before flipping. Backend `2.8.36` stops re-exporting the old Mongo-heavy `marketplaceRouter` from the marketplace service index. | -| Payments and escrow state | MongoDB primary | Request Network, AMN scanner, webhook, admin, release/refund, adapter, reconciliation, and legacy payment paths still create/update `Payment` Mongoose documents directly. Payment repository methods exist but are not broadly wired into runtime services yet. The SHKeeper migration report lazy-loads `Payment` and `FundsLedgerEntry` as of backend `2.8.33`, but the report remains Mongo-backed. | -| Funds ledger | Repository-backed, default Mongo | `appendFundsLedgerEntry` and `getFundsBalanceBy*` now call `getPaymentRepo()`. In default mode that is `MongoPaymentRepo`; `REPO_PAYMENT=dual`/`pg` can exercise the PG ledger implementation after backfill/soak. Drizzle balance reads support both UUID refs and external/string refs used by template checkout. | -| Derived destinations and sweeps | Repository-backed, default Mongo | Wallet destination allocation and sweep paths use `getDerivedDestinationRepo()` / `getPaymentRepo()`, but `REPO_DERIVED_DESTINATION` still defaults to Mongo and has not been flipped in dev. | -| Points/referrals/transactions | Repository-backed, default Mongo | `PointsService` uses `getPointsRepo()` and level configuration is PG-capable, but `REPO_POINTS` defaults to Mongo and point transaction/user-point flows have not been flipped in dev. | -| Chat/messages | Repository-backed, default Mongo | Chat service uses `getChatRepo()`, but `REPO_CHAT` / `CHAT_STORE` defaults to Mongo and chat is still treated as a document-shaped domain until a deliberate PG cutover. | -| Notifications | PG-backed in dev deployment; code default Mongo | `NotificationService` routes through `getNotificationRepo()`, so `NOTIFICATION_STORE=postgres` / `REPO_NOTIFICATION=pg` can exercise the Drizzle repo. Backend `2.8.34` adds backfill and smoke coverage; deployment `8764fdf` includes notification in the dev PG baseline; backend `2.8.37` ensures `postgres` is a valid repo-mode alias. | -| Disputes/blog/content/admin cleanup | Mixed | Disputes and blog are repository-backed, but their code defaults still resolve to Mongo until `REPO_DISPUTE`/`BLOG_STORE` are flipped. Admin cleanup still lazy-loads and calls Mongoose models directly for maintenance cleanup/stat/user-data actions. | -| Runtime config outside confirmation thresholds | MongoDB | `ConfigSetting` and `ConfigSettingHistory` are PG-capable for confirmation thresholds only; any future admin-editable settings need to route through the same config-store boundary before they count as cut over. | -| Telegram link/session/temp verification | PG-backed in dev deployment; code default MongoDB | These records move with `AUTH_STORE=postgres`. Dev compose defaults that flag to `postgres`; environments without the override remain Mongo until the flag is flipped. | +| User reads | MongoDB authoritative | Auth-owned users can be PG-backed for writes, but reads remain Mongo-authoritative. Still-Mongo domains expect Mongo ObjectId user references; PG-mode writes maintain a Mongo mirror until all consumers cut over. | +| Admin cleanup / seed address tooling | MongoDB | User-facing address CRUD is PG-capable, but admin cleanup scripts still operate on Mongo first. Seed scripts backfill addresses to PG when `ADDRESS_STORE=postgres`. | +| Marketplace requests/offers/templates reads | Repository-backed writes; Mongo reads | `getMarketplaceRepo()` wires all marketplace writes. `REPO_MARKETPLACE` defaults to Mongo; full PG/dual read cutover needs smoke coverage before flipping. The legacy `marketplaceRouter` is detached from the service index (`2.8.36`). | +| Payments and escrow state reads | MongoDB primary | Request Network, AMN scanner, webhook, admin, release/refund, adapter, reconciliation, and legacy payment paths still create/update `Payment` Mongoose documents directly for reads. Payment Drizzle repo and dual-write repo exist; `REPO_PAYMENT=dual` enables dual-writes, but read paths remain Mongo. | +| Derived destinations and sweeps | Repository-backed writes; Mongo reads | `getDerivedDestinationRepo()` wires writes; `REPO_DERIVED_DESTINATION` defaults to Mongo and has not been flipped in dev. | +| Points/referrals/transactions | Repository-backed writes; Mongo reads | `getPointsRepo()` wires writes; `REPO_POINTS` defaults to Mongo. Level config is PG-capable for reads; point transaction and user-point flows are not flipped in dev. | +| Chat/messages | Repository-backed writes (JSONB shim); Mongo reads | `getChatRepo()` wires writes. `REPO_CHAT` / `CHAT_STORE` defaults to Mongo. `DrizzleChatRepo` uses JSONB blobs for participants and messages — Chat normalization into relational child tables is required before safe read cutover. No dual-write wrapper exists. | +| Disputes/blog | Repository-backed writes; Mongo reads | Both have Drizzle repos and dual-write wrappers. Code defaults resolve to Mongo until `REPO_DISPUTE`/`BLOG_STORE` are flipped. | +| ReleaseHold | Drizzle repo only; default Mongo | `getReleaseHoldRepo()` returns Drizzle or Mongo; no dual-write wrapper — `dual` mode silently uses Mongo. Separate cutover needed. | +| Runtime config outside confirmation thresholds | MongoDB | `ConfigSetting` and `ConfigSettingHistory` are PG-capable for confirmation thresholds only; other admin-editable settings need to route through the config-store boundary before counting as cut over. | +| Telegram link/session/temp verification reads | PG-backed writes in dev; Mongo reads in code default | These records move with `AUTH_STORE=postgres`. Dev compose defaults that flag to `postgres`; read cutover follows the auth-store flag. | ## Env Flag Reality -The backend code defaults every store flag below to `mongo`. Dev deployment overrides eight PG-capable store flags to `postgres` in `deployment/docker-compose.yml` as of `deployment@8764fdf`. +The backend code defaults every store flag below to `mongo`. Dev deployment overrides eight PG-capable store flags to `postgres` in `deployment/docker-compose.yml`. Repository factory normalizes `postgres` and `pg` as equivalent. | Flag | Current meaning | |---|---| @@ -60,21 +129,50 @@ The backend code defaults every store flag below to `mongo`. Dev deployment over | `CONFIG_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes confirmation-threshold settings/history through Postgres. | | `ADDRESS_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes `/api/addresses` through Postgres. | | `CATEGORY_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes marketplace category reads/writes through Postgres. Active PG categories are unique by normalized visible name. | -| `LEVEL_CONFIG_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes level configuration reads and seed replacement through Postgres. `LEVEL_STORE=postgres` is accepted as a compatibility alias. | -| `SHOP_SETTINGS_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes shop settings, review gates, and seller payment rails through Postgres. | +| `LEVEL_CONFIG_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes level configuration reads and seed replacement through Postgres. `LEVEL_STORE=postgres` accepted as compatibility alias. | +| `SHOP_SETTINGS_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes shop settings, review gates, and seller payment rails through Postgres. `2.8.56` fixes seller shop lookup tolerance for uuid vs legacy id formats. | | `REVIEW_STORE` | Code default `mongo`; dev deployment default `postgres`. Routes marketplace reviews through Postgres. | -| `NOTIFICATION_STORE` / `REPO_NOTIFICATION` | Code default `mongo`; dev deployment default `postgres`. Routes notification inbox create/list/read/delete/count through the Drizzle notification repo. Backend `2.8.34` adds `npm run backfill:notification:postgres`, ordered-runner step `notifications`, and `scripts/smoke/notifications-postgres.sh`; backend `2.8.37` makes repository factory flags accept both `postgres` and `pg`. | +| `NOTIFICATION_STORE` / `REPO_NOTIFICATION` | Code default `mongo`; dev deployment default `postgres`. Routes notification inbox create/list/read/delete/count through the Drizzle notification repo. `postgres` and `pg` both resolve correctly since `2.8.37`. | | `PG_URL` | Makes PG code importable/reachable. Required for any `*_STORE=postgres` flag; does not cut over unrelated app domains by itself. | | `MIGRATION_PG_URL` | Used by backfill scripts and migration runbooks; not part of normal request handling. Marketplace-core dry-run/non-dry backfills also require `MIGRATION_MONGO_URL`. | -| `REPO_PAYMENT` | Code default `mongo`. As of backend `2.8.20`, funds ledger appends and balance reads use this flag through `getPaymentRepo()`. Do not flip broad payment runtime to `pg` yet; most payment services still call Mongoose directly. | -| `REPO_USER`, `REPO_POINTS`, `REPO_MARKETPLACE`, `REPO_DEFAULT` | Repository factory flags exist, but broad services are not yet wired through the factory. Treat them as migration controls that need integration verification before relying on them. The factory lazy-loads PG/dual implementations so importing it in Mongo mode does not require `PG_URL`; as of backend `2.8.37`, `postgres` and `pg` both resolve to PG mode. | -| `REPO_RELEASE_HOLD` / `RELEASE_HOLD_STORE` | Code default `mongo`. Release-hold mode must be flipped explicitly; backend `2.8.37` removed the previous fallback where `REPO_DISPUTE=pg` also made release holds look PG-backed. | -| `ORACLE_QUOTING_ENABLED` | Enables server-side quote computation and the only current PG write path in normal checkout: `payment_quotes`, when a PG parent row can be resolved. | +| `REPO_PAYMENT` | Code default `mongo`. Funds ledger appends and balance reads route through this flag. `dual` mode enables dual-write for the ledger seam. Do not flip broad payment runtime to `pg` yet; most payment services still call Mongoose directly for reads. | +| `REPO_MARKETPLACE` | Code default `mongo`. All marketplace writes route through `getMarketplaceRepo()`. Full read cutover needs smoke coverage. | +| `REPO_USER`, `REPO_POINTS`, `REPO_DERIVED_DESTINATION`, `REPO_TREZOR` | Factory flags with full trio (mongo/dual/pg). Writes are wired; reads remain Mongo-authoritative until each flag is flipped and verified. | +| `REPO_DISPUTE` / `DISPUTE_STORE`, `REPO_BLOG` / `BLOG_STORE` | Code default `mongo`. Dual-write wrappers exist; flip per-domain after verification. | +| `REPO_CHAT` / `CHAT_STORE` | Code default `mongo`. Chat normalization (JSONB→relational) is a structural blocker; no dual-write wrapper. | +| `REPO_RELEASE_HOLD` / `RELEASE_HOLD_STORE` | Code default `mongo`. No dual-write wrapper; `dual` silently uses Mongo. Must be flipped explicitly. As of `2.8.37`, `REPO_DISPUTE=pg` no longer leaks into release hold mode. | +| `ORACLE_QUOTING_ENABLED` | Enables server-side quote computation and the `payment_quotes` PG write in checkout when a PG parent payment row exists. | +| `MONGO_CONNECT_MODE` | Handled in Mongoose connection setup (not in the repository factory). `auto`/`never` allow PG-only boot; lazy Mongoose health check skips when Mongo is optional. | + +## Overall Migration Phase + +| Phase | Status | +|---|---| +| Schema design | Complete — 32 tables, 18 migrations (0000–0017) | +| Drizzle repos | Complete — all 11 factory domains have a Drizzle repo | +| Dual-write wrappers | Mostly complete — 9 of 11 domains have a dual-write wrapper (Chat and ReleaseHold are exceptions) | +| Write cutover (dual-write active) | Not yet enabled by default — `REPO_DEFAULT` is still `mongo`; must be flipped per-domain with care | +| Read cutover | Not started for any domain — Mongo remains authoritative for all reads | +| Prod backfill | Not run — backfill scripts are operator-ready but not executed against production | +| Chat normalization | Blocked — participants/messages stored as JSONB; relational normalization required before Chat read cutover | + +Estimated overall: **schema and infrastructure phase complete; write-seam phase substantially complete; read cutover and backfill execution remain for every domain.** + +## Recent Progress Since Last Update (2.8.37 → 2.8.56) + +- **2.8.38–2.8.46:** Complete dual-write repos for all remaining domains; Drizzle migrations pipeline finalized; TTL scheduler added; shop lookup bug-fixed. +- **2.8.47:** Seeds made Postgres-capable and idempotent for PG-only boot (`MONGO_CONNECT_MODE=never`). +- **2.8.48–2.8.49:** Fresh-DB PG migrate + seed path corrected; 0013/0014 migrations made valid for a fresh `drizzle-kit migrate` run. +- **2.8.50:** Admin user counts routed through postgres-capable stores; admin user management works end-to-end under PG. +- **2.8.51–2.8.53:** PG response serialization and id resolution corrected in marketplace; user creation and purchase request creation unblocked from PG FK constraint errors. +- **2.8.54:** `guard` user role added to `user_role` enum (migration 0017); guard role support across auth and user management. +- **2.8.55:** Chat routes fixed; notifications delivered in real time alongside chat. +- **2.8.56:** Seller shop lookup made tolerant of both uuid and legacy id formats; `dataCleanupService` guarded against `MONGO_CONNECT_MODE=never`. ## Next Cutover Work -1. Apply Drizzle migrations to the target Postgres database. -2. For dev/test data, either run the existing backfills below or reseed acceptable test data before relying on the PG-backed stores. The deployment default flip does not move historical Mongo rows by itself. +1. Apply Drizzle migrations to the target Postgres database (0000–0017 must be in-order; 0002 is a reset migration — confirm idempotency on existing instances). +2. For dev/test data, run the existing backfills or reseed acceptable test data before relying on PG-backed stores. The deployment default flip does not move historical Mongo rows. 3. For auth cutover, run `PG_URL=... npm run backfill:auth:postgres`, verify counts, and confirm `AUTH_STORE=postgres` in the target runtime. 4. For confirmation-threshold config cutover, run `PG_URL=... npm run backfill:config:postgres`, verify counts/history, and confirm `CONFIG_STORE=postgres`. 5. For address cutover, run `PG_URL=... npm run backfill:address:postgres`, verify one-primary invariants, and confirm `ADDRESS_STORE=postgres`. @@ -83,16 +181,17 @@ The backend code defaults every store flag below to `mongo`. Dev deployment over - `PG_URL=... npm run backfill:level-config:postgres` - `PG_URL=... npm run backfill:shop-settings:postgres` - `PG_URL=... npm run backfill:review:postgres` -7. Run `PG_URL=... scripts/smoke/categories-postgres-unique.sh` and `PG_URL=... MONGODB_URI=... scripts/smoke/reference-stores-postgres.sh`, then confirm `CATEGORY_STORE=postgres LEVEL_CONFIG_STORE=postgres SHOP_SETTINGS_STORE=postgres REVIEW_STORE=postgres` in non-prod. -8. For marketplace-core data, run `MIGRATION_MONGO_URL=... MIGRATION_PG_URL=... npm run backfill:marketplace-core:postgres:dry-run`, then the non-dry `npm run backfill:marketplace-core:postgres` against non-prod. The group runs root dependencies, RequestTemplate rows, PurchaseRequest main rows, SellerOffer rows, then the selected-offer remap. +7. Run `PG_URL=... scripts/smoke/categories-postgres-unique.sh` and `PG_URL=... MONGODB_URI=... scripts/smoke/reference-stores-postgres.sh`, then confirm reference store flags in non-prod. +8. For marketplace-core data, run `MIGRATION_MONGO_URL=... MIGRATION_PG_URL=... npm run backfill:marketplace-core:postgres:dry-run`, then the non-dry run. The group runs root dependencies, RequestTemplate rows, PurchaseRequest main rows, SellerOffer rows, then the selected-offer remap. 9. Run `scripts/smoke/marketplace-core-postgres-backfill.sh` with the same migration DSNs and record row-count/checksum results. -10. For notifications, run `PG_URL=... npm run backfill:notification:postgres` and `PG_URL=... scripts/smoke/notifications-postgres.sh` against dev to validate the new default. -11. Continue payment-domain wiring after the ledger seam: add the missing payment repo methods for provider lookups, transaction-hash/webhook lookups, metadata/blockchain patching, template duplicate cleanup, and quote updates before moving `paymentService`, `paymentCoordinator`, RN, or AMN scanner routes. -12. Add a derived-destination/sweep repository seam before payment PG cutover; destination allocation is payment-address state and should not stay Mongo-only once payments become PG-backed. -13. Wire remaining services to repository interfaces one domain at a time. -14. Enable `dual` mode per large domain only after wiring is proven by tests and smoke checks. -15. Run shadow-read/reconcile during a soak window. -16. Flip reads to `pg` per domain only after zero-diff shadow reads and a rollback plan are in place. +10. For notifications, run `PG_URL=... npm run backfill:notification:postgres` and `PG_URL=... scripts/smoke/notifications-postgres.sh` against dev to validate the current default. +11. Enable `REPO_MARKETPLACE=dual`, `REPO_PAYMENT=dual`, `REPO_POINTS=dual`, `REPO_DERIVED_DESTINATION=dual`, `REPO_TREZOR=dual`, `REPO_DISPUTE=dual`, `REPO_BLOG=dual` one domain at a time after backfill verification, and run a soak window before flipping reads. +12. Continue payment-domain wiring: add missing payment repo methods for provider lookups, transaction-hash/webhook lookups, metadata/blockchain patching, template duplicate cleanup, and quote updates before moving `paymentService`, `paymentCoordinator`, RN, or AMN scanner routes. +13. Add a derived-destination/sweep repository seam before payment PG read cutover; destination allocation is payment-address state and should not remain Mongo-only once payments become PG-backed for reads. +14. Resolve Chat normalization: design relational child tables for participants and messages; migrate `DrizzleChatRepo` away from JSONB blobs; add `DualWriteChatRepo`; flip `REPO_CHAT=dual` only after normalization. +15. Add `DualWriteReleaseHoldRepo`; flip `REPO_RELEASE_HOLD=dual` explicitly after wiring is proven. +16. Flip reads to `pg` per domain only after zero-diff shadow reads and a documented rollback plan are in place. +17. Run prod backfill under a maintenance window with `MIGRATION_PG_URL` pointing at prod; validate row counts before cutting over reads. ## Related Docs diff --git a/02 - Data Models/PurchaseRequest.md b/02 - Data Models/PurchaseRequest.md index e578fe1..ab0b416 100644 --- a/02 - Data Models/PurchaseRequest.md +++ b/02 - Data Models/PurchaseRequest.md @@ -1,20 +1,28 @@ --- title: PurchaseRequest -tags: [data-model, mongoose] +tags: [data-model, mongoose, postgres, drizzle] aliases: [Purchase Request, Buy Request, IPurchaseRequest] --- # PurchaseRequest -> **Last updated:** 2026-05-31 — `budget.currency` aligned with template/Postgres enum (`USD`, `EUR`, `IRR`, `USDT`, `USDC`); template checkout now preserves seller-owned delivery mode and overlays buyer address/email. +> **Last updated:** 2026-06-03 — added Postgres / Drizzle schema section, child-table breakdowns, migration status, and dispute/escrow hold fields present in both Mongo and PG schemas. The central buyer-side document. A `PurchaseRequest` captures what a buyer wants to acquire (physical product, digital product, service, or consultation), the budget envelope, urgency, delivery details, and the entire lifecycle from creation through payment, delivery, and completion. Sellers respond by attaching [[SellerOffer]] documents; the buyer accepts one, a [[Payment]] is opened, and delivery is verified by a 6-digit code. -> [!note] Source -> `backend/src/models/PurchaseRequest.ts:95` — schema definition -> `backend/src/models/PurchaseRequest.ts:387` — model export +> [!note] Sources +> Mongo model: `backend/src/models/PurchaseRequest.ts:95` — schema definition; `:387` — model export +> Drizzle schema: `backend/src/db/schema/purchaseRequest.ts` -## Schema +## Migration Status + +**DUAL-WRITE active** — part of `DualWriteMarketplaceRepo`. Writes go to both Mongo and Postgres; reads still come from Mongo. Backfill and read-cutover are human-gated and not yet executed. + +--- + +## Mongo Schema + +### Fields | Field | Type | Required | Default | Validation | Index | Description | | --- | --- | --- | --- | --- | --- | --- | @@ -74,12 +82,18 @@ The central buyer-side document. A `PurchaseRequest` captures what a buyer wants | `serviceInfo.location` | String | no | — | trim, maxlength 200 | — | Service location. | | `serviceInfo.requirements[]` | String[] | no | — | trim | — | Pre-requisites. | | `attachments[]` | String[] | no | — | — | — | Attached file URLs. | -| `offers[]` | ObjectId → [[SellerOffer]] | no | — | — | — | Offers received. | +| `offers[]` | ObjectId → [[SellerOffer]] | no | — | — | — | Offers received. **Dropped in PG** — query `SellerOffer WHERE purchase_request_id = ?` instead. | | `selectedOfferId` | ObjectId → [[SellerOffer]] | no | `null` | — | — | Accepted offer. | | `rating` | Number | no | `null` | min 1, max 5 | — | Buyer's post-delivery rating. | | `feedback` | String | no | `null` | maxlength 1000 | — | Buyer's feedback text. | | `deliveryConfirmed` | Boolean | no | `false` | — | — | Buyer confirmation flag. | | `deliveryConfirmedAt` | Date | no | `null` | — | — | Confirmation timestamp. | +| `disputeRaised` | Boolean | no | `false` | — | — | Escrow: whether a dispute has been raised. | +| `disputeRaisedAt` | Date | no | `null` | — | — | When the dispute was raised. | +| `disputeResolved` | Boolean | no | `false` | — | — | Escrow: whether dispute is resolved. | +| `disputeResolvedAt` | Date | no | `null` | — | — | When it was resolved. | +| `disputeHoldReason` | String | no | `null` | — | — | Human-readable hold reason. | +| `holdUntil` | Date | no | `null` | — | — | Escrow hold expiry; partial index in PG for expiry sweeps. | | `metadata.source` | String | no | `manual` | enum: `manual` / `template` / `api` | — | Where the request came from. | | `metadata.templateId` | String | no | — | trim | — | Originating [[RequestTemplate]] id. | | `metadata.version` | String | no | — | trim | — | Schema version. | @@ -92,13 +106,13 @@ The central buyer-side document. A `PurchaseRequest` captures what a buyer wants **Note:** `finalized` and `archived` are **not** valid status values and do not appear in the `IPurchaseRequest` frontend type or the Mongoose schema enum. Using either would cause a validation error. -## Virtuals +### Virtuals None defined. -## Indexes +### Mongo Indexes -Single-field — `backend/src/models/PurchaseRequest.ts:376-381`: +Single-field — `backend/src/models/PurchaseRequest.ts:414-419`: - `{ buyerId: 1 }` - `{ categoryId: 1 }` @@ -107,26 +121,262 @@ Single-field — `backend/src/models/PurchaseRequest.ts:376-381`: - `{ createdAt: -1 }` - `{ urgency: 1 }` -Compound — `backend/src/models/PurchaseRequest.ts:384-385`: +Compound — `backend/src/models/PurchaseRequest.ts:422-423`: - `{ productType: 1, status: 1 }` - `{ categoryId: 1, productType: 1 }` -## Pre/Post Hooks +### Pre/Post Hooks None declared at the schema level. -## Instance Methods +### Instance Methods None defined. -## Static Methods +### Static Methods None defined. +--- + +## Postgres / Drizzle Schema + +Source: `backend/src/db/schema/purchaseRequest.ts` + +The PG model normalises the embedded Mongo subdocuments into 7 tables. The `offers[]` array is dropped; [[SellerOffer]] holds `purchase_request_id` as a back-reference. + +### Enums (PG-level) + +| Enum name | Values | +| --- | --- | +| `purchase_request_status` | `pending_payment`, `pending`, `active`, `received_offers`, `in_negotiation`, `payment`, `processing`, `delivery`, `delivered`, `confirming`, `completed`, `cancelled`, `seller_paid` | +| `product_type` | `physical_product`, `digital_product`, `service`, `consultation` | +| `request_urgency` | `low`, `medium`, `high`, `urgent` | +| `delivery_type` | `physical`, `online` | +| `service_session_type` | `online`, `in_person`, `hybrid` | +| `pr_metadata_source` | `manual`, `template`, `api` | +| `budget_currency` | `USD`, `EUR`, `IRR`, `USDT`, `USDC` | + +### Table: `purchase_requests` (main) + +| Column | PG type | Nullable | Default | Notes | +| --- | --- | --- | --- | --- | +| `id` | uuid PK | no | `gen_random_uuid()` | | +| `legacy_object_id` | text | yes | — | 24-char Mongo ObjectId; partial-unique index | +| `buyer_id` | uuid | no | — | FK → `users(id)` | +| `category_id` | uuid | no | — | FK → `categories(id)` | +| `title` | varchar(200) | no | — | | +| `description` | text | no | — | | +| `product_type` | enum | yes | `physical_product` | | +| `product_link` | varchar(2000) | yes | — | CHECK: `^https?://.+` | +| `size` | varchar(100) | yes | — | | +| `color` | varchar(100) | yes | — | | +| `brand` | varchar(100) | yes | — | | +| `quantity` | integer | yes | `1` | CHECK ≥ 1 | +| `budget_min` | numeric(38,18) | yes | — | CHECK ≥ 0 | +| `budget_max` | numeric(38,18) | yes | — | CHECK ≥ 0 | +| `budget_currency` | enum | yes | `USDT` | | +| `urgency` | enum | no | `medium` | | +| `status` | enum | no | `pending` | 13-value escrow-critical enum | +| `is_public` | boolean | yes | `true` | | +| `tags` | text[] | yes | `'{}'` | | +| `attachments` | text[] | yes | `'{}'` | | +| `selected_offer_id` | uuid | yes | — | FK → `seller_offers(id)` | +| `rating` | smallint | yes | — | CHECK 1–5 or NULL | +| `feedback` | text | yes | — | CHECK length ≤ 1000 or NULL | +| `delivery_confirmed` | boolean | yes | `false` | | +| `delivery_confirmed_at` | timestamptz | yes | — | | +| `dispute_raised` | boolean | no | `false` | | +| `dispute_raised_at` | timestamptz | yes | — | | +| `dispute_resolved` | boolean | no | `false` | | +| `dispute_resolved_at` | timestamptz | yes | — | | +| `dispute_hold_reason` | text | yes | — | | +| `hold_until` | timestamptz | yes | — | Partial index WHERE NOT NULL | +| `metadata_source` | enum | yes | `manual` | | +| `metadata_template_id` | varchar(100) | yes | — | | +| `metadata_version` | varchar(50) | yes | — | | +| `created_at` | timestamptz | no | `now()` | | +| `updated_at` | timestamptz | no | `now()` | | + +**Indexes on `purchase_requests`:** + +| Index | Type | Columns / condition | +| --- | --- | --- | +| `idx_pr_buyer_id` | btree | `buyer_id` | +| `idx_pr_category_id` | btree | `category_id` | +| `idx_pr_product_type` | btree | `product_type` | +| `idx_pr_status` | btree | `status` | +| `idx_pr_created_at` | btree | `created_at` | +| `idx_pr_urgency` | btree | `urgency` | +| `purchase_requests_legacy_object_id_uq` | partial-unique | `legacy_object_id` WHERE NOT NULL | +| `idx_pr_product_type_status` | btree | `(product_type, status)` | +| `idx_pr_category_product_type` | btree | `(category_id, product_type)` | +| `idx_pr_hold_until` | partial btree | `hold_until` WHERE NOT NULL | +| `idx_pr_dispute_raised` | partial btree | `dispute_raised` WHERE `dispute_raised = true` | + +**CHECK constraints on `purchase_requests`:** + +| Name | Expression | +| --- | --- | +| `pr_rating_ck` | `rating IS NULL OR (rating >= 1 AND rating <= 5)` | +| `pr_feedback_len_ck` | `feedback IS NULL OR length(feedback) <= 1000` | +| `pr_quantity_min_ck` | `quantity IS NULL OR quantity >= 1` | +| `pr_budget_min_ck` | `budget_min IS NULL OR budget_min >= 0` | +| `pr_budget_max_ck` | `budget_max IS NULL OR budget_max >= 0` | +| `pr_product_link_ck` | `product_link IS NULL OR product_link ~ '^https?://.+'` | + +--- + +### Table: `purchase_request_delivery_info` (1:1) + +Child of `purchase_requests`. Holds all delivery logistics. + +| Column | PG type | Nullable | Default | Notes | +| --- | --- | --- | --- | --- | +| `id` | uuid PK | no | random | | +| `legacy_object_id` | text | yes | — | Parent PR's legacy ObjectId for traceability | +| `purchase_request_id` | uuid UNIQUE | no | — | FK → `purchase_requests(id)` CASCADE | +| `delivery_type` | enum | no | `physical` | | +| `address` | varchar(500) | yes | — | | +| `preferred_date` | timestamptz | yes | — | | +| `notes` | text | yes | — | | +| `email` | varchar(255) | yes | — | CHECK: email regex or NULL | +| `delivery_date_time` | timestamptz | yes | — | | +| `delivery_date` | date | yes | — | | +| `shipped_at` | timestamptz | yes | — | | +| `delivery_code` | varchar(6) | yes | — | CHECK: length = 6 or NULL | +| `delivery_code_generated_at` | timestamptz | yes | — | | +| `delivery_code_expires_at` | timestamptz | yes | — | | +| `delivery_code_used` | boolean | yes | `false` | | +| `delivery_code_used_at` | timestamptz | yes | — | | +| `delivery_code_used_by` | uuid | yes | — | FK → `users(id)` | +| `delivered_at` | timestamptz | yes | — | | +| `created_at` | timestamptz | no | `now()` | | +| `updated_at` | timestamptz | no | `now()` | | + +**Indexes:** `idx_pr_delivery_info_pr_id` on `purchase_request_id` + +**CHECK constraints:** `pr_di_delivery_code_len_ck` (`length = 6 or NULL`), `pr_di_email_fmt_ck` (email regex) + +--- + +### Table: `purchase_request_delivery_address` (1:1 under delivery_info) + +| Column | PG type | Nullable | Notes | +| --- | --- | --- | --- | +| `id` | uuid PK | no | | +| `legacy_object_id` | text | yes | | +| `delivery_info_id` | uuid UNIQUE | no | FK → `purchase_request_delivery_info(id)` CASCADE | +| `recipient_name` | varchar(200) | yes | | +| `phone_number` | varchar(20) | yes | | +| `full_address` | text | yes | | +| `address_type` | varchar(50) | yes | e.g. Home / Office | + +**Index:** `idx_pr_delivery_addr_info_id` on `delivery_info_id` + +--- + +### Table: `purchase_request_seller_delivery_info` (1:1 under delivery_info) + +| Column | PG type | Nullable | Default | Notes | +| --- | --- | --- | --- | --- | +| `id` | uuid PK | no | random | | +| `legacy_object_id` | text | yes | — | | +| `delivery_info_id` | uuid UNIQUE | no | — | FK → `purchase_request_delivery_info(id)` CASCADE | +| `estimated_delivery_date` | timestamptz | yes | — | | +| `estimated_delivery_time` | varchar(50) | yes | — | | +| `tracking_number` | varchar(100) | yes | — | | +| `delivery_notes` | text | yes | — | | +| `shipping_method` | varchar(100) | yes | — | | +| `download_link` | varchar(2000) | yes | — | | +| `digital_files` | text[] | yes | `'{}'` | | +| `created_at` | timestamptz | no | `now()` | | +| `updated_at` | timestamptz | no | `now()` | | + +**Index:** `idx_pr_seller_di_info_id` on `delivery_info_id` + +--- + +### Table: `delivery_attempts` (1:N under delivery_info) + +Append-only audit log of code-entry attempts. + +| Column | PG type | Nullable | Default | Notes | +| --- | --- | --- | --- | --- | +| `id` | uuid PK | no | random | | +| `delivery_info_id` | uuid | no | — | FK → `purchase_request_delivery_info(id)` CASCADE | +| `seller_id` | uuid | no | — | FK → `users(id)` | +| `attempted_at` | timestamptz | no | `now()` | | +| `success` | boolean | no | — | | +| `code` | varchar(100) | yes | — | Only stored on successful attempts | + +**Indexes:** `idx_delivery_attempts_info_id`, `idx_delivery_attempts_seller_id`, `idx_delivery_attempts_success` + +--- + +### Table: `purchase_request_service_info` (1:1) + +Only populated for `service` / `consultation` product types. + +| Column | PG type | Nullable | Default | Notes | +| --- | --- | --- | --- | --- | +| `id` | uuid PK | no | random | | +| `legacy_object_id` | text | yes | — | | +| `purchase_request_id` | uuid UNIQUE | no | — | FK → `purchase_requests(id)` CASCADE | +| `duration` | numeric(5,2) | yes | — | CHECK ≥ 0.5 | +| `session_type` | enum | yes | — | `online` / `in_person` / `hybrid` | +| `location` | varchar(200) | yes | — | | +| `requirements` | text[] | yes | `'{}'` | | +| `created_at` | timestamptz | no | `now()` | | +| `updated_at` | timestamptz | no | `now()` | | + +**Index:** `idx_pr_service_info_pr_id` +**CHECK:** `pr_si_duration_min_ck` (`duration IS NULL OR duration >= 0.5`) + +--- + +### Table: `purchase_request_specifications` (1:N) + +Queryable `{key, value, label}` specs extracted from the Mongo embedded array. + +| Column | PG type | Nullable | Default | Notes | +| --- | --- | --- | --- | --- | +| `id` | uuid PK | no | random | | +| `purchase_request_id` | uuid | no | — | FK → `purchase_requests(id)` CASCADE | +| `key` | varchar(255) | no | — | | +| `value` | text | no | — | | +| `label` | varchar(255) | yes | — | | +| `position` | integer | no | `0` | Preserves array order for round-trip fidelity | + +**Indexes:** `idx_pr_specs_pr_id`, `idx_pr_specs_key`, partial-unique `purchase_request_specifications_request_key_uq` on `(purchase_request_id, key)` + +--- + +### Table: `purchase_request_preferred_sellers` (N:M junction) + +| Column | PG type | Nullable | Notes | +| --- | --- | --- | --- | +| `purchase_request_id` | uuid | no | FK → `purchase_requests(id)` | +| `seller_id` | uuid | no | FK → `users(id)` | + +**Indexes:** composite unique `idx_pr_preferred_sellers_uq` on `(purchase_request_id, seller_id)`; `idx_pr_preferred_sellers_seller_id` on `seller_id` + +--- + +### Design Notes + +- **`offers[]` dropped in PG.** The Mongo `offers[]` array is not migrated. Query `SellerOffer WHERE purchase_request_id = ?` instead. +- **Money scale.** `budget_min` / `budget_max` use `numeric(38,18)` (project-wide crypto convention) rather than the `numeric(15,8)` suggested in the migration guide, for consistency with `Payment` and `FundsLedgerEntry`. +- **`tags` / `attachments`** stored as `text[]` (not JSONB) to enable `ANY()` array queries without a child table. +- **`legacy_object_id`** on every table uses a partial-unique index (`WHERE NOT NULL`) for idempotent backfill upserts. +- **Dispute / escrow hold fields** (`dispute_raised`, `dispute_raised_at`, `dispute_resolved`, `dispute_resolved_at`, `dispute_hold_reason`, `hold_until`) are present in both the Mongo interface (`IPurchaseRequest`) and the PG main table. They were added to the Mongo schema before the PG migration and are considered escrow-critical. + +--- + ## Relationships -- **References**: [[User]] (`buyerId`, `preferredSellerIds[]`, `deliveryInfo.deliveryCodeUsedBy`, `deliveryInfo.deliveryAttempts[].sellerId`), [[Category]] (`categoryId`), [[SellerOffer]] (`offers[]`, `selectedOfferId`). +- **References**: [[User]] (`buyerId`, `preferredSellerIds[]`, `deliveryInfo.deliveryCodeUsedBy`, `deliveryInfo.deliveryAttempts[].sellerId`), [[Category]] (`categoryId`), [[SellerOffer]] (`offers[]` Mongo only, `selectedOfferId`). - **Referenced by**: [[SellerOffer]] (`purchaseRequestId`), [[Payment]] (`purchaseRequestId`), [[Dispute]] (`purchaseRequestId`), [[Chat]] (`relatedTo.id` when `relatedTo.type === 'PurchaseRequest'`), [[Review]] (`purchaseRequestId`). ## Template Checkout Mapping @@ -175,7 +425,7 @@ PurchaseRequest.find({ isPublic: true, status: 'active' }).sort({ createdAt: -1 // Sellers' eligible queue PurchaseRequest.find({ productType, status: 'active', categoryId }); -// Populate offers +// Populate offers (Mongo only — offers[] array is not in PG) PurchaseRequest.findById(id).populate('offers').populate('selectedOfferId'); // Redeem delivery code @@ -183,6 +433,12 @@ PurchaseRequest.findOneAndUpdate( { _id: id, 'deliveryInfo.deliveryCode': code, 'deliveryInfo.deliveryCodeUsed': false }, { $set: { 'deliveryInfo.deliveryCodeUsed': true, 'deliveryInfo.deliveryCodeUsedAt': new Date() } } ); + +// PG: offers for a request +// SELECT * FROM seller_offers WHERE purchase_request_id = $1; + +// PG: find requests with live escrow hold +// SELECT * FROM purchase_requests WHERE hold_until IS NOT NULL AND hold_until > now(); ``` Related: [[SellerOffer]], [[Payment]], [[Chat]], [[Dispute]], [[Review]], [[RequestTemplate]], [[Category]]. diff --git a/02 - Data Models/SellerOffer.md b/02 - Data Models/SellerOffer.md index 1a0b84c..ba57225 100644 --- a/02 - Data Models/SellerOffer.md +++ b/02 - Data Models/SellerOffer.md @@ -1,21 +1,28 @@ --- title: SellerOffer -tags: [data-model, mongoose] +tags: [data-model, mongoose, postgres] aliases: [Seller Offer, Bid, ISellerOffer] --- # SellerOffer -> **Last updated:** 2026-05-31 — added `TRY` pricing support for oracle/depeg quoting. +> **Last updated:** 2026-06-03 — added Postgres/Drizzle table definition and migration status. A seller's bid against a [[PurchaseRequest]]. Stores the proposed price, the delivery time commitment, optional notes/attachments, and a small status machine (`pending` / `accepted` / `rejected` / `withdrawn`). The parent `PurchaseRequest` keeps the array of offer ids in `offers[]` and the chosen one in `selectedOfferId`. > [!note] Source -> `backend/src/models/SellerOffer.ts:24` — schema definition -> `backend/src/models/SellerOffer.ts:100` — model export +> `backend/src/models/SellerOffer.ts:24` — Mongoose schema definition +> `backend/src/models/SellerOffer.ts:100` — Mongoose model export +> `backend/src/db/schema/sellerOffer.ts` — Drizzle/Postgres table definition + +## Migration Status + +**DUAL-WRITE** — part of `DualWriteMarketplaceRepo`. Writes go to both MongoDB and Postgres; reads still come from MongoDB. ## Schema +### Mongoose (MongoDB) + | Field | Type | Required | Default | Validation | Index | Description | | --- | --- | --- | --- | --- | --- | --- | | `sellerId` | ObjectId → [[User]] | yes | — | — | yes | Seller submitting the bid. | @@ -37,11 +44,62 @@ A seller's bid against a [[PurchaseRequest]]. Stores the proposed price, the del > **Status enum note:** `active` is accepted by the current backend schema for marketplace/listing flows, in addition to the negotiation statuses `pending | accepted | rejected | withdrawn`. +### Postgres (Drizzle) — `seller_offers` + +Table: `seller_offers` | Schema file: `backend/src/db/schema/sellerOffer.ts` + +| PG Column | Drizzle Type | Nullable | Default | Notes | +| --- | --- | --- | --- | --- | +| `id` | `uuid` PK | no | `gen_random_uuid()` | PG primary key | +| `legacy_object_id` | `text` | yes | — | Mongo ObjectId bridge; partial-unique WHERE NOT NULL | +| `seller_id` | `uuid` FK → `users` CASCADE | no | — | Maps from `sellerId` | +| `purchase_request_id` | `uuid` FK → `purchase_requests` CASCADE | no | — | Maps from `purchaseRequestId` | +| `title` | `varchar(200)` | no | — | | +| `description` | `varchar(1000)` | no | — | | +| `price_amount` | `numeric(18,8)` | no | — | CHECK `price_amount >= 0` | +| `price_currency` | `offer_currency` enum | no | — | `USD \| EUR \| IRR \| USDT \| USDC \| TRY` | +| `delivery_time_amount` | `int` | no | — | CHECK `delivery_time_amount >= 1` | +| `delivery_time_unit` | `delivery_unit` enum | no | — | `hours \| days \| weeks` | +| `status` | `offer_status` enum | no | `pending` | `pending \| accepted \| rejected \| withdrawn \| active` | +| `attachments` | `text[]` | yes | — | | +| `notes` | `text` | yes | — | | +| `valid_until` | `timestamp with time zone` | yes | — | Maps from `validUntil` | +| `require_aml_check` | `boolean` | yes | — | | +| `aml_block_on_failure` | `boolean` | yes | — | CHECK: block requires check (AML coherence) | +| `created_at` | `timestamp with time zone` | no | `now()` | | +| `updated_at` | `timestamp with time zone` | no | `now()` | | + +**Enums used:** + +| Enum name | Values | +| --- | --- | +| `offer_status` | `pending`, `accepted`, `rejected`, `withdrawn`, `active` | +| `offer_currency` | `USD`, `EUR`, `IRR`, `USDT`, `USDC`, `TRY` | +| `delivery_unit` | `hours`, `days`, `weeks` | + +**Constraints:** +- `CHECK (price_amount >= 0)` +- `CHECK (delivery_time_amount >= 1)` +- AML coherence check: `aml_block_on_failure = true` requires `require_aml_check = true` + +**Money precision note:** `price_amount` uses `numeric(18,8)` — differs from the `numeric(38,18)` used by `payments` and `funds_ledger_entries`. This matches the Migration Guide specification for offer amounts. + +#### Postgres Indexes + +| Index | Type | Notes | +| --- | --- | --- | +| `seller_id` | btree | | +| `purchase_request_id` | btree | | +| `status` | btree | | +| `created_at DESC` | btree | | +| `(purchase_request_id, seller_id)` | btree | composite | +| `legacy_object_id` | partial-unique | WHERE NOT NULL; idempotent backfill upserts | + ## Virtuals None defined. -## Indexes +## Mongoose Indexes Defined at `backend/src/models/SellerOffer.ts:95-98`: @@ -80,6 +138,8 @@ The frontend exposes this via the `withdrawOffer(offerId)` action in `src/action - **References**: [[User]] (`sellerId`), [[PurchaseRequest]] (`purchaseRequestId`). - **Referenced by**: [[PurchaseRequest]] (`offers[]`, `selectedOfferId`), [[Payment]] (`sellerOfferId`), [[Chat]] (`relatedTo.id` when `relatedTo.type === 'SellerOffer'`). +- **PG FKs**: `seller_offers.seller_id → users.id CASCADE`, `seller_offers.purchase_request_id → purchase_requests.id CASCADE`. +- **Referenced by (PG)**: `payments.seller_offer_id` (polymorphic triple), `payment_quotes` (via payment join). ## State Transitions @@ -96,6 +156,8 @@ stateDiagram-v2 ## Common Queries +### MongoDB + ```ts // Offers for a request SellerOffer.find({ purchaseRequestId }).sort({ createdAt: -1 }); @@ -113,4 +175,36 @@ SellerOffer.updateMany( SellerOffer.find({ validUntil: { $lt: new Date() }, status: 'pending' }); ``` +### Postgres (Drizzle) + +```ts +// Offers for a request +db.select().from(sellerOffers) + .where(eq(sellerOffers.purchaseRequestId, requestId)) + .orderBy(desc(sellerOffers.createdAt)); + +// Seller's pending offers +db.select().from(sellerOffers) + .where(and( + eq(sellerOffers.sellerId, sellerId), + eq(sellerOffers.status, 'pending') + )); + +// Reject siblings on accept +db.update(sellerOffers) + .set({ status: 'rejected' }) + .where(and( + eq(sellerOffers.purchaseRequestId, purchaseRequestId), + ne(sellerOffers.id, acceptedId), + eq(sellerOffers.status, 'pending') + )); + +// Cleanup expired offers +db.select().from(sellerOffers) + .where(and( + lt(sellerOffers.validUntil, new Date()), + eq(sellerOffers.status, 'pending') + )); +``` + Related: [[PurchaseRequest]], [[Payment]], [[User]]. diff --git a/02 - Data Models/User.md b/02 - Data Models/User.md index ef1b0fa..d229670 100644 --- a/02 - Data Models/User.md +++ b/02 - Data Models/User.md @@ -1,14 +1,108 @@ --- title: User -tags: [data-model, mongoose] +tags: [data-model, mongoose, postgres, dual-write] aliases: [User Model, IUser, Account] --- # User -> **Last updated:** 2026-05-29 — aligned with code (see [Doc vs Code Audit Report](../09%20-%20Audits/Doc%20vs%20Code%20Audit%20Report%20-%202026-05-29.md)) +> **Last updated:** 2026-06-03 — added Postgres/Drizzle schema, `guard` role (migration 0017), dual-write status. Previous update: 2026-05-29 (see [Doc vs Code Audit Report](../09%20-%20Audits/Doc%20vs%20Code%20Audit%20Report%20-%202026-05-29.md)) -The core identity document for every actor in the marketplace: buyers, sellers, and admins. Stores credentials (password + WebAuthn passkeys), profile/preference data, referral bookkeeping, point balances, and a soft-delete status flag. Almost every other model carries an `ObjectId` reference back to `User`, so this collection is the relational hub of the system. +The core identity document for every actor in the marketplace: buyers, sellers, and admins. Stores credentials (password + WebAuthn passkeys), profile/preference data, referral bookkeeping, point balances, and a soft-delete status flag. Almost every other model carries an `ObjectId` (Mongo) or `uuid` (Postgres) reference back to `User`, so this collection is the relational hub of the system. + +> [!info] Migration status: DUAL-WRITE +> Writes go to **both** MongoDB (`User` collection) and Postgres (`users` table) via `DualWriteUserRepo`. +> Reads still come from **MongoDB** — PG reads are not yet enabled. +> Repositories: `DrizzleUserRepo`, `MongoUserRepo`, `DualWriteUserRepo` +> Postgres table: **`users`** — `backend/src/db/schema/users.ts` + +--- + +## Postgres Table: `users` + +> [!note] Source +> `backend/src/db/schema/users.ts` + +### Columns + +| Column | PG Type | Nullable | Default | Notes | +| --- | --- | --- | --- | --- | +| `id` | `uuid` | no | `gen_random_uuid()` | Primary key | +| `legacy_object_id` | `text` | yes | — | Mongo ObjectId; partial-unique index WHERE NOT NULL; used for idempotent backfill upserts | +| `email` | `varchar(255)` | yes | — | Partial-unique index WHERE NOT NULL | +| `password` | `varchar(255)` | yes | — | Hashed | +| `first_name` | `text` | yes | — | — | +| `last_name` | `text` | yes | — | — | +| `role` | `user_role` enum | no | `buyer` | Values: `admin`, `buyer`, `seller`, `resolver`, `guard` (added migration 0017) | +| `is_email_verified` | `bool` | yes | `false` | — | +| `auth_provider` | `auth_provider` enum | no | `email` | Values: `email`, `google`, `telegram` | +| `telegram_verified` | `bool` | yes | `false` | — | +| `email_verification_token` | `text` | yes | — | Legacy token flow | +| `email_verification_code` | `text` | yes | — | OTP code | +| `email_verification_code_expires` | `timestamptz` | yes | — | — | +| `password_reset_token` | `text` | yes | — | — | +| `password_reset_expires` | `timestamptz` | yes | — | — | +| `password_reset_code` | `text` | yes | — | — | +| `password_reset_code_expires` | `timestamptz` | yes | — | — | +| `profile` | `jsonb` | yes | — | Stores avatar, photoURL, phone, address, bio, website, walletAddress, walletType, walletProvider, walletProofVerified, walletProofTimestamp, isPublic | +| `preferences` | `jsonb` | yes | — | Stores language, currency, notifications.{email,sms,push} | +| `status` | `user_status` enum | yes | `active` | Values: `active`, `suspended`, `deleted` | +| `last_login_at` | `timestamptz` | yes | — | — | +| `referral_code` | `varchar(255)` | yes | — | Partial-unique index | +| `referred_by_id` | `uuid` | yes | — | Self-FK → `users(id)`; index | +| `points_total` | `int` | yes | `0` | — | +| `points_available` | `int` | yes | `0` | — | +| `points_used` | `int` | yes | `0` | — | +| `points_level` | `int` | yes | `1` | Indexed | +| `referral_stats_total` | `int` | yes | `0` | — | +| `referral_stats_active` | `int` | yes | `0` | — | +| `referral_stats_total_earned` | `int` | yes | `0` | — | +| `created_at` | `timestamptz` | no | `now()` | — | +| `updated_at` | `timestamptz` | no | `now()` | — | + +### Child Tables + +**`user_passkeys`** — WebAuthn credentials extracted from the embedded array: + +| Column | Type | Notes | +| --- | --- | --- | +| `id` | `text` (PK) | WebAuthn credential ID | +| `user_id` | `uuid FK→users CASCADE` | Owner | +| `public_key` | `text` | Stored public key | +| `counter` | `int` | Signature counter | +| `device_type` | `passkey_device_type` enum | `platform` / `cross-platform` | +| `device_name` | `text` | Optional human label | +| `created_at` | `timestamptz` | — | + +**`user_refresh_tokens`** — Active JWT refresh tokens extracted from the Mongo array: + +| Column | Type | Notes | +| --- | --- | --- | +| `token` | `text` (PK) | The refresh token string | +| `user_id` | `uuid FK→users CASCADE` | Owner | + +### Indexes (Postgres) + +| Index | Type | Condition | +| --- | --- | --- | +| `users_email_unique` | partial-unique | WHERE `email IS NOT NULL` | +| `users_referral_code_unique` | partial-unique | WHERE `referral_code IS NOT NULL` | +| `users_legacy_object_id_unique` | partial-unique | WHERE `legacy_object_id IS NOT NULL` | +| `users_role_idx` | btree | — | +| `users_status_idx` | btree | — | +| `users_auth_provider_idx` | btree | — | +| `users_referral_code_idx` | btree | — | +| `users_referred_by_id_idx` | btree | — | +| `users_points_level_idx` | btree | — | + +### Relations + +- Self-referential: `referred_by_id → users.id` (parent/children for referral tree) +- One-to-many: `user_passkeys.user_id`, `user_refresh_tokens.user_id` + +--- + +## MongoDB Collection: `User` (legacy — reads still active) > [!note] Source > `backend/src/models/User.ts:70` — schema definition @@ -20,7 +114,7 @@ The core identity document for every actor in the marketplace: buyers, sellers, > [!note] Wallet ownership proof > `PATCH /api/user/wallet-address` accepts both EVM and TON wallets. EVM addresses require an EIP-191 signature (`ethers.verifyMessage`); TON addresses are format-validated and may include an optional TonProof. A successful proof sets `profile.walletProofVerified = true` and `profile.walletProofTimestamp`. -## Schema +### Schema | Field | Type | Required | Default | Validation | Index | Description | | --- | --- | --- | --- | --- | --- | --- | @@ -28,8 +122,8 @@ The core identity document for every actor in the marketplace: buyers, sellers, | `password` | String | no | — | minlength 6 | — | Hashed password. Optional to support passkey-only, Google, and Telegram accounts. | | `firstName` | String | no | `"کاربر"` | trim | — | Persian default ("user"). | | `lastName` | String | no | `"جدید"` | trim | — | Persian default ("new"). | -| `role` | String | yes | `"buyer"` | enum: `admin` / `buyer` / `seller` / `resolver` | yes | Authorisation tier. `resolver` was added in commit `fce8a19` — can view and resolve disputes, and bypass chat membership checks, but has no other admin privileges. | -| `isEmailVerified` | Boolean | no | `false` | — | — | Set to true after the email verification code is consumed. ⚠️ Changing the email via `PUT /api/user/profile` **resets this to `false`** and dispatches a fresh **6-digit** verification code to the new address (see Email verification note below). | +| `role` | String | yes | `"buyer"` | enum: `admin` / `buyer` / `seller` / `resolver` / `guard` | yes | Authorisation tier. `resolver` (commit `fce8a19`): can view/resolve disputes and bypass chat membership checks. `guard` (migration 0017): added in PG schema; purpose TBD. | +| `isEmailVerified` | Boolean | no | `false` | — | — | Set to true after the email verification code is consumed. Warning: Changing the email via `PUT /api/user/profile` **resets this to `false`** and dispatches a fresh **6-digit** verification code to the new address. | | `authProvider` | String | yes | `"email"` | enum: `email` / `google` / `telegram` | yes | Provider used to create the account. Existing email/password accounts remain `email`; Telegram-only users are `telegram`. | | `telegramVerified` | Boolean | no | `false` | — | — | Set when Telegram identity has been signature-verified and linked through `TelegramLink`. | | `emailVerificationToken` | String | no | — | — | — | Legacy token-based email verification. | @@ -39,7 +133,7 @@ The core identity document for every actor in the marketplace: buyers, sellers, | `passwordResetExpires` | Date | no | — | — | — | Expiry of `passwordResetToken`. | | `passwordResetCode` | String | no | — | — | — | OTP reset code. | | `passwordResetCodeExpires` | Date | no | — | — | — | Expiry for OTP reset code. | -| `passkeys[]` | Subdocument array | no | `[]` | — | — | WebAuthn credentials (see below). | +| `passkeys[]` | Subdocument array | no | `[]` | — | — | WebAuthn credentials. Extracted to `user_passkeys` table in PG. | | `passkeys[].id` | String | yes | — | — | — | Credential ID. | | `passkeys[].publicKey` | String | yes | — | — | — | Stored public key. | | `passkeys[].counter` | Number | yes | `0` | — | — | Signature counter. | @@ -49,7 +143,7 @@ The core identity document for every actor in the marketplace: buyers, sellers, | `profile.avatar` | String | no | — | — | — | Avatar URL. | | `profile.photoURL` | String | no | — | — | — | Alternative photo URL. | | `profile.phone` | String | no | — | — | — | Contact phone. | -| `profile.address.street` | String | no | — | — | — | Inline address (separate from [[Address]] book). | +| `profile.address.street` | String | no | — | — | — | Inline address (separate from Address book). | | `profile.address.city` | String | no | — | — | — | — | | `profile.address.state` | String | no | — | — | — | — | | `profile.address.zipCode` | String | no | — | — | — | — | @@ -69,26 +163,26 @@ The core identity document for every actor in the marketplace: buyers, sellers, | `preferences.notifications.push` | Boolean | no | `true` | — | — | Opt-in for push notifications. | | `status` | String | no | `"active"` | enum: `active` / `suspended` / `deleted` | yes | Soft-delete and moderation flag. | | `lastLoginAt` | Date | no | — | — | — | Updated by auth middleware. | -| `refreshTokens[]` | String[] | no | `[]` | — | — | Array of currently active JWT refresh tokens. ⚠️ Reset to `[]` on password change and on password reset, which invalidates every outstanding session and forces re-login everywhere. | -| `referralCode` | String | no | — | — | unique, sparse | **Not yet implemented** in `User.ts` — planned for referral programme. | -| `referredBy` | ObjectId → User | no | — | — | yes | **Not yet implemented** in `User.ts` — planned for referral programme. | -| `points.total` | Number | no | `0` | — | — | **Not yet implemented** in `User.ts` — planned for loyalty system. | -| `points.available` | Number | no | `0` | — | — | **Not yet implemented** in `User.ts`. | -| `points.used` | Number | no | `0` | — | — | **Not yet implemented** in `User.ts`. | -| `points.level` | Number | no | `1` | — | yes (`points.level`) | **Not yet implemented** in `User.ts` — planned for [[LevelConfig]] lookup. | -| `referralStats.totalReferrals` | Number | no | `0` | — | — | **Not yet implemented** in `User.ts`. | -| `referralStats.activeReferrals` | Number | no | `0` | — | — | **Not yet implemented** in `User.ts`. | -| `referralStats.totalEarned` | Number | no | `0` | — | — | **Not yet implemented** in `User.ts`. | +| `refreshTokens[]` | String[] | no | `[]` | — | — | Array of currently active JWT refresh tokens. Warning: Reset to `[]` on password change and on password reset, invalidating every outstanding session. Extracted to `user_refresh_tokens` table in PG. | +| `referralCode` | String | no | — | — | unique, sparse | **Not yet implemented** — planned for referral programme. | +| `referredBy` | ObjectId -> User | no | — | — | yes | **Not yet implemented** — planned for referral programme. | +| `points.total` | Number | no | `0` | — | — | **Not yet implemented** — planned for loyalty system. | +| `points.available` | Number | no | `0` | — | — | **Not yet implemented.** | +| `points.used` | Number | no | `0` | — | — | **Not yet implemented.** | +| `points.level` | Number | no | `1` | — | yes | **Not yet implemented** — planned for LevelConfig lookup. | +| `referralStats.totalReferrals` | Number | no | `0` | — | — | **Not yet implemented.** | +| `referralStats.activeReferrals` | Number | no | `0` | — | — | **Not yet implemented.** | +| `referralStats.totalEarned` | Number | no | `0` | — | — | **Not yet implemented.** | | `createdAt` | Date | auto | — | — | — | Mongoose timestamp. | | `updatedAt` | Date | auto | — | — | — | Mongoose timestamp. | -## Virtuals +### Virtuals | Virtual | Returns | Definition | | --- | --- | --- | | `fullName` | `${firstName} ${lastName}` | `backend/src/models/User.ts:238` | -## Indexes +### Indexes (MongoDB) Defined explicitly: @@ -97,27 +191,44 @@ Defined explicitly: - `{ status: 1 }` — `backend/src/models/User.ts:179` - `{ authProvider: 1 }` — supports provider-level account reporting and cleanup. -> [!warning] Missing indexes -> The schema currently defines only `role` and `status` indexes. The `referralCode`, `referredBy`, and `points.level` indexes documented below are **not yet present** in `User.ts`: +> [!warning] Missing indexes in Mongo schema +> The schema currently defines only `role` and `status` indexes. The `referralCode`, `referredBy`, and `points.level` indexes documented below are **not yet present** in `User.ts`. -## Pre/Post Hooks +### Pre/Post Hooks None declared at the schema level. -## Instance Methods +### Instance Methods | Signature | Purpose | | --- | --- | | `toJSON(): object` | Strips `password`, `refreshTokens`, all `emailVerification*` and `passwordReset*` fields before serialisation. Defined at `backend/src/models/User.ts:243`. | -## Static Methods +### Static Methods None defined on the schema. +--- + +## Roles + +| Role | Added | Capabilities | +| --- | --- | --- | +| `admin` | original | Full platform access | +| `buyer` | original | Place purchase requests, confirm delivery | +| `seller` | original | Submit offers, manage shop | +| `resolver` | commit `fce8a19` | View/resolve disputes; bypass chat membership checks; no other admin privileges | +| `guard` | migration 0017 (PG only) | Purpose TBD — defined in `user_role` PG enum, not yet in Mongo schema | + +> [!warning] Role enum drift +> The Postgres `user_role` enum includes `guard`; the Mongo schema enum does not. Until the Mongo schema is updated, any `guard`-role user created through PG will not be representable in Mongo and will break dual-write for that record. + +--- + ## Relationships -- **References**: [[User]] (self, via `referredBy`). -- **Referenced by**: [[PurchaseRequest]] (`buyerId`, `preferredSellerIds`, `deliveryInfo.deliveryCodeUsedBy`, `deliveryInfo.deliveryAttempts[].sellerId`), [[SellerOffer]] (`sellerId`), [[Payment]] (`buyerId`, `sellerId`), [[Chat]] (`participants[].userId`, `messages[].senderId`, `metadata.createdBy`), [[Notification]] (`userId` as string), [[RequestTemplate]] (`sellerId`), [[Dispute]] (`buyerId`, `sellerId`, `adminId`), [[BlogPost]] (`author.id`), [[Address]] (`userId`), [[Review]] (`sellerId`, `reviewerId`), [[PointTransaction]] (`user`, `referredUser`), [[ShopSettings]] (`sellerId`). +- **References**: User (self, via `referredBy` / `referred_by_id`). +- **Referenced by**: PurchaseRequest (`buyerId`, `preferredSellerIds`, `deliveryInfo.deliveryCodeUsedBy`, `deliveryInfo.deliveryAttempts[].sellerId`), SellerOffer (`sellerId`), Payment (`buyerId`, `sellerId`), Chat (`participants[].userId`, `messages[].senderId`, `metadata.createdBy`), Notification (`userId` as string), RequestTemplate (`sellerId`), Dispute (`buyerId`, `sellerId`, `adminId`), BlogPost (`author.id`), Address (`userId`), Review (`sellerId`, `reviewerId`), PointTransaction (`user`, `referredUser`), ShopSettings (`sellerId`). ## State Transitions @@ -134,20 +245,31 @@ stateDiagram-v2 ## Common Queries ```ts -// Find by email (login) +// Mongo — Find by email (login) User.findOne({ email: email.toLowerCase() }); -// Active sellers +// Mongo — Active sellers User.find({ role: 'seller', status: 'active' }); -// Validate referral +// Mongo — Validate referral User.findOne({ referralCode: code }); -// Leaderboard by points +// Mongo — Leaderboard by points User.find({ status: 'active' }).sort({ 'points.total': -1 }).limit(10); -// Promote level +// Mongo — Promote level User.updateOne({ _id: id }, { $set: { 'points.level': newLevel } }); ``` -Related: [[TempVerification]], [[LevelConfig]], [[PointTransaction]], [[ShopSettings]]. +```sql +-- PG — Find by email +SELECT * FROM users WHERE email = lower($1) AND email IS NOT NULL; + +-- PG — Active sellers +SELECT * FROM users WHERE role = 'seller' AND status = 'active'; + +-- PG — Leaderboard by points +SELECT * FROM users WHERE status = 'active' ORDER BY points_total DESC LIMIT 10; +``` + +Related: TempVerification, LevelConfig, PointTransaction, ShopSettings. diff --git a/04 - Flows/Telegram Mini App.md b/04 - Flows/Telegram Mini App.md new file mode 100644 index 0000000..785b48a --- /dev/null +++ b/04 - Flows/Telegram Mini App.md @@ -0,0 +1,441 @@ +--- +title: Telegram Mini App Flow +tags: [flow, telegram, mini-app, auth, bilingual, RTL] +related_models: ["[[User]]"] +related_apis: ["POST /api/auth/telegram", "[[Auth API]]"] +task: "5.4" +--- + +> **Last updated:** 2026-06-03 +> **Status:** IN PROGRESS — Task 5.4 (dependencies: 5.1 auth infra, 5.2 Telegram sign-in endpoint) +> **Frontend branch:** `integrate-main-into-development` · v2.8.44 +> **Entry point:** `src/sections/telegram/` · route `/telegram` + +# Telegram Mini App Flow + +End-to-end specification for the **Amaneh Telegram Mini App** — a fully self-contained marketplace shell surfaced inside Telegram's in-app browser via the WebApp SDK. Buyers and sellers can browse requests, create new escrow requests, review offer state, follow payments, and message each other without leaving Telegram. + +--- + +## 1. Architecture Overview + +``` +Telegram Client + └─ Mini App iframe (https://amn.gg/telegram) + └─ TelegramMiniAppView ← shell orchestrator + ├─ useTelegramLiveContext ← SDK probe + polling + ├─ useTelegramLanguage ← EN / FA detection + ├─ useTelegramAutoSignIn ← silent JWT exchange + ├─ useTelegramMainButton ← native chrome sync + ├─ useTelegramBackButton ← native chrome sync + ├─ useTelegramHaptic ← haptic wrapper + │ + ├─ [state: loading] → TelegramLoadingState + ├─ [state: unsupported] → TelegramUnsupportedState + ├─ [state: unlinked] → TelegramUnlinkedState + └─ [state: linked] + ├─ TelegramHeader + ├─ TelegramTabBar (Home / Requests / Chat / Account) + │ + ├─ TelegramHomeView + ├─ TelegramRequestsView → TelegramRequestDetailView + ├─ TelegramChatView → TelegramChatThreadView + ├─ TelegramAccountView + └─ [overlay] TelegramNewRequestView +``` + +The shell is a **single-page, no-router** design: all navigation (tabs, overlays, detail drilldowns) is pure React state in `TelegramMiniAppView`. `window.location.assign` is only used as a final escape hatch to the full web dashboard. + +--- + +## 2. Launch Points + +| Entry | Mechanism | `startapp` context | +|---|---|---| +| Bot profile | User opens bot → taps "Open App" | none | +| Menu button | Pinned button in any chat with the bot | none | +| Inline button | Bot sends a card with an embedded button | `req_` | +| Direct deep link | `https://t.me/AmanehBot/app?startapp=req_` | `req_` | +| Web fallback | Browser at `/telegram` | none (unsupported state) | + +`startapp` / `tgWebAppStartParam` is read from either the WebApp SDK (`window.Telegram.WebApp`) or from URL query/hash params (for older Telegram clients that append them directly). + +--- + +## 3. SDK Initialisation & Context Probe + +**File:** `src/utils/telegram-webapp.ts` · `getTelegramContext()` + +The function assembles a `TelegramContext` object from: + +1. `window.Telegram.WebApp` — primary SDK surface (available when the app is opened inside Telegram). +2. URL query/hash fallback — `tgWebAppStartParam`, `tgWebAppData`, `tgWebAppVersion`, `tgWebAppPlatform` — used by older clients or during dev testing. + +**Fields extracted:** + +| Field | Source | Notes | +|---|---|---| +| `isMiniApp` | Any Telegram signal present | Drives unsupported vs unlinked state | +| `initData` | `webApp.initData` or `tgWebAppData` URL param | HMAC-signed payload sent to `/api/auth/telegram` | +| `initDataUnsafe` | `webApp.initDataUnsafe` | Client-side user identity (not trusted) | +| `safeArea` | `contentSafeAreaInset` or `safe_area_insets` | Parsed to `{top, right, bottom, left}` in px | +| `theme` | `webApp.themeParams` | Both camelCase and snake_case normalised | +| `platform` | `webApp.platform` or URL param | e.g. `ios`, `android`, `tdesktop` | +| `startParam` | `startapp` / `tgWebAppStartParam` / `start_param` | Deep-link context | +| `isUnsupported` | `!webApp && Boolean(startParam)` | Partial signal — no SDK but has URL param | + +**Polling on mount** (`useTelegramLiveContext`): Telegram sometimes finishes injecting the WebApp object after the first React render. The hook re-probes at 0 ms, 100 ms, 500 ms, and 1000 ms after mount, and also re-probes on `hashchange` events (triggered by the native back-button on some platforms). + +--- + +## 4. Shell State Machine + +`getTelegramStatus(context, hasWebAccount)` returns one of three states: + +``` +unsupported ─── !context.isMiniApp + (opened in browser, not Telegram) + +unlinked ─────── isMiniApp && (!user || !telegramUser.id) + (inside Telegram but no JWT session linked) + +linked ──────── isMiniApp && user && telegramUser.id + (authenticated, full shell rendered) +``` + +State transitions occur on: +- Auth session check completing (`loading → false`) +- Telegram auto sign-in completing (`tgAuthLoading → false`) +- Manual sign-in button tap (unlinked → linked) + +--- + +## 5. Authentication Flow + +### 5.1 Silent Auto Sign-In + +**Hook:** `useTelegramAutoSignIn` · **File:** `hooks/use-telegram-auto-sign-in.ts` + +On mount, if `context.isMiniApp && context.initData && !user`: + +1. Exchange `initData` for a JWT by calling `signInWithTelegram({ initData })` → `POST /api/auth/telegram`. +2. On success, call `checkUserSession()` to refresh the auth context. +3. If the backend returns `isNewUser: true`, show `TelegramOnboardingSheet`. +4. A `useRef` deduplication guard (`attemptedInitDataRef`) prevents re-runs under React Strict Mode's double-effect behaviour. + +### 5.2 Manual Sign-In (Unlinked State) + +When `initData` is present but auto sign-in failed (or hasn't run yet), `TelegramUnlinkedState` renders: +- **Continue with Telegram** — calls the same `signIn()` function from `useTelegramAutoSignIn`. +- **Sign in with email** — `window.location.assign(paths.auth.jwt.signIn)`. +- **Create an account** — `window.location.assign(paths.auth.jwt.register)`. + +When `initData` is absent (accessed via a path that skips Telegram context), only the email/register buttons appear. + +### 5.3 Backend Endpoint + +`POST /api/auth/telegram` — expects `{ initData: string }`. Backend verifies the HMAC using the Telegram bot token, extracts `user` from the payload, upserts a `User` record (`telegramId`, `telegramVerified: true`), and issues a JWT + refresh token. Returns `{ token, refreshToken, isNewUser }`. + +--- + +## 6. Navigation Model + +All navigation is in-shell React state — no Next.js router is involved. + +``` +activeTab : 'home' | 'requests' | 'chat' | 'account' +overlayScreen : 'new-request' | null +openConversationId : string | null +openRequestId : string | null +``` + +**Priority rendering** (first match wins): + +1. `openConversationId` → `TelegramChatThreadView` +2. `openRequestId` → `TelegramRequestDetailView` +3. `overlayScreen === 'new-request'` → `TelegramNewRequestView` +4. `activeTab` → appropriate tab view + +**Back button** (Telegram native `BackButton`) dismisses in reverse priority order: chat thread → request detail → overlay → returns to `home` tab. + +`BackButton` visibility: shown whenever `state === 'linked'` and either an overlay/drilldown is active, or `activeTab !== 'home'`. + +`MainButton` visibility: hidden while any overlay is open. When visible: +- **Linked** → "New Request" (opens `overlayScreen = 'new-request'`) +- **Unlinked** → "Sign In" (navigates to the JWT sign-in page) + +Both chrome buttons are styled with the amaneh saffron palette (`color: #C2410C`, `text_color: #FFFFFF`) via `setParams` (WebApp SDK >= 6.1). + +--- + +## 7. Supported Flows + +### 7.1 Browse Requests (Requests Tab) + +- `TelegramRequestsView` fetches the user's purchase requests via `useTelegramMyRequests` (GET `/api/purchase-requests/my`). +- Displays a skeleton loader, then a scrollable list of `TelegramRequestRow` items. +- Each row shows: title, status chip, budget, creation date. +- Tap → sets `openRequestId` → renders `TelegramRequestDetailView`. + +### 7.2 Request Detail with Stepper + +- `TelegramRequestDetailView` fetches a single request via `useTelegramRequest`. +- Renders `TelegramRequestStepper` — a visual timeline of the escrow status flow from `pending_payment` → `completed`. +- `determineCurrentStepFromStatus` maps the current `status` to a step index. +- Also renders: budget, description, creation date, category, urgency. +- Dates formatted via `toLocaleDateString` with `fa-IR` locale for Persian. + +### 7.3 Create New Request + +- `TelegramNewRequestView` is a full-screen overlay (not a routed page). +- Form fields: title, description, category (fetched from `/api/categories`), budget min/max, urgency. +- On submit: calls `createPurchaseRequest()` → POST `/api/purchase-requests`. +- On success: closes overlay, switches `activeTab` to `'requests'`. +- `MainButton` is hidden while the overlay is open (submit lives in the form itself). + +### 7.4 Chat + +- `TelegramChatView` shows the user's active conversations via `useTelegramConversations`. +- Tap a row → sets `openConversationId` → renders `TelegramChatThreadView`. +- `TelegramChatThreadView` loads messages via `useTelegramChatThread`, renders `TelegramChatBubble` items, and includes `TelegramChatComposer` for sending. +- Optimistic send: message appears immediately, confirmed/rolled back on API response. + +### 7.5 Account + +- `TelegramAccountView` shows profile info (name, email, Telegram username, `telegramVerified` status), linked wallet (if any), and notification preferences. +- Contains sign-out action and language toggle. + +--- + +## 8. Bilingual Support (EN / FA) + +**Language detection priority** (`useTelegramLanguage`): + +1. `localStorage` key `amn_tg_lang` — user's persisted manual selection. +2. `initDataUnsafe.user.language_code` — Telegram-reported language (`"fa"` or `"fa-IR"` → Persian). +3. Fallback → English. + +**Language toggle:** `TelegramLanguageToggle` in the header — two buttons `[ EN | فا ]`. On tap: haptic light + language switch + persist to `localStorage`. + +**RTL layout:** + +| Element | EN (LTR) | FA (RTL) | +|---|---|---| +| Root `dir` attribute | `ltr` | `rtl` | +| Font family | IBM Plex Sans | Vazirmatn | +| Arrow icons | `→` | `←` | +| Text alignment | left | right (inherits from `dir`) | +| Chip list wrap | left-to-right | right-to-left | + +Font size bumps for Persian: body 13 px → 14 px, labels 10 px → 11 px (Vazirmatn renders optically smaller). + +**Translation structure:** + +```ts +// src/sections/telegram/locales/en.ts + fa.ts +const TR = { + en: { loading, unsupported, unlinked, header, home, requests, + chat, account, newRequest, tabs, main, onboarding, errors, displayName, dir }, + fa: { /* same keys, Farsi strings, dir: 'rtl' */ }, +}; +``` + +All JSX uses `t.
.` — no inline strings in components. + +--- + +## 9. Design System + +**File:** `src/sections/telegram/constants.ts` · `src/sections/telegram/telegram-shell-css.ts` + +The Mini App has a distinct visual identity (cream/saffron Persian palette) that does not inherit from the main dashboard theme. All tokens are feature-scoped. + +**Palette:** `TG_PALETTE` + +| Token | Hex | Usage | +|---|---|---| +| `cream50` | `#FBF6EB` | Page background | +| `ink900` | `#1C1410` | Primary text | +| `ink600` | `#6B5D4E` | Secondary text / labels | +| `saffron600` | `#C2410C` | Primary action, MainButton | +| `saffron500` | `#D97757` | Hover states | +| `pistachio700` | `#3D6B4F` | Success / released states | +| `pomegranate700` | `#8E2424` | Error / disputed states | +| `bgPage` | `#E7DFCB` | Shell outer background | + +**Fonts:** `TG_FONTS` — Source Serif 4 (headings), IBM Plex Sans (body LTR), Vazirmatn (body RTL), IBM Plex Mono (amounts/addresses). + +**CSS:** `buildTelegramShellCss()` injects a `