docs: update PG migration status, data models, architecture + add Telegram Mini App flow (v2.8.59)

- Postgres Runtime Cutover Status: 17 migrations (0000–0017), dual-write repo matrix - Backend Architecture: dual-DB architecture, repo factory, MONGO_CONNECT_MODE modes - Data Model Overview: 23-model index with PG table names and migration status - User, PurchaseRequest, SellerOffer, Chat, Dispute: Drizzle schema + cutover status added - 04 - Flows/Telegram Mini App.md: new doc covering Mini App architecture and flows - mongo-to-pg-migration-prd.md: status block prepended with 2026-06-03 milestone tracking Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-03 10:29:48 +04:00
parent 6f13903644
commit d072238fe8
10 changed files with 1998 additions and 171 deletions
--- a/Architecture/Backend
+++ b/Architecture/Backend
@@ -2,14 +2,15 @@
 title: Backend Architecture
 tags: [architecture, backend]
 created: 2026-05-23
+updated: 2026-06-03
 ---

 # Backend Architecture

-Module-level architecture of the Express 5 + TypeScript backend. MongoDB/Mongoose is still the primary runtime persistence layer; the `integrate-main-into-development` backend also contains the Drizzle/Postgres migration layer.
+Module-level architecture of the Express 5 + TypeScript backend. The system is mid-migration: MongoDB/Mongoose remains the authoritative read store for most domains, with PostgreSQL (Drizzle ORM) running in dual-write mode across 17 landed migrations. The repository factory pattern (`src/db/repositories/factory.ts`) controls which backend each domain reads and writes through env flags.

 > [!info]
-> Repo: `git@git.manko.yoga:222/nick/backend.git` · Active integration branch: `integrate-main-into-development` · Current baseline: backend `2.6.79` at `3a50dc4`
+> Repo: `git@git.manko.yoga:222/nick/backend.git` · Current version: `2.8.56` · 17 Drizzle migrations landed · Dual-write active across all major domains

 ---

@@ -24,10 +25,13 @@ backend/src/
 │   ├── database/               # Mongoose connection, retries, graceful shutdown
 │   └── socket/socketService.ts # Socket.IO server, rooms, emit helpers
 ├── models/                     # Mongoose models — see 02 - Data Models/
-├── db/                         # Drizzle/Postgres migration layer: schemas, migrations, repos, backfill, verify
+├── db/                         # Drizzle/Postgres layer: schemas, migrations, repos, backfill, verify
+│   ├── schema/                 # Per-table Drizzle schema files + index.ts barrel
+│   ├── migrations/             # 17 numbered SQL migration files
+│   └── repositories/           # Drizzle repos, dual-write wrappers, factory.ts
 ├── routes/                     # Express Router definitions (mounted in app.ts)
 ├── scripts/                    # CLI utilities (seed:users, seed:categories, ...)
-├── seeds/                      # Seed data fixtures
+├── seeds/                      # Seed data fixtures (Postgres-capable as of v2.8.47)
 ├── services/
 │   ├── ai/                     # OpenAI integration (descriptions, moderation)
 │   ├── auth/                   # JWT, OAuth, Passkey, password reset
@@ -61,8 +65,8 @@ backend/src/
 └── utils/                      # Pure utility fns (logger, currencyUtils, etc.)
 ```

-> [!warning] Postgres is not the default runtime store yet
-> `src/db/repositories/factory.ts` can select `mongo`, `dual`, or `pg` implementations for user, payment, points, and marketplace domains, but the broad service layer still imports Mongoose models directly. A code scan on 2026-05-31 found no runtime calls to `createRepositories()` / `getPaymentRepo()` / `getMarketplaceRepo()` outside the factory itself. See [[Postgres Runtime Cutover Status]] before assuming a `REPO_*` flag changes live behavior.
+> [!warning] Reads still go to Mongo for all dual-write domains
+> Even when `REPO_*=dual`, Mongo is the authoritative read source. The dual-write seam exists and is exercised in production, but no domain has been cut over to PG reads yet. See [[Postgres Runtime Cutover Status]] before assuming a `REPO_*` flag changes live read behavior.

 > [!tip]
 > Service folders are self-contained: each typically has `<feature>Service.ts`, `<feature>Controller.ts`, `<feature>Routes.ts`, `<feature>Validation.ts`. This makes each service movable to a microservice later with minimal coupling.
@@ -75,22 +79,25 @@ The bootstrap is intentionally linear and easy to audit. Execution order:

 1. **Imports & env load** — `dotenv` (if used), then `import { config } from './shared/config'`.
 2. **Express app construction** — `const app = express();`
-3. **Trust proxy** — `app.set('trust proxy', config.trustProxy)` so X-Forwarded-For works behind Nginx.
+3. **Trust proxy** — `app.set('trust proxy', config.trustProxy)` so X-Forwarded-For works behind Traefik.
 4. **Security headers** — `app.use(helmet({ ... }))`.
 5. **CORS** — `cors({ origin: config.frontendUrl, credentials: true, methods: [...] })`.
 6. **Body parsers** — `express.json({ limit: '10mb' })`, `express.urlencoded({ extended: true })`.
 7. **Static uploads** — `app.use('/uploads', express.static(uploadDir))`.
-8. **Health endpoint** — `GET /health` for Docker healthcheck and external monitors.
+8. **Health endpoint** — `GET /health` for Docker healthcheck and external monitors. Now surfaces active Postgres store modes.
 9. **Route mounting** — every `/api/*` route registered before the error handler.
 10. **404 handler** — catches unmatched `/api/*`.
 11. **Error handler** — central `errorHandler` middleware formats responses via `response-handler.ts`.
 12. **HTTP server creation** — `const server = http.createServer(app)`.
 13. **Socket.IO attach** — `initSocket(server, corsOptions)` (see [[Real-time Layer]]).
-14. **DB connect** — `await connectDatabase()` for MongoDB/Mongoose. Postgres connects lazily only when PG modules are imported (for example oracle quote persistence with `ORACLE_QUOTING_ENABLED=true`) and requires `PG_URL`.
+14. **DB connect** — controlled by `MONGO_CONNECT_MODE`:
+    - `always` (default) — connects Mongoose (Mongo) and PostgreSQL (via `PG_URL`) on boot.
+    - `never` — skips Mongo entirely; Postgres is the only persistence layer. Seeds are Postgres-capable in this mode.
+    - `optional` — connects Postgres; Mongo is attempted but failures are non-fatal.
 15. **Redis connect** — `await connectRedis()`.
 16. **Listen** — `server.listen(config.port, ...)`.
 17. **Graceful shutdown** — SIGTERM/SIGINT handlers close server, drain sockets, close Mongoose, close Redis.
-18. **Optional dev seeding** — when `NODE_ENV === 'development'` and `SEED_USERS !== 'false'`, the bootstrap calls the seed scripts to provision default test users.
+18. **Optional dev seeding** — when `NODE_ENV === 'development'` and `SEED_USERS !== 'false'`, the bootstrap calls the seed scripts to provision default test users. Seeds are store-aware and run correctly against both Mongo and PG.

 ---

@@ -104,14 +111,14 @@ The bootstrap is intentionally linear and easy to audit. Execution order:
 | 4 | `morgan` (dev only) | global | HTTP request log to stdout. |
 | 5 | `requestId` | global | Adds `X-Request-Id` for log correlation. |
 | 6 | `authMiddleware` | per-route | Verifies JWT, attaches `req.user`. Mounted only on protected routes. |
-| 7 | `roleGuard('admin'|'seller'|...)` | per-route | RBAC check after auth. |
+| 7 | `roleGuard('admin'\|'seller'\|'guard'\|...)` | per-route | RBAC check after auth. Roles: `admin`, `buyer`, `seller`, `resolver`, `guard`. |
 | 8 | `validate(schema)` | per-route | express-validator + zod inputs. |
 | 9 | `controllerFn` | per-route | Delegates to service layer. |
 | 10 | `notFound` | tail | Returns 404 envelope for unmatched routes. |
 | 11 | `errorHandler` | tail | Catches thrown errors, formats response. |

 > [!note]
-> Rate-limit middleware is **active** as of 2026-05-24: auth 10 req/15 min, payment 30/15 min, AI 20/15 min, global 100/15 min. Request Network and Telegram webhooks are exempt from the global limiter. Counters are in-memory — a Redis adapter is planned for distributed deployments.
+> Rate-limit middleware is **active**: auth 10 req/15 min, payment 30/15 min, AI 20/15 min, global 100/15 min. `GET /api/payment/:id` is exempt from the payment limiter (polling route). Request Network and Telegram webhooks are exempt from the global limiter. Counters are in-memory — a Redis adapter is planned for distributed deployments.

 ---

@@ -128,7 +135,7 @@ The full route table mounted by `app.ts`:
 | `/api/marketplace/offers` | `services/marketplace/controllerRoutes.ts` | JWT (seller) | SellerOffer CRUD |
 | `/api/marketplace/templates` | `services/marketplace/controllerRoutes.ts` | JWT (seller) | RequestTemplate CRUD |
 | `/api/marketplace/categories` | `services/marketplace/controllerRoutes.ts` | public read | Category list |
-| `/api/marketplace/shop-settings` | `services/marketplace/shopSettingsController.ts` | JWT (seller) | Shop profile |
+| `/api/marketplace/shop-settings` | `services/marketplace/shopSettingsController.ts` | JWT (seller) | Shop profile; lookup tolerant of uuid/legacy id formats |
 | `/api/payment` | `services/payment/paymentControllerRoutes.ts` + `paymentRoutes.ts` | JWT | Payment CRUD, health, export |
 | `/api/payment/decentralized` | `services/payment/decentralizedPaymentRoutes.ts` | mixed | Legacy/manual Web3 save, verify, receiver |
 | `/api/payment/request-network` | `services/payment/requestNetwork/requestNetworkRoutes.ts` | mixed + HMAC sig on webhook | Request Network pay-in creation, in-house checkout rehydrate, webhooks |
@@ -136,12 +143,12 @@ The full route table mounted by `app.ts`:
 | `/api/admin/rn/networks` | `services/payment/requestNetwork/networkRegistryRoutes.ts` | JWT (admin) | Supported RN chain/token registry |
 | `/api/admin/settings/confirmation-thresholds` | `services/admin/confirmationThresholdRoutes.ts` | JWT (admin) | Runtime min-confirmation thresholds |
 | `/api/admin/payments/awaiting-confirmation` | `services/admin/awaitingConfirmationRoutes.ts` | JWT (admin) | Payments blocked on safety confirmations |
-| `/api/telegram` | `services/telegram/telegramRoutes.ts` | mixed (some JWT, webhook uses secret-token) | Mini App verify/session, identity link/unlink, bot webhook |
+| `/api/telegram` | `services/telegram/telegramRoutes.ts` | mixed (some JWT, webhook uses secret-token) | Mini App verify/session, identity link/unlink, bot webhook; notifications delivered via Telegram as of v2.8.56 |
 | `/api/chat` | `services/chat/chatRoutes.ts` | JWT | Conversations, messages |
 | `/api/notification` | `services/notification/notificationRoutes.ts` + `notificationControllerRouter` | JWT | List, mark read |
 | `/api/disputes` | `routes/disputeRoutes.ts` + `services/dispute/disputeRoutes.ts` | JWT | Dispute CRUD plus release-hold helpers |
 | `/api/blog` | `services/blog/blogRoutes.ts` | mixed | Public reads, admin writes |
-| `/api/admin/cleanup` | `services/admin/dataCleanupRoutes.ts` | JWT (admin) | Data cleanup operations |
+| `/api/admin/cleanup` | `services/admin/dataCleanupRoutes.ts` | JWT (admin) | Data cleanup; scoped by provider to avoid wiping RN/multi-seller records |
 | `/api/points` | `services/points/pointsRoutes.ts` | JWT | Points, levels, referrals |
 | `/api/ai` | `services/ai/aiRoutes.ts` | JWT | OpenAI-backed helpers |
 | `/api/files` | `services/file/fileRoutes.ts` | JWT | Multipart upload |
@@ -209,10 +216,11 @@ flowchart TB
    points -.-> notify
    notify --> socket
    notify --> email
+    notify --> telegram
 ```

 > [!note]
-> `socket` and `email` are leaf services — every notification path funnels through them. Mocking these two in tests covers most side-effect verification.
+> `socket`, `email`, and `telegram` are leaf notification sinks — every notification path funnels through them. Mocking these three in tests covers most side-effect verification. Telegram notification delivery was added in v2.8.56.

 ---

@@ -254,12 +262,26 @@ Full table in [[Environment Variables]]. Critical ones:
 | Key | Default | Notes |
 |---|---|---|
 | `PORT` | `5001` | Listen port |
-| `MONGODB_URI` | `mongodb://localhost:27017/nickapp` | Includes db name |
+| `MONGODB_URI` | `mongodb://localhost:27017/nickapp` | Includes db name; not required when `MONGO_CONNECT_MODE=never` |
+| `MONGO_CONNECT_MODE` | `always` | `always` \| `never` \| `optional` — controls whether Mongoose connects on boot |
+| `PG_URL` | required for PG | PostgreSQL connection string for Drizzle; required when any `REPO_*=pg\|dual` |
 | `REDIS_URI` | `redis://localhost:6379` | + `REDIS_PASSWORD` |
 | `JWT_SECRET` | required | ≥32 chars |
 | `JWT_EXPIRES_IN` | `7d` | |
 | `REFRESH_TOKEN_EXPIRES_IN` | `30d` | |
 | `FRONTEND_URL` | `http://localhost:3000` | CORS origin |
+| `REPO_DEFAULT` | `mongo` | Global fallback store mode for all domains (`mongo` \| `dual` \| `pg`) |
+| `REPO_USER` | inherits `REPO_DEFAULT` | Per-domain override for user store |
+| `REPO_PAYMENT` | inherits `REPO_DEFAULT` | Per-domain override for payment store |
+| `REPO_POINTS` | inherits `REPO_DEFAULT` | Per-domain override for points store |
+| `REPO_MARKETPLACE` | inherits `REPO_DEFAULT` | Per-domain override for marketplace store |
+| `REPO_TREZOR` | inherits `REPO_DEFAULT` | Per-domain override for trezor store |
+| `REPO_DERIVED_DESTINATION` | inherits `REPO_DEFAULT` | Per-domain override for derived destination store |
+| `REPO_BLOG` \| `BLOG_STORE` | inherits `REPO_DEFAULT` | Per-domain override for blog store |
+| `REPO_NOTIFICATION` \| `NOTIFICATION_STORE` | inherits `REPO_DEFAULT` | Per-domain override for notification store |
+| `REPO_DISPUTE` \| `DISPUTE_STORE` | inherits `REPO_DEFAULT` | Per-domain override for dispute store |
+| `REPO_CHAT` \| `CHAT_STORE` | inherits `REPO_DEFAULT` | Chat dual-write not implemented; `dual` silently uses Mongo |
+| `REPO_RELEASE_HOLD` \| `RELEASE_HOLD_STORE` | inherits `REPO_DEFAULT` | Release-hold dual-write not implemented; `dual` silently uses Mongo |
 | `REQUEST_NETWORK_API_BASE_URL` | `https://api.request.network` | Request Network API |
 | `REQUEST_NETWORK_API_KEY` | required | Request Network API credential |
 | `REQUEST_NETWORK_WEBHOOK_SECRET` | required | Webhook HMAC key |
@@ -268,16 +290,77 @@ Full table in [[Environment Variables]]. Critical ones:
 | `DERIVED_DESTINATION_SWEEP_SIGNER` | `build-only` | Target hardware/Safe-backed signer |
 | `SMTP_*` | required | Nodemailer |
 | `OPENAI_API_KEY` | required | |
+| `ORACLE_QUOTING_ENABLED` | `false` | Enables oracle-based depeg-protected payment quotes; requires `PG_URL` |

 ---

 ## 9. Database & connection management

- **Mongoose** is the ODM. Connection in `src/infrastructure/database/`.
+The backend runs a **dual-database architecture** during the Mongo→Postgres migration. Both stores may be active simultaneously; which one serves each domain is controlled by `REPO_*` env flags.
+
+### MongoDB / Mongoose
+
+- ODM: Mongoose. Connection in `src/infrastructure/database/`.
 - Connection options enable retryable writes, exponential backoff on reconnect.
- Indexes are defined on each model and auto-created on connect (Mongoose `autoIndex: true` in dev, recommend `false` in prod with explicit migration).
+- Indexes defined on each model and auto-created on connect (`autoIndex: true` in dev; recommend `false` in prod with explicit migration scripts).
+- Remains the **authoritative read store** for all dual-write domains until read cutover is explicitly executed per domain.
 - See [[Data Model Overview]] for the relational map and per-model docs.

+### PostgreSQL / Drizzle
+
+- ORM: Drizzle. Schemas in `src/db/schema/`, migrations in `src/db/migrations/` (17 migrations landed as of 2026-06-03).
+- Managed via `drizzle-kit migrate` — never edit migration files manually.
+- Connects lazily when any PG-capable store is imported, or eagerly on boot when `MONGO_CONNECT_MODE=never`.
+- Every migrated table carries a `legacy_object_id text` column with a partial-unique index for idempotent backfill upserts.
+- Money columns use `numeric(38,18)` (except `seller_offers`: `numeric(18,8)`). Blockchain balance columns use `numeric(78,0)` to hold uint256 without overflow.
+- See [[Drizzle Schema Reference]] for the full per-table breakdown.
+
+### Repository factory — `src/db/repositories/factory.ts`
+
+The factory is the single routing layer between service code and the underlying store. It exposes per-domain getters and resolves the mode (`mongo` | `dual` | `pg`) in this order:
+
+1. Per-domain env flag (e.g. `REPO_PAYMENT`)
+2. `REPO_DEFAULT` (global staging-wide fallback)
+3. Hardcoded default: `mongo`
+
+Unrecognized values silently fall back to `mongo` — intentional safety net against typos on money writes.
+
+| Domain | Getter | Dual-write | PG-only |
+|---|---|---|---|
+| user | `getUserRepo` | Yes (full trio) | Yes |
+| payment | `getPaymentRepo` | Yes (full trio) | Yes |
+| points | `getPointsRepo` | Yes (full trio) | Yes |
+| marketplace | `getMarketplaceRepo` | Yes (full trio) | Yes |
+| trezor | `getTrezorRepo` | Yes (full trio) | Yes |
+| derivedDestination | `getDerivedDestinationRepo` | Yes (full trio) | Yes |
+| blog | `getBlogRepo` | Yes (full trio) | Yes |
+| notification | `getNotificationRepo` | Yes (full trio) | Yes |
+| dispute | `getDisputeRepo` | Yes (full trio) | Yes |
+| releaseHold | `getReleaseHoldRepo` | No — `dual` silently uses Mongo | Yes |
+| chat | `getChatRepo` | No — `dual` silently uses Mongo | Yes |
+
+> [!warning] `MONGO_CONNECT_MODE` is not handled by the factory
+> `MONGO_CONNECT_MODE` is consumed by the Mongoose connection module, not by `factory.ts`. The factory only reads `REPO_*` flags. These two controls are orthogonal: `MONGO_CONNECT_MODE=never` prevents Mongoose from connecting, while `REPO_*=pg` prevents the factory from routing to Mongo. For a full PG-only boot, set **both**.
+
+### Migration phase status (as of 2026-06-03)
+
+| Phase | Status |
+|---|---|
+| Schema / migrations | Done — 17 migrations landed, all domain tables exist in PG |
+| Dual-write seam | Done — active for all major domains via factory |
+| Backfill tooling | Done — backfill + verification harness in `src/db/` |
+| Reads cutover | Not started — all reads still served from Mongo |
+| Chat normalization | Blocked — Chat stored as JSONB blobs; normalization required before PG read cutover |
+| Mongo retirement | Future — blocked on per-domain read cutover completion |
+
+### Infrastructure / bridge tables (PG-only)
+
+- **`id_map`** — ObjectId → UUID bridge; every migrated entity upserts here during backfill/dual-write.
+- **`pg_dualwrite_gaps`** — Append-only reconciliation log for failed PG dual-writes; includes severity, resolver notes, and error stack.
+- **`payment_quotes`** — Oracle-based depeg-protected quote snapshots (1:1 with payments); PG-only, no Mongo equivalent. Only active when `ORACLE_QUOTING_ENABLED=true`.
+
+### Redis
+
 Redis client (in `src/services/redis/`) provides:
 - Session caching (login attempts, lockout counters)
 - Rate-limit counters (when middleware is enabled)
@@ -323,6 +406,8 @@ Run with `npm run test:all`. CI runs the same. Reach for `npm run test:models`,
 | `src/shared/utils/response-handler.ts` | Standard response shape |
 | `src/shared/middleware/auth.ts` | JWT verify + RBAC |
 | `src/infrastructure/socket/socketService.ts` | All socket plumbing |
+| `src/db/repositories/factory.ts` | Store routing — which backend each domain uses |
+| `src/db/schema/index.ts` | Drizzle schema barrel — all 25+ PG tables |
 | `src/services/payment/requestNetwork/requestNetworkRoutes.ts` | Request Network checkout and webhook route |
 | `src/services/payment/ledger/fundsLedgerService.ts` | Immutable payment ledger writes |
 | `src/services/marketplace/PurchaseRequestService.ts` | Core marketplace state machine |
@@ -338,5 +423,7 @@ Run with `npm run test:all`. CI runs the same. Reach for `npm run test:models`,
 - [[Frontend Architecture]] — how the FE talks to this BE
 - [[Real-time Layer]] — Socket.IO room model
 - [[Security Architecture]] — JWT, passkeys, webhook HMAC
- [[Data Model Overview]] — entity-relationship map
+- [[Data Model Overview]] — entity-relationship map (Mongoose)
+- [[Drizzle Schema Reference]] — PostgreSQL table definitions, enums, migration status
+- [[Postgres Runtime Cutover Status]] — per-domain read cutover tracker
 - [[Authentication Flow]] · [[Escrow Flow]] · [[Dispute Flow]]