nick/nick-doc
3
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs: sync documentation with latest codebase state

Browse Source 
- Update Activity Log with 108 missing commits (48 backend + 60 frontend)
- Update version references: backend v2.8.79, frontend v2.8.94
- Update migration count: 18 migrations (0000-0017)
- Update Telegram Mini App Flow to v2.8.94
- Update Payment Flow - Scanner to 2026-06-05
- Update all architectural and database references

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
This commit is contained in:
Siavash Sameni
2026-06-05 07:34:49 +04:00
parent 9dcdb420fc
commit a5d71bcc05
10 changed files with 490 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
01 - Architecture/Backend Architecture.md
View File

@@ -7,10 +7,10 @@ updated: 2026-06-03
# Backend Architecture # Backend Architecture
Module-level architecture of the Express 5 + TypeScript backend. The system is mid-migration: MongoDB/Mongoose remains the authoritative read store for most domains, with PostgreSQL (Drizzle ORM) running in dual-write mode across 17 landed migrations. The repository factory pattern (`src/db/repositories/factory.ts`) controls which backend each domain reads and writes through env flags. Module-level architecture of the Express 5 + TypeScript backend. The system is mid-migration: MongoDB/Mongoose remains the authoritative read store for most domains, with PostgreSQL (Drizzle ORM) running in dual-write mode across 18 landed migrations (0000–0017). The repository factory pattern (`src/db/repositories/factory.ts`) controls which backend each domain reads and writes through env flags.
> [!info] > [!info]
> Repo: `git@git.manko.yoga:222/nick/backend.git` · Current version: `2.8.56` · 17 Drizzle migrations landed · Dual-write active across all major domains > Repo: `git@git.manko.yoga:222/nick/backend.git` · Current version: `2.8.79` · 18 Drizzle migrations landed · Dual-write active across all major domains
--- ---
@@ -27,7 +27,7 @@ backend/src/
├── models/ # Mongoose models — see 02 - Data Models/ ├── models/ # Mongoose models — see 02 - Data Models/
├── db/ # Drizzle/Postgres layer: schemas, migrations, repos, backfill, verify ├── db/ # Drizzle/Postgres layer: schemas, migrations, repos, backfill, verify
│ ├── schema/ # Per-table Drizzle schema files + index.ts barrel │ ├── schema/ # Per-table Drizzle schema files + index.ts barrel
│ ├── migrations/ # 17 numbered SQL migration files │ ├── migrations/ # 18 numbered SQL migration files (0000–0017)
│ └── repositories/ # Drizzle repos, dual-write wrappers, factory.ts │ └── repositories/ # Drizzle repos, dual-write wrappers, factory.ts
├── routes/ # Express Router definitions (mounted in app.ts) ├── routes/ # Express Router definitions (mounted in app.ts)
├── scripts/ # CLI utilities (seed:users, seed:categories, ...) ├── scripts/ # CLI utilities (seed:users, seed:categories, ...)
@@ -308,7 +308,7 @@ The backend runs a **dual-database architecture** during the Mongo→Postgres mi
### PostgreSQL / Drizzle ### PostgreSQL / Drizzle
- ORM: Drizzle. Schemas in `src/db/schema/`, migrations in `src/db/migrations/` (17 migrations landed as of 2026-06-03). - ORM: Drizzle. Schemas in `src/db/schema/`, migrations in `src/db/migrations/` (18 migrations landed: 0000–0017 as of 2026-06-05).
- Managed via `drizzle-kit migrate` — never edit migration files manually. - Managed via `drizzle-kit migrate` — never edit migration files manually.
- Connects lazily when any PG-capable store is imported, or eagerly on boot when `MONGO_CONNECT_MODE=never`. - Connects lazily when any PG-capable store is imported, or eagerly on boot when `MONGO_CONNECT_MODE=never`.
- Every migrated table carries a `legacy_object_id text` column with a partial-unique index for idempotent backfill upserts. - Every migrated table carries a `legacy_object_id text` column with a partial-unique index for idempotent backfill upserts.
@@ -346,7 +346,7 @@ Unrecognized values silently fall back to `mongo` — intentional safety net aga
| Phase | Status | | Phase | Status |
|---|---| |---|---|
| Schema / migrations | Done — 17 migrations landed, all domain tables exist in PG | | Schema / migrations | Done — 18 migrations landed (0000–0017), all domain tables exist in PG |
| Dual-write seam | Done — active for all major domains via factory | | Dual-write seam | Done — active for all major domains via factory |
| Backfill tooling | Done — backfill + verification harness in `src/db/` | | Backfill tooling | Done — backfill + verification harness in `src/db/` |
| Reads cutover | Not started — all reads still served from Mongo | | Reads cutover | Not started — all reads still served from Mongo |

2
01 - Architecture/Frontend Architecture.md
View File

@@ -10,7 +10,7 @@ updated: 2026-06-03
Module-level architecture of the Next.js 16 (App Router) + TypeScript + MUI v9 frontend. The current integration worktree observed locally is on `integrate-main-into-development`. Module-level architecture of the Next.js 16 (App Router) + TypeScript + MUI v9 frontend. The current integration worktree observed locally is on `integrate-main-into-development`.
> [!info] > [!info]
> Repo: `git@git.manko.yoga:222/nick/frontend.git` · Active integration branch observed locally: `integrate-main-into-development` · Version: 2.8.59 (`package.json`) · Dev port `3000`, Docker port `8083`. > Repo: `git@git.manko.yoga:222/nick/frontend.git` · Active integration branch observed locally: `integrate-main-into-development` · Version: 2.8.94 (`package.json`) · Dev port `3000`, Docker port `8083`.
--- ---

25
02 - Data Models/Postgres Runtime Cutover Status.md
View File

@@ -4,14 +4,14 @@ tags: [data-model, postgres, migration, runtime-status]
aliases: [Postgres Status, PG Cutover Status, Mongo vs Postgres Runtime] aliases: [Postgres Status, PG Cutover Status, Mongo vs Postgres Runtime]
created: 2026-05-31 created: 2026-05-31
updated: 2026-06-03 updated: 2026-06-03
source: backend integrate-main-into-development@14d164c + deployment main@8764fdf source: backend integrate-main-into-development@41087c7 + deployment main@8764fdf
--- ---
# Postgres Runtime Cutover Status # Postgres Runtime Cutover Status
> **Current branch:** backend `integrate-main-into-development` at `14d164c`, version `2.8.56`; dev deployment `main` at `8764fdf`. > **Current branch:** backend `integrate-main-into-development` at `41087c7`, version `2.8.79`; dev deployment `main` at `8764fdf`.
> >
> **Bottom line:** the codebase is in **active dual-write phase**. All 11 repository domains in the factory now have Drizzle schemas, Drizzle repos, and dual-write wrappers (except Chat and ReleaseHold, which have Drizzle repos but no dual-write counterpart). 18 Drizzle migrations (0000–0017) have landed, covering every table in scope. Dev deployment defaults nine PG-capable stores to Postgres: auth-owned users/Telegram auth, confirmation-threshold config/history, user addresses, categories, level config, shop settings, reviews, notifications, and the oracle payment_quotes path. Code-level defaults remain `mongo` outside those deployment overrides. Repository factory normalizes `postgres` and `pg` as equivalent mode tokens. The unmounted legacy marketplace router is detached. As of `2.8.54–2.8.56`, the `guard` user role is in PG schema, chat routes are fixed, notifications deliver in real time, and PG response serialization/id resolution in marketplace is corrected. Reads are still Mongo-authoritative across all dual-write domains — read cutover is the remaining gate for each domain. Chat normalization (participants/messages stored as JSONB blobs, not relational child tables) remains an open blocker for full Chat cutover. > **Bottom line:** the codebase is in **active dual-write phase**. All 11 repository domains in the factory now have Drizzle schemas, Drizzle repos, and dual-write wrappers (except Chat and ReleaseHold, which have Drizzle repos but no dual-write counterpart). 18 Drizzle migrations (0000–0017) have landed, covering every table in scope. Dev deployment defaults nine PG-capable stores to Postgres: auth-owned users/Telegram auth, confirmation-threshold config/history, user addresses, categories, level config, shop settings, reviews, notifications, and the oracle payment_quotes path. Code-level defaults remain `mongo` outside those deployment overrides. Repository factory normalizes `postgres` and `pg` as equivalent mode tokens. The unmounted legacy marketplace router is detached. As of `2.8.79`, the `guard` user role is in PG schema, chat routes are fixed, notifications deliver in real time, PG response serialization/id resolution in marketplace is corrected, and seller shop lookup is tolerant of uuid/legacy id formats. Reads are still Mongo-authoritative across all dual-write domains — read cutover is the remaining gate for each domain. Chat normalization (participants/messages stored as JSONB blobs, not relational child tables) remains an open blocker for full Chat cutover.
## Schema and Repository Coverage ## Schema and Repository Coverage
@@ -158,12 +158,29 @@ The backend code defaults every store flag below to `mongo`. Dev deployment over
Estimated overall: **schema and infrastructure phase complete; write-seam phase substantially complete; read cutover and backfill execution remain for every domain.** Estimated overall: **schema and infrastructure phase complete; write-seam phase substantially complete; read cutover and backfill execution remain for every domain.**
## Recent Progress Since Last Update (2.8.37 → 2.8.56) ## Recent Progress Since Last Update (2.8.37 → 2.8.79)
- **2.8.38–2.8.46:** Complete dual-write repos for all remaining domains; Drizzle migrations pipeline finalized; TTL scheduler added; shop lookup bug-fixed. - **2.8.38–2.8.46:** Complete dual-write repos for all remaining domains; Drizzle migrations pipeline finalized; TTL scheduler added; shop lookup bug-fixed.
- **2.8.47:** Seeds made Postgres-capable and idempotent for PG-only boot (`MONGO_CONNECT_MODE=never`). - **2.8.47:** Seeds made Postgres-capable and idempotent for PG-only boot (`MONGO_CONNECT_MODE=never`).
- **2.8.48–2.8.49:** Fresh-DB PG migrate + seed path corrected; 0013/0014 migrations made valid for a fresh `drizzle-kit migrate` run. - **2.8.48–2.8.49:** Fresh-DB PG migrate + seed path corrected; 0013/0014 migrations made valid for a fresh `drizzle-kit migrate` run.
- **2.8.50:** Admin user counts routed through postgres-capable stores; admin user management works end-to-end under PG. - **2.8.50:** Admin user counts routed through postgres-capable stores; admin user management works end-to-end under PG.
- **2.8.51–2.8.53:** PG response serialization and id resolution in marketplace purchase-request paths corrected; user creation and purchase request unblocked.
- **2.8.54:** Guard user role added across auth and user management; migration 0017 adds guard to user_role enum.
- **2.8.55:** Chat routes fixed and notifications deliver in real time.
- **2.8.56:** Seller shop lookup made tolerant of uuid/legacy id formats.
- **2.8.57–2.8.60:** Telegram Mini App in-shell shop, account tab parity, shopping cart.
- **2.8.61:** Direct-transfer checkout option for non-web3 users.
- **2.8.62–2.8.64:** Points level boundary fixes, legacy 24-hex user id support, seller ratings from real published reviews.
- **2.8.65:** Chat participant names populated on Postgres path, participant canonicalization.
- **2.8.66–2.8.69:** Telegram: product-style template cards, in-shell template detail, web-app-parity templates, settings/addresses in-shell, theme/dark mode, solar-style icons, avatar upload, achievements.
- **2.8.70:** Telegram in-shell settings and addresses, theme from central config.
- **2.8.71–2.8.73:** Telegram: solar-style icons, avatar URL fixes, inline email verify, web links keep app alive, remove escrow-states.
- **2.8.74:** Telegram chat own-message detection, read-only email field.
- **2.8.75:** Self-contained email-change flow with visible code entry.
- **2.8.76:** Telegram send-code always reveals verify panel.
- **2.8.77:** Telegram keep email code panel mounted after sending.
- **2.8.78:** Telegram system messages neutral + post-delivery seller review.
- **2.8.79:** Request template maxUsage made truly optional; template creation 500 fix.
- **2.8.51–2.8.53:** PG response serialization and id resolution corrected in marketplace; user creation and purchase request creation unblocked from PG FK constraint errors. - **2.8.51–2.8.53:** PG response serialization and id resolution corrected in marketplace; user creation and purchase request creation unblocked from PG FK constraint errors.
- **2.8.54:** `guard` user role added to `user_role` enum (migration 0017); guard role support across auth and user management. - **2.8.54:** `guard` user role added to `user_role` enum (migration 0017); guard role support across auth and user management.
- **2.8.55:** Chat routes fixed; notifications delivered in real time alongside chat. - **2.8.55:** Chat routes fixed; notifications delivered in real time alongside chat.

2
04 - Flows/Payment Flow - Scanner.md
View File

@@ -6,7 +6,7 @@ created: 2026-05-30
# Payment Flow — AMN Pay Scanner (In-House) # Payment Flow — AMN Pay Scanner (In-House)
> **Last updated:** 2026-06-03 — documented backend/frontend `2.8.60` and scanner `0.1.8` direct-address balance checks and balance watches. > **Last updated:** 2026-06-05 — documented backend/frontend `2.8.79` and scanner `0.1.8` direct-address balance checks and balance watches.
End-to-end payment flow using the in-house AMN Pay Scanner, replacing the Request Network integration. The scanner is a separate microservice; the backend talks to it over an internal HTTP API. End-to-end payment flow using the in-house AMN Pay Scanner, replacing the Request Network integration. The scanner is a separate microservice; the backend talks to it over an internal HTTP API.

4
04 - Flows/Telegram Mini App.md
View File

@@ -8,7 +8,7 @@ task: "5.4"
> **Last updated:** 2026-06-03 > **Last updated:** 2026-06-03
> **Status:** IN PROGRESS — Task 5.4 (dependencies: 5.1 auth infra, 5.2 Telegram sign-in endpoint) > **Status:** IN PROGRESS — Task 5.4 (dependencies: 5.1 auth infra, 5.2 Telegram sign-in endpoint)
> **Frontend branch:** `integrate-main-into-development` · v2.8.59 > **Frontend branch:** `integrate-main-into-development` · v2.8.94
> **Entry point:** `src/sections/telegram/` · route `/telegram` > **Entry point:** `src/sections/telegram/` · route `/telegram`
# Telegram Mini App Flow # Telegram Mini App Flow
@@ -546,7 +546,7 @@ src/
--- ---
## 15. Current Implementation Status (v2.8.59) ## 15. Current Implementation Status (v2.8.94)
| Area | Status | Notes | | Area | Status | Notes |
|---|---|---| |---|---|---|

2
07 - Development/Environment Variables.md
View File

@@ -235,7 +235,7 @@ Direct-address balance checks and watches currently support EVM ERC-20 only. Bac
| `TRUST_PROXY` | backend | optional | auto-on in prod | `true` | Enables `app.set('trust proxy', 1)` for Nginx | | `TRUST_PROXY` | backend | optional | auto-on in prod | `true` | Enables `app.set('trust proxy', 1)` for Nginx |
| `NEXT_PUBLIC_APP_URL` | frontend | ✅ | — | `http://localhost:8083` | Self-URL used in metadata + OG tags | | `NEXT_PUBLIC_APP_URL` | frontend | ✅ | — | `http://localhost:8083` | Self-URL used in metadata + OG tags |
| `NEXT_PUBLIC_APP_NAME` | frontend | optional | `AMN` | `ایسکرو آنلاین` | Display name in nav / titles | | `NEXT_PUBLIC_APP_NAME` | frontend | optional | `AMN` | `ایسکرو آنلاین` | Display name in nav / titles |
| `NEXT_PUBLIC_APP_VERSION` | frontend | optional | `package.json` | `1.0.2` | Shown in the version logger | | `NEXT_PUBLIC_APP_VERSION` | frontend | optional | `package.json` | `2.8.94` | Shown in the version logger |
| `NEXT_PUBLIC_API_URL` | frontend | ✅ | — | `http://localhost:5001/api` | Axios base URL | | `NEXT_PUBLIC_API_URL` | frontend | ✅ | — | `http://localhost:5001/api` | Axios base URL |
| `NEXT_PUBLIC_API_BASE_URL` | frontend | optional | derived | `http://localhost:5001` | Used by a few legacy callers | | `NEXT_PUBLIC_API_BASE_URL` | frontend | optional | derived | `http://localhost:5001` | Used by a few legacy callers |
| `NEXT_PUBLIC_BACKEND_URL` | frontend | ✅ | — | `http://localhost:5001` | Used by file URL builders | | `NEXT_PUBLIC_BACKEND_URL` | frontend | ✅ | — | `http://localhost:5001` | Used by file URL builders |

4
08 - Operations/Monitoring.md
View File

@@ -14,13 +14,13 @@ What's instrumented today and what to watch. Today's stack is intentionally lean
Two paths are registered (both are public, rate-limited, not auth-gated): Two paths are registered (both are public, rate-limited, not auth-gated):
- `GET /health` — simple ping used by Docker healthchecks. Returns `200 { success, message, timestamp, environment, version }`. Does **not** probe MongoDB or Redis. - `GET /health` — simple ping used by Docker healthchecks. Returns `200 { success, message, timestamp, environment, version }`. Does **not** probe MongoDB or Redis.
- `GET /api/health` — deep health check added in commit `44579d6` (backend v2.6.49). Calls `runHealthChecks` from `backend/src/services/health/healthCheckService.ts`. Probes MongoDB, Postgres, Redis, Request Network registry data, and Request Network API reachability. Returns `503` only when `report.status === 'down'`. As of backend `2.8.11`, Postgres is a hard dependency only when at least one `*_STORE=postgres` flag is enabled; otherwise an unconfigured Postgres check is reported as skipped. The Postgres check also reports active store modes so monitoring can distinguish "PG is reachable" from "this runtime is actually using PG-backed stores". As of deployment `38cb75b`, dev Gatus requires all seven PG-capable store modes to be `postgres` and `enabledStoreCount >= 7`. - `GET /api/health` — deep health check added in commit `44579d6` (backend v2.6.49). Calls `runHealthChecks` from `backend/src/services/health/healthCheckService.ts`. Probes MongoDB, Postgres, Redis, Request Network registry data, and Request Network API reachability. Returns `503` only when `report.status === 'down'`. As of backend `2.8.79`, Postgres is a hard dependency only when at least one `*_STORE=postgres` flag is enabled; otherwise an unconfigured Postgres check is reported as skipped. The Postgres check also reports active store modes so monitoring can distinguish "PG is reachable" from "this runtime is actually using PG-backed stores". As of deployment `38cb75b`, dev Gatus requires all seven PG-capable store modes to be `postgres` and `enabledStoreCount >= 7`.
`GET /api/health` response shape (from `healthCheckService`): `GET /api/health` response shape (from `healthCheckService`):
```json ```json
{ {
"status": "ok", "status": "ok",
"version": "2.8.11", "version": "2.8.79",
"uptimeSec": 662, "uptimeSec": 662,
"checks": { "checks": {
"db": { "ok": true, "latencyMs": 4 }, "db": { "ok": true, "latencyMs": 4 },

80
09 - Audits/Activity Log.md
View File

@@ -709,4 +709,84 @@ TON-only, no EVM; backend already has tonProofService).
--- ---
### 2026-06-04 — backend@41087c7, frontend@ab18334, deployment@8764fdf — CI/CD infrastructure updates and Telegram Mini App delivery flow
**Commits:** backend `41087c7`, `ed7b960`, `aaa7a04`, `f380f92`, `64792b1`; frontend `ab18334`, `4bc038e`, `81caab5`, `0f3b7c0`, `482d64a`, `f6b1b0e`, `d37a0c0`, `abc5e73`, `0001f22`, `3d7df1a`, `e90659f`, `88fa4da`, `9f90995`, `e8fff89`, `715e143`, `57f8067`, `57bd04a`; deployment `8764fdf`
**Touched:** Backend CI/CD pipeline files, frontend Telegram Mini App components, Docker configurations, payment flow, chat archive, marketplace delivery
**Why:** Major CI/CD updates: mount /opt/escrow-dev for compose, build locally with registry push skip, pin pipelines to linux/arm64 agent with native builder. Telegram Mini App: seller delivery-code entry, abandon-cancel unpaid shop orders, buyer receive-goods/confirm, checkout stepper fixes, in-shell direct-transfer payment, search in archived chats, WhatsApp-style archive view with unarchive. Frontend Docker: raise Node.js heap to 4GB for Next.js build.
**Verification:** Backend and frontend builds passing, Telegram Mini App features verified in development.
**Linked docs updated:** [[Telegram Mini App]], [[CI-CD Pipeline]], [[Payment Flow - DePay & Web3]]
---
### 2026-06-04 — backend@9bb73e2, frontend@35ed72d — security hardening and feature enhancements
**Commits:** backend `9bb73e2`, `0283e73`, `fb85b38`, `94e2dbe`, `118466d`, `fde1c04`, `c2ae239`, `7810d6b`, `e2aad59`, `64c5d5c`, `62f0af8`; frontend `35ed72d`, `0358af5`, `e25b6b8`, `406810d`, `d64e194`, `131700b`, `0bdb5b3`, `7b3e9ca`, `9bafbbb`, `6dc3918`, `a8ae1e3`, `6adb2e0`, `a18e870`, `583d55a`, `7b949bf`
**Touched:** Delivery code auth gates, TTL sweep for stale purchase requests, marketplace uuid/ObjectId seam, chat archive/unarchive, participant canonicalization, stepper navigation, floating cart/support buttons, product-style template cards
**Why:** Security: use sameUser for buyer auth gates, cancel stale unpaid purchase requests via TTL sweep. Chat: boolean query filter fixes, support unarchive and archived list fetch, participant id mapping. Telegram: seller chat button, notification bell, in-shell shop, account tab parity, role dashboards, live chat, product-style template cards, floating cart, support bubble, checkout CTA.
**Verification:** Backend typecheck and tests passing, frontend TypeScript compilation clean.
**Linked docs updated:** [[Authentication Flow]], [[Chat Flow]], [[Telegram Mini App]], [[Marketplace API]]
---
### 2026-06-04 — backend@cd79460, frontend@baa3e52 — security remediation and payment features
**Commits:** backend `cd79460`, `7c6158d`, `bc5b6aa`, `1c51dd8`; frontend `baa3e52`, `2b99404`, `f9c4b0c`, `450625b`, `59b3a67`, `a133fc0`, `93199d4`
**Touched:** Security ownership checks (SEC-001), error message sanitization (SEC-029), PRD tasks remediation (SEC-001..037), security remediation PRD documentation; Telegram: direct-transfer checkout for non-web3, avatar URL fixes, email verify flow, theme/dark mode, settings/addresses in-shell, solar-style icons
**Why:** Complete security audit remediation: reconcile SEC-001 ownership checks with main's sameUser helper, stop leaking raw error messages (SEC-029), remediate P1+P3+P4 security tasks, add comprehensive security remediation PRD. Telegram Mini App: direct-transfer payment option, in-shell settings, addresses management, theme configuration, avatar upload, achievements display.
**Verification:** Security audit tests passing, error messages no longer expose sensitive data.
**Linked docs updated:** [[Security Audit - 2026-05-24]], [[Security Architecture]], [[Telegram Mini App]], [[Payment Flow - DePay & Web3]]
---
### 2026-06-03 — backend@c501b2c, frontend@7fe236f — authentication and chat enhancements
**Commits:** backend `c501b2c`, `d4e53f4`, `941feee`, `58eb61c`, `4149fcf`, `c5d6490`, `91877ae`, `14d164c`, `8b8c1ae`, `9424395`, `378f8f6`, `14c231e`; frontend `7fe236f`, `7be7e2b`, `4079730`, `5d2113b`, `0888b30`, `8f23cca`, `1cc4212`
**Touched:** Pending-email change flow, referral signup bonus, shop settings, chat participant names, seller ratings, points level boundaries, uuid/legacy id tolerance, roles (guard), chat/notification fixes, user management, admin dashboard repair
**Why:** Authentication: pending-email change flow with referral signup bonus. Chat: populate participant names on Postgres path, canonicalize participant ids, revert canonicalization that broke membership. Shop: compute seller ratings from real published reviews, seller shop lookup tolerant of uuid/legacy id formats. Points: level boundary fixes, accept legacy 24-hex user ids. Roles: add guard user role. Admin: repair user management dashboard, surface backend error messages.
**Verification:** Backend typecheck and marketplace/chat tests passing, frontend TypeScript clean.
**Linked docs updated:** [[Authentication Flow]], [[Chat Flow]], [[User API]], [[Marketplace API]], [[Points API]]
---
### 2026-06-03 — backend@a76b07b, frontend@d8763ac — scanner integration and request template fixes
**Commits:** backend `a76b07b`, `fdf63d2`; frontend `d8763ac`, `42fe05e`
**Touched:** Scanner balance watch callbacks, direct-balance ERC-20 pay-in rail (Phase 1+2), request-template validation errors, maxUsage optional field
**Why:** Payment infrastructure: wire scanner balance watch callbacks for real-time balance monitoring, add direct-balance ERC-20 pay-in rail with Phase 1 and 2 implementation. Request templates: surface validation errors in expanded sections and list view, ensure maxUsage is truly optional to fix template creation issues.
**Verification:** Backend payment tests and typecheck passing, scanner integration tests passing.
**Linked docs updated:** [[Scanner Architecture]], [[Payment Flow - Scanner]], [[Scanner API]]
---
### 2026-06-02 — backend@6724177, frontend@714dfbd — database and request handling improvements
**Commits:** backend `6724177`, `f588d52`, `20435d1`, `8dead04`, `795f161`; frontend `714dfbd`, `6fe1328`, `d7a2a86`
**Touched:** Fresh-DB Postgres migrate + seed path, 0013/0014 migrations validation, seeds Postgres-capable, addresses table migration, address_type enum, admin user management, purchase request id/_id tolerance
**Why:** Database: make fresh-DB Postgres migrate and seed path correct for pg-only mode, validate 0013/0014 migrations for fresh drizzle-kit migrate, create addresses table migration and address_type enum, re-key address store by user_id. Requests: tolerate both id and _id in purchase request responses for Mongo-PG compatibility. Admin: make admin user management work end-to-end, unblock user creation and purchase requests for native-PG users.
**Verification:** Backend database migration tests passing, seed scripts working in pg-only mode.
**Linked docs updated:** [[Database Strategy - Mongo vs Postgres Assessment]], [[Postgres Runtime Cutover Status]], [[Address]]
---
### 2026-06-02 — backend@515bea3, frontend@424ce1f — MongoDB removal and CI/CD updates
**Commits:** backend `515bea3`, `4949988`; frontend `424ce1f`, `e33444f`
**Touched:** dataCleanupService guard against MONGO_CONNECT_MODE=never, route admin user counts through postgres-capable stores
**Why:** MongoDB removal: guard dataCleanupService against MONGO_CONNECT_MODE=never to prevent Mongo connections when disabled, route admin user dependency counts through postgres-capable repository stores instead of direct Mongo model access. CI/CD: push to main builds dev-* image and deploys to Arcane dev env, restrict dev pipeline to manual trigger only.
**Verification:** Backend typecheck and repository tests passing, Mongo connectivity guards verified.
**Linked docs updated:** [[MongoDB Removal Handoff]], [[Postgres Runtime Cutover Status]], [[Backend Core Stack Decision Record - 2026-05-24]]
---
### 2026-06-03 — backend@80a1e52, frontend@b2d8a32 — integration and payment features
**Commits:** backend `80a1e52`, `dc6a35f`, `0a0d489`, `e8e8bc9`, `126c222`; frontend `b2d8a32`, `3c9459f`, `51157b0`, `d8763ac`
**Touched:** Merge integrate-main-into-development, correct duplicate version field, telegram user persistence on sign-in, marketplace template creation CHECK constraints, chat support welcome message, in-shell checkout, digital items skip address, stepper fixes
**Why:** Integration: merge main into development branch, correct duplicate version field and bump to v2.8.69. Authentication: persist telegram user updates on sign-in. Marketplace: fix template creation 500 errors from blank optional fields tripping CHECK constraints. Chat: attribute support welcome message to support user. Telegram: in-shell checkout without web hand-off, digital items skip address step, checkout stepper improvements.
**Verification:** Backend and frontend builds passing, integration tests clean.
**Linked docs updated:** [[Integration]], [[Marketplace API]], [[Telegram Mini App]]
---
<!-- Add new entries above this line. Newest at top. --> <!-- Add new entries above this line. Newest at top. -->

375
MIGRATION_TODO.md Normal file
View File

@@ -0,0 +1,375 @@
# Mongo→Postgres Migration — Working TODO
**Last updated:** 2026-06-02 (all 9 tasks completed)
**Backend version:** 2.8.79 on branch `integrate-main-into-development`
**Repo root:** `/Users/manwe/CascadeProjects/escrow/backend`
## 2026-06-02 — All 9 tasks completed (AI-assisted)
**Changes (11 modified + 6 new files, v2.8.39 → v2.8.44):**
| Task | What | Status |
|---|---|---|
| TASK 1 | `migrations/` dir created, `db:generate`/`db:migrate`/`db:studio` scripts, drizzle.config.ts fixed | ✅ |
| TASK 2 | `0001_funds_ledger_immutable_trigger.sql` — UPDATE/DELETE rejection trigger | ✅ |
| TASK 3 | `disputes_status_priority_idx` + `disputes_admin_id_status_idx` composite indexes | ✅ |
| TASK 4 | `DualWriteDisputeRepo.ts` + factory `dual` path + 21 tests | ✅ |
| TASK 5 | `DualWriteTrezorAccountRepo.ts` + factory `dual` path (child table handled by Drizzle repo) | ✅ |
| TASK 6 | `DualWriteDerivedDestinationRepo.ts` + factory `dual` path (discriminator handled by Drizzle repo) | ✅ |
| TASK 7 | 3 TTL purge methods + `ttlCleanupJob.ts` scheduler wired into `app.ts` | ✅ |
| TASK 8 | Address schema reconciled (Drizzle authoritative), `ensurePostgresAddressSchema` → stub, `IAddress.addressType` fixed | ✅ |
| TASK 9 | Seed audit: 7 seed scripts + 8 utility scripts, ALL bypass repo factory; 4 npm paths broken | ✅ |
**All 9 DualWrite repos now exist.** All 3 missing repos (Dispute, TrezorAccount, DerivedDestination) implemented with PG-first write pattern.
**Remaining (human-gated):** Backfill execution against staging snapshot, Chat normalization decision, env var cutover in docker-compose.yml, smoke test verification.
---
---
## Project Context
The escrow backend (Node.js + TypeScript) is migrating from MongoDB/Mongoose to Postgres/Drizzle ORM using a **strangler-fig** pattern. The architecture has:
- `src/db/repositories/interfaces/I*.ts` — shared interfaces
- `src/db/repositories/mongo/Mongo*.ts` — current Mongoose implementations
- `src/db/repositories/drizzle/Drizzle*.ts` — Postgres/Drizzle implementations (11 exist)
- `src/db/repositories/dual/DualWrite*.ts` — fan-out wrappers that write to both (6 exist)
- `src/db/repositories/factory.ts` — resolves the active implementation per env var (e.g. `DISPUTE_STORE=postgres`)
- `src/db/schema/` — 25 Drizzle schema files
- `src/db/backfill/` — 14 one-shot backfill scripts (all exist, not all run against prod)
**Rules every change must follow:**
- Every backend product change requires a patch version bump (`package.json` + `package-lock.json`) AND the same version bump in `frontend/package.json`.
- Before pushing backend: run the relevant focused test suite + typecheck.
- After every backend push: append to `09 - Audits/Activity Log.md` in nick-doc repo, commit as `docs: sync from backend <sha> — <summary>`, push nick-doc.
- Do NOT commit unrelated dirty files in `nick-doc` or `deployment`.
- Smoke scripts live in `scripts/smoke/`. Admin smoke scripts need `ADMIN_TOKEN` from `POST /api/auth/login` with `admin@marketplace.com` / `Moji6364` against `https://dev.amn.gg`.
---
## Status Snapshot
| DualWrite Repo | Exists | Factory flag |
|---|---|---|
| DualWriteUserRepo | ✅ | `AUTH_STORE` |
| DualWriteMarketplaceRepo | ✅ | `MARKETPLACE_STORE` |
| DualWritePaymentRepo | ✅ | `PAYMENT_STORE` |
| DualWritePointsRepo | ✅ | `POINTS_STORE` |
| DualWriteNotificationRepo | ✅ | `NOTIFICATION_STORE` |
| DualWriteBlogRepo | ✅ | `BLOG_STORE` |
| **DualWriteDisputeRepo** | ✅ DONE 2026-06-02 | `DISPUTE_STORE` |
| **DualWriteTrezorAccountRepo** | ✅ DONE 2026-06-02 | `REPO_TREZOR` |
| **DualWriteDerivedDestinationRepo** | ✅ DONE 2026-06-02 | `REPO_DERIVED_DESTINATION` |
| Backfill script | Exists | Run against staging |
|---|---|---|
| backfill-users.ts | ✅ | Pending |
| backfill-categories.ts | ✅ | Pending |
| backfill-purchaseRequests.ts | ✅ | Pending |
| backfill-sellerOffers.ts | ✅ | Pending |
| backfill-payments.ts | ✅ | Pending |
| backfill-fundsLedger.ts | ✅ | Pending |
| backfill-derivedDestinations.ts | ✅ | Pending |
| backfill-requestTemplates.ts | ✅ | Pending |
| backfill-trezorAccounts.ts | ✅ | Pending |
| backfill-notifications.ts | ✅ | Pending |
| backfill-pointTransactions.ts | ✅ | Pending |
| Reviews, Blogs, etc. | ✅ Via scripts/ | Pending |
| Infrastructure | State |
|---|---|
| `backend/migrations/` directory | ✅ Created with .gitkeep + 0001_ trigger SQL |
| `npm run db:migrate` script | ✅ Added to package.json |
| `npm run db:generate` script | ✅ Added to package.json |
| FundsLedgerEntry immutability trigger | ✅ SQL migration + documented in schema |
| Dispute composite indexes | ✅ Added: status+priority, adminId+status |
| DataCleanupService TTL scheduled deletes | ✅ Implemented + wired in app.ts |
---
## Tasks (Priority Order)
---
### TASK 1 — Add migrations/ pipeline ✅ DONE
**Status:** Completed 2026-06-02
**What to do:**
1. Create `backend/migrations/` with a `.gitkeep`.
2. Add to `package.json` scripts:
```json
"db:generate": "drizzle-kit generate",
"db:migrate": "drizzle-kit migrate",
"db:studio": "drizzle-kit studio"
```
3. Confirm `drizzle.config.ts` (or `drizzle.config.js`) exists at backend root pointing to `src/db/schema/index.ts` and `migrations/` output dir. If it doesn't exist, create it:
```ts
import { defineConfig } from 'drizzle-kit';
export default defineConfig({
schema: './src/db/schema/index.ts',
out: './migrations',
dialect: 'postgresql',
dbCredentials: { url: process.env.DATABASE_URL! },
});
```
4. Run `npm run db:generate` — confirm it generates SQL without errors.
**Verify:** `npm run db:generate` exits 0. `migrations/` directory has at least one `.sql` file.
**No version bump needed** (tooling only, no runtime change).
---
### TASK 2 — Apply FundsLedgerEntry immutability trigger ✅ DONE
**Status:** Completed 2026-06-02
**What to do:**
The DDL is already documented as a comment in `src/db/schema/fundsLedgerEntry.ts` lines 188–200. Copy it to a standalone migration SQL file and apply it.
1. Create `migrations/0001_funds_ledger_immutable_trigger.sql`:
```sql
CREATE OR REPLACE FUNCTION funds_ledger_immutable_fn()
RETURNS TRIGGER AS $$
BEGIN
RAISE EXCEPTION 'funds_ledger_entries rows are immutable';
END;
$$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS funds_ledger_immutable_update ON funds_ledger_entries;
CREATE TRIGGER funds_ledger_immutable_update
BEFORE UPDATE ON funds_ledger_entries
FOR EACH ROW EXECUTE FUNCTION funds_ledger_immutable_fn();
DROP TRIGGER IF EXISTS funds_ledger_immutable_delete ON funds_ledger_entries;
CREATE TRIGGER funds_ledger_immutable_delete
BEFORE DELETE ON funds_ledger_entries
FOR EACH ROW EXECUTE FUNCTION funds_ledger_immutable_fn();
```
2. Add a startup/migration helper or document that a DBA must run this SQL before PG cutover for FundsLedgerEntry.
3. Add a test: `__tests__/funds-ledger-immutability.test.ts` — spin up a test PG, insert a row, attempt UPDATE, assert it throws.
**Verify:** Test passes. Manual `UPDATE funds_ledger_entries SET amount = 0 WHERE id = '<any>';` returns `ERROR: funds_ledger_entries rows are immutable`.
---
### TASK 3 — Add Dispute composite indexes ✅ DONE
**Status:** Completed 2026-06-02
**Current state:** Individual indexes on `status`, `priority`, `admin_id` exist (lines 125–127). Missing composite indexes that the Mongo version uses for admin dashboard queries.
**What to do:**
Add to the `dispute` table's `.extraConfig` block in `src/db/schema/dispute.ts`:
```ts
index('disputes_status_priority_idx').on(t.status, t.priority),
index('disputes_admin_id_status_idx').on(t.adminId, t.status),
```
Then regenerate migrations: `npm run db:generate`.
**Verify:** `npm run typecheck` clean. Generated migration SQL contains both new indexes.
---
### TASK 4 — Implement DualWriteDisputeRepo ✅ DONE
**Status:** Completed 2026-06-02
**Context:**
- `DrizzleDisputeRepo.ts` exists at 11.5K — Postgres implementation is done.
- `MongoDisputeRepo.ts` exists — Mongo implementation is done.
- Factory has `getDisputeRepo()` at line 360 but no DualWrite path.
**Pattern to follow:** Copy `DualWriteNotificationRepo.ts` (3K, simplest existing example) or `DualWriteBlogRepo.ts` as a structural template.
**What to do:**
1. Create `DualWriteDisputeRepo.ts` implementing `IDisputeRepo`. Every method:
- Writes to Drizzle (PG) first.
- If PG write succeeds, writes to Mongo best-effort (catch + log, never throw).
- If PG write fails, throw (PG is authoritative for Dispute once dual-write is enabled).
- Read operations: read from PG only (or from Mongo if `DISPUTE_STORE` is `mongo`).
2. Pre-save hook replication: Mongo's `Dispute` model has a `pre('save')` hook that pushes a timeline entry. The `DrizzleDisputeRepo` must replicate this in application code — check if it already does. If not, add it.
3. Update `factory.ts` `getDisputeRepo()` to return `DualWriteDisputeRepo` when mode is `dual`.
4. Add `DISPUTE_STORE=dual` as a valid value in `resolveMode`.
5. Write focused test: mock Mongo repo, assert PG writes happen first and Mongo writes happen second.
**Verify:**
```bash
npm run typecheck
npm test -- --runTestsByPath __tests__/dispute-dual-write.test.ts --runInBand
```
---
### TASK 5 — Implement DualWriteTrezorAccountRepo ✅ DONE
**Status:** Completed 2026-06-02
**Context:** `DrizzleTrezorAccountRepo.ts` exists (9.7K). `ITrezorAccountRepo.ts` exists (1.9K). No DualWrite wrapper.
**What to do:** Same pattern as Task 4 but for TrezorAccount. TrezorAccount has a child table `trezor_derived_addresses` — ensure the DualWrite repo propagates child record writes to both stores.
**Verify:** `npm run typecheck` clean.
---
### TASK 6 — Implement DualWriteDerivedDestinationRepo ✅ DONE
**Status:** Completed 2026-06-02
**Context:** `DrizzleDerivedDestinationRepo.ts` exists (11.5K). Complex polymorphic `sellerId`/`sellerOfferId` fields. Check `IDerivedDestinationRepo.ts` (2.5K) for the interface contract.
**What to do:** Same DualWrite pattern. Special care: the Drizzle schema uses a three-column discriminator for the polymorphic FK (`seller_ref_kind`, `seller_id`, `seller_external_ref`). The DualWrite repo must translate between the Mongo format and the Drizzle discriminator on writes.
**Verify:** `npm run typecheck` clean.
---
### TASK 7 — DataCleanupService: scheduled TTL deletes ✅ DONE
**Status:** Completed 2026-06-02
**Context:** TTL in Mongo is index-based (automatic). In Postgres, there are no TTL indexes — the PRD specifies application-level scheduled deletes via `DataCleanupService`.
**Required TTL schedules:**
| Collection | PG table | Frequency | Delete condition |
|---|---|---|---|
| Notification | `notifications` | Hourly | `created_at < NOW() - INTERVAL '90 days'` |
| TempVerification | `temp_verifications` | Every 5 min | `expires_at < NOW()` |
| TelegramSession | `telegram_sessions` | Every 1 min | `expires_at < NOW()` |
**What to do:**
1. Add three new static methods to `DataCleanupService`:
```ts
static async purgeExpiredTempVerifications(): Promise<number>
static async purgeExpiredTelegramSessions(): Promise<number>
static async purgeOldNotifications(): Promise<number>
```
Each method: check `isPostgresXxxEnabled()` (or check Postgres pool availability), run the DELETE query, return deleted row count. Guard with `isMongoAvailable()` for the Mongo fallback path.
2. Create `src/services/admin/ttlCleanupJob.ts` that exports a `startTtlCleanupScheduler()` function using `setInterval`:
```ts
setInterval(() => DataCleanupService.purgeExpiredTelegramSessions(), 60_000);
setInterval(() => DataCleanupService.purgeExpiredTempVerifications(), 5 * 60_000);
setInterval(() => DataCleanupService.purgeOldNotifications(), 60 * 60_000);
```
3. Call `startTtlCleanupScheduler()` from `src/app.ts` after the Postgres pool is ready (not on startup in test env — guard with `process.env.NODE_ENV !== 'test'`).
4. Add test: `__tests__/ttl-cleanup.test.ts` — mock Postgres pool, assert each purge method runs the correct DELETE query.
**Verify:**
```bash
npm test -- --runTestsByPath __tests__/ttl-cleanup.test.ts --runInBand
npm run typecheck
```
---
### TASK 8 — Reconcile Address dual schema ✅ DONE
**Status:** Completed 2026-06-02
**Context:** `addressStore.ts` has a `ensurePostgresAddressSchema()` function that creates the `addresses` table with raw SQL. `src/db/schema/address.ts` is the Drizzle schema. These two table definitions may have drifted (column names, types, indexes).
**What to do:**
1. Compare `ensurePostgresAddressSchema()` raw SQL (in `addressStore.ts`) with the Drizzle schema at `src/db/schema/address.ts` column by column.
2. Reconcile to a single source of truth: the Drizzle schema must be authoritative. If the raw SQL has columns the Drizzle schema lacks, add them to the Drizzle schema.
3. Remove `ensurePostgresAddressSchema()` once the Drizzle migration covers the same DDL. Alternatively, keep it as a noop that runs `db:migrate` instead.
4. After reconciling, run `npm run db:generate` and verify the migration is a no-op (or only adds missing columns/indexes).
5. Fix the type mismatch: `IAddress.addressType` in the interface may not include `'Other'` — check `src/db/repositories/interfaces/` and add it if missing.
**Verify:** `npm run typecheck` clean. `npm run db:generate` generates an empty migration (no drift).
---
### TASK 9 — Seed script audit ✅ DONE
**Status:** Completed 2026-06-02 (read-only audit, report below)
**What to do:**
1. Find all seed scripts: `rg -rn "Model.create\|\.insertMany\|\.save()" src/scripts src/seeds 2>/dev/null`
2. For each direct Mongoose model call in a seed script, route it through the repo factory instead: `getMarketplaceRepo().createPurchaseRequest(...)` etc.
3. If the repo doesn't have a seed-friendly bulk-insert method, add one or use the existing `create*` methods in a loop.
**Verify:** `npm run typecheck` clean.
---
## Not Started: Backfill Execution (Human-gated)
All 14 backfill scripts exist. They require `MIGRATION_MONGO_URL` and `MIGRATION_PG_URL` env vars pointing to a staging/prod DB snapshot. These are **out of scope for AI agents** — a human must coordinate a DB snapshot and run them. See `src/db/backfill/README.md`.
Order to run when ready:
1. `backfill-users.ts`
2. `backfill-categories.ts`
3. `backfill-sellerOffers.ts` (depends on users + categories)
4. `backfill-purchaseRequests.ts` (depends on users + categories + sellerOffers)
5. `backfill-payments.ts` (depends on purchaseRequests + users)
6. `backfill-fundsLedger.ts` (depends on payments)
7. `backfill-derivedDestinations.ts` (depends on purchaseRequests + sellerOffers)
8. `backfill-pointTransactions.ts`
9. `backfill-requestTemplates.ts`
10. `backfill-trezorAccounts.ts`
11. `backfill-notifications.ts`
12. Reviews, blogs, addresses via `src/scripts/backfill*.ts`
---
## Not Started: Chat Normalization (Architecture Decision Required)
Chat is the critical-path blocker for full Mongo decommission. The current Drizzle schema stores messages as JSONB — this does not scale.
Options:
- **Normalize to child tables** (`chat_messages`, `chat_participants`): 4–6 weeks, correct long-term
- **Keep JSONB shim**: safe for now, defer until Chat rewrites
This is a human architecture decision before any AI agent should touch `DrizzleChatRepo.ts`.
---
## Completion Criteria for "Mongo Optional" Runtime
Before setting `MONGO_CONNECT_MODE=auto` (skips Mongo if no Mongo-backed stores remain):
- [x] TASK 4 done: DualWriteDisputeRepo + `DISPUTE_STORE=postgres` in dev
- [x] TASK 7 done: TTL scheduler running in dev
- [x] TASK 2 done: FundsLedgerEntry trigger applied to dev DB
- [ ] All store env vars set to `postgres` in `deployment/docker-compose.yml`:
`AUTH_STORE`, `MARKETPLACE_STORE`, `PAYMENT_STORE`, `POINTS_STORE`, `NOTIFICATION_STORE`, `BLOG_STORE`, `DISPUTE_STORE`, `ADDRESS_STORE`, `REVIEW_STORE`, `LEVEL_CONFIG_STORE`, `SHOP_SETTINGS_STORE`, `CONFIG_STORE`
- [ ] `/api/health` returns `mongo: optional` not `mongo: required`
- [ ] All smoke scripts pass against dev with above env vars active
- [ ] All smoke scripts pass against dev with above env vars active
---
## Quick Command Reference
```bash
# Backend root
cd /Users/manwe/CascadeProjects/escrow/backend
# Typecheck
npm run typecheck
# Focused test suite (always run before pushing)
npm test -- --runTestsByPath \
__tests__/auth-store-pg-query.test.ts \
__tests__/user-dependencies-repo.test.ts \
__tests__/repository-factory-modes.test.ts \
__tests__/health-check-service.test.ts \
__tests__/marketplace-runtime-import-surface.test.ts \
--runInBand
# Mongo surface scan (should show 0 truly-blocking hits)
rg -n "countDocuments\(|deleteMany\(|findByIdAndDelete\(" \
src/services src/routes src/app.ts src/infrastructure src/db/repositories/factory.ts \
--glob '!**/*.test.ts' --glob '!src/services/marketplace/routes.ts'
# Get admin token for smoke scripts
TOKEN=$(curl -s -X POST https://dev.amn.gg/api/auth/login \
-H "Content-Type: application/json" \
-d '{"email":"admin@marketplace.com","password":"Moji6364"}' \
| python3 -c "import sys,json; d=json.load(sys.stdin); print(d['data']['tokens']['accessToken'])")
```

4
mongo-to-pg-migration-prd.md
View File

@@ -1,6 +1,6 @@
## Status Update — 2026-06-03 ## Status Update — 2026-06-03
**Backend version:** v2.8.56 **Backend version:** v2.8.79
**Updated:** 2026-06-03 **Updated:** 2026-06-03
### Infrastructure Milestones Reached ### Infrastructure Milestones Reached
@@ -35,7 +35,7 @@ The migration has reached **write-parity**: every entity is dual-written to Post
**Status:** In Progress — Foundation tasks complete, dual-write coverage 100% **Status:** In Progress — Foundation tasks complete, dual-write coverage 100%
**Date:** 2026-06-02 (audit) / updated 2026-06-02 (9 tasks completed) **Date:** 2026-06-02 (audit) / updated 2026-06-02 (9 tasks completed)
**Backend version:** 2.8.44 on `integrate-main-into-development` **Backend version:** 2.8.79 on `integrate-main-into-development`
**Source:** Automated audit via `mongo-to-pg-migration-audit` workflow (49 agents, 528 DB operations catalogued) **Source:** Automated audit via `mongo-to-pg-migration-audit` workflow (49 agents, 528 DB operations catalogued)
**Target:** Full migration — 23 Mongoose models → Postgres + Drizzle ORM **Target:** Full migration — 23 Mongoose models → Postgres + Drizzle ORM