diff --git a/08 - Operations/MONGODB_REMOVAL_HANDOFF_2026-06-02.md b/08 - Operations/MONGODB_REMOVAL_HANDOFF_2026-06-02.md new file mode 100644 index 0000000..595b473 --- /dev/null +++ b/08 - Operations/MONGODB_REMOVAL_HANDOFF_2026-06-02.md @@ -0,0 +1,454 @@ +# MongoDB Runtime Removal Handoff + +Date: 2026-06-02 +Workspace: `/Users/manwe/CascadeProjects/escrow` +Goal: remove MongoDB as a runtime dependency by migrating remaining Mongo-backed backend domains and cutover paths to Postgres-compatible repositories. + +## Current State + +No commits or pushes were made for the current WIP. The work is local only. + +Repo heads at handoff time: + +| Repo | Branch | HEAD | State | +| --- | --- | --- | --- | +| `backend` | `integrate-main-into-development` | `cf59726` | dirty WIP | +| `frontend` | `integrate-main-into-development` | `a2b972b` | dirty version bump only | +| `nick-doc` | `main` | `345c585` | dirty unrelated local docs/profile files | +| `deployment` | `main` | `8764fdf` | dirty unrelated `.env` and `docker-compose.yml` | + +Backend/frontend package versions: backend at `2.8.79`, frontend at `2.8.94`. + +Important repo rules: + +- Any backend/frontend product change requires patch version bump in both repos. +- Before any backend push, run the relevant focused tests and smoke script. +- After every backend push, sync `nick-doc`: + - append `09 - Audits/Activity Log.md` + - update relevant data/architecture docs + - commit as `docs: sync from backend — ` + - push `nick-doc` +- Do not stage unrelated dirty files in `nick-doc` or `deployment`. + +## Dirty Files + +Backend dirty files: + +```text +package-lock.json +package.json +src/db/repositories/drizzle/DrizzleMarketplaceRepo.ts +src/db/repositories/drizzle/DrizzlePaymentRepo.ts +src/db/repositories/dual/DualWriteMarketplaceRepo.ts +src/db/repositories/dual/DualWritePaymentRepo.ts +src/db/repositories/interfaces/IMarketplaceRepo.ts +src/db/repositories/interfaces/IPaymentRepo.ts +src/db/repositories/mongo/MongoMarketplaceRepo.ts +src/db/repositories/mongo/MongoPaymentRepo.ts +src/services/auth/authStore.ts +src/services/user/userController.ts +__tests__/auth-store-pg-query.test.ts +__tests__/user-dependencies-repo.test.ts +scripts/smoke/user-admin-postgres.sh +scripts/smoke/user-dependencies.sh +``` + +Frontend dirty files: + +```text +package.json +``` + +## What This WIP Changes + +### 1. Admin User Dependencies Endpoint + +Endpoint affected: + +```text +GET /api/user/admin/:userId/dependencies +``` + +Before this WIP, `src/services/user/userController.ts` dynamically imported Mongo models and counted dependencies directly: + +- `RequestTemplate.countDocuments(...)` +- `PurchaseRequest.countDocuments(...)` +- `Payment.countDocuments(...)` +- `Chat.countDocuments(...)` + +This WIP replaces that direct model access with repository calls: + +- `getMarketplaceRepo().getUserDependencyCounts(userId)` +- `getPaymentRepo().countByParticipant(userId)` +- `getChatRepo().count({ 'participants.userId': userId, 'participants.isActive': true })` + +New/extended repository contract methods: + +- `IMarketplaceRepo.getUserDependencyCounts(userId)` +- `IPaymentRepo.countByParticipant(userId)` + +Implementations added: + +- `MongoMarketplaceRepo.getUserDependencyCounts` +- `DrizzleMarketplaceRepo.getUserDependencyCounts` +- `DualWriteMarketplaceRepo.getUserDependencyCounts` +- `MongoPaymentRepo.countByParticipant` +- `DrizzlePaymentRepo.countByParticipant` +- `DualWritePaymentRepo.countByParticipant` + +Behavior note: + +- Postgres counts seller-side marketplace dependencies by joining `purchase_requests.selected_offer_id` to `seller_offers.id` and checking `seller_offers.seller_id`. +- Mongo implementation supports selected-offer-id style and also keeps compatibility with legacy embedded `selectedOffer.sellerId`. + +New test: + +```text +__tests__/user-dependencies-repo.test.ts +``` + +New smoke script: + +```text +scripts/smoke/user-dependencies.sh +``` + +Smoke usage: + +```bash +BASE_URL=https://dev.amn.gg ADMIN_TOKEN= USER_ID= scripts/smoke/user-dependencies.sh +``` + +The smoke script checks: + +- HTTP 200 +- `success === true` +- `data.user` exists +- dependency counters are non-negative numbers +- `total` equals the sum of component counters + +### 2. AuthUser Postgres Query Facade Hardening + +Files affected: + +```text +src/services/auth/authStore.ts +__tests__/auth-store-pg-query.test.ts +``` + +The mounted user/admin list and stats routes already use `AuthUser` from `authStore`, not raw Mongoose imports. However, the Postgres `PgQuery` wrapper only sorted arrays by `createdAt`; admin list accepts arbitrary `sortBy`. + +This WIP hardens the Postgres query wrapper so `AuthUser.find(...).select(...).sort(...).skip(...).limit(...).lean()` behaves more like the existing Mongoose chain: + +- generic sorting by requested field +- nested path sorting support, e.g. `profile.avatar` +- date, number, boolean, and string comparison +- multi-field sort support +- keeps existing skip/limit/select/lean chain behavior + +It also adds alias support in `buildUserWhere`: + +- `isActive: true` maps to `status = 'active'` +- `isActive: false` maps to `status <> 'active'` +- `isVerified` maps to `is_email_verified` + +This matters because: + +- `src/services/user/userController.ts` builds filters with `isActive` / `isVerified` +- `src/services/user/userRoutes.ts` builds filters with `status` / `isEmailVerified` +- both now work in Postgres auth mode + +New test: + +```text +__tests__/auth-store-pg-query.test.ts +``` + +New smoke script: + +```text +scripts/smoke/user-admin-postgres.sh +``` + +Smoke usage: + +```bash +BASE_URL=https://dev.amn.gg ADMIN_TOKEN= scripts/smoke/user-admin-postgres.sh +``` + +The smoke script checks: + +- `GET /api/user/admin/list?page=1&limit=5&sortBy=firstName&sortOrder=asc` +- `GET /api/users/admin/stats` +- response shape and numeric stats + +## Verification Already Run + +Passed: + +```bash +npm test -- --runTestsByPath __tests__/auth-store-pg-query.test.ts __tests__/user-dependencies-repo.test.ts __tests__/repository-factory-modes.test.ts __tests__/health-check-service.test.ts __tests__/marketplace-runtime-import-surface.test.ts --runInBand +``` + +Result: + +```text +Test Suites: 5 passed, 5 total +Tests: 11 passed, 11 total +``` + +Passed: + +```bash +npm run typecheck +``` + +Passed: + +```bash +git diff --check +``` + +for both backend and frontend. + +Passed syntax checks: + +```bash +bash -n scripts/smoke/user-admin-postgres.sh +bash -n scripts/smoke/user-dependencies.sh +``` + +Not run end-to-end: + +- `scripts/smoke/user-dependencies.sh` +- `scripts/smoke/user-admin-postgres.sh` + +Reason: both require an `ADMIN_TOKEN`; `user-dependencies.sh` also requires `USER_ID`. + +## How To Pick This Up + +Start with: + +```bash +cd /Users/manwe/CascadeProjects/escrow/backend +git status --short --branch +git diff --stat +``` + +Review the exact WIP: + +```bash +git diff -- src/services/user/userController.ts src/services/auth/authStore.ts +git diff -- src/db/repositories/interfaces/IMarketplaceRepo.ts src/db/repositories/interfaces/IPaymentRepo.ts +git diff -- src/db/repositories/mongo/MongoMarketplaceRepo.ts src/db/repositories/drizzle/DrizzleMarketplaceRepo.ts src/db/repositories/dual/DualWriteMarketplaceRepo.ts +git diff -- src/db/repositories/mongo/MongoPaymentRepo.ts src/db/repositories/drizzle/DrizzlePaymentRepo.ts src/db/repositories/dual/DualWritePaymentRepo.ts +git diff -- __tests__/auth-store-pg-query.test.ts __tests__/user-dependencies-repo.test.ts +git diff -- scripts/smoke/user-admin-postgres.sh scripts/smoke/user-dependencies.sh +``` + +Re-run local verification: + +```bash +npm test -- --runTestsByPath __tests__/auth-store-pg-query.test.ts __tests__/user-dependencies-repo.test.ts __tests__/repository-factory-modes.test.ts __tests__/health-check-service.test.ts __tests__/marketplace-runtime-import-surface.test.ts --runInBand +npm run typecheck +git diff --check +``` + +If you have a dev admin token: + +```bash +BASE_URL=https://dev.amn.gg ADMIN_TOKEN= USER_ID= scripts/smoke/user-dependencies.sh +BASE_URL=https://dev.amn.gg ADMIN_TOKEN= scripts/smoke/user-admin-postgres.sh +``` + +If you commit this WIP, suggested commit shape: + +Backend: + +```bash +git add package.json package-lock.json \ + src/db/repositories/interfaces/IMarketplaceRepo.ts \ + src/db/repositories/interfaces/IPaymentRepo.ts \ + src/db/repositories/mongo/MongoMarketplaceRepo.ts \ + src/db/repositories/drizzle/DrizzleMarketplaceRepo.ts \ + src/db/repositories/dual/DualWriteMarketplaceRepo.ts \ + src/db/repositories/mongo/MongoPaymentRepo.ts \ + src/db/repositories/drizzle/DrizzlePaymentRepo.ts \ + src/db/repositories/dual/DualWritePaymentRepo.ts \ + src/services/user/userController.ts \ + src/services/auth/authStore.ts \ + __tests__/auth-store-pg-query.test.ts \ + __tests__/user-dependencies-repo.test.ts \ + scripts/smoke/user-admin-postgres.sh \ + scripts/smoke/user-dependencies.sh + +git commit -m "fix: route admin user counts through postgres-capable stores" +``` + +Frontend: + +```bash +cd ../frontend +git add package.json +git commit -m "chore: sync frontend version to 2.8.38" +``` + +Do not push without doing the `nick-doc` sync afterward. + +## Remaining Runtime Mongo Scan + +Latest scan command: + +```bash +rg -n --pcre2 "^import (?!type).*from ['\"]mongoose['\"]|^import (?!type).*from ['\"][^'\"]*models/|await import\(['\"][^'\"]*models/|countDocuments\(|deleteMany\(|findByIdAndDelete\(" \ + src/services src/routes src/app.ts src/infrastructure src/db/repositories/factory.ts \ + --glob '!**/*.test.ts' \ + --glob '!src/services/marketplace/routes.ts' +``` + +Important results and interpretation: + +### Auth/User Routes + +Paths still visible in scans: + +```text +src/app.ts:672 AuthUser.countDocuments({ role: { $ne: 'admin' } }) +src/services/user/userController.ts:306 User.countDocuments(filter) +src/services/user/userRoutes.ts multiple User.countDocuments(...) +src/services/auth/authStore.ts AuthUser.countDocuments implementation +``` + +Interpretation: + +- These are mostly through the `AuthUser` facade, not direct Mongoose imports. +- In `AUTH_STORE=postgres` mode, `AuthUser.countDocuments` uses Postgres. +- This WIP improved the Postgres query-chain behavior and filter aliases. +- Future work should reduce the noisy model-shaped facade API over time, but these are not necessarily active Mongo runtime blockers when auth store is Postgres and Mongo fallback/mirroring are disabled. + +Recommended follow-up: + +- Add a `UserAdminRepo` or explicit auth-store helper methods for admin list/stats to replace model-shaped route code. +- After that, route `src/services/user/userController.ts` and `src/services/user/userRoutes.ts` through helper methods and remove the remaining `User.countDocuments` call sites from route/controller code. + +### Admin Data Cleanup Service + +High-priority blocker: + +```text +src/services/admin/dataCleanupService.ts +``` + +Scan hits include dynamic model counting/deleting: + +```text +Model.countDocuments(query) +Model.deleteMany(query) +User.countDocuments() +PurchaseRequest.countDocuments() +SellerOffer.countDocuments() +Payment.countDocuments() +Chat.countDocuments() +Notification.countDocuments() +RequestTemplate.countDocuments() +Address.countDocuments() +Category.countDocuments() +TempVerification.countDocuments() +User.findByIdAndDelete(userId) +``` + +Interpretation: + +- This is the biggest remaining direct runtime Mongo/model surface. +- It likely imports or dynamically resolves models and is unsuitable for `MONGO_CONNECT_MODE=never`. + +Recommended migration: + +- Replace cleanup stats with repo-backed counts: + - auth/user counts from AuthUser/Postgres helper + - marketplace counts from MarketplaceRepo + - payments from PaymentRepo + - notifications from NotificationRepo + - chat from ChatRepo + - addresses/categories/reviews/temp verification through their Postgres-capable stores +- For destructive cleanup operations, either: + - implement explicit Postgres cleanup repo methods with strong safety guards, or + - disable Mongo-only cleanup actions when Mongo is disabled and return a clear `501`/unsupported result. + +### Store Facades Still Exposing Model-Style Methods + +Scan hits: + +```text +src/services/points/levelConfigStore.ts deleteMany(...) +src/services/address/addressStore.ts findByIdAndDelete(...), countDocuments(...) +src/services/marketplace/reviewStore.ts countDocuments(...) +src/services/auth/authController.ts TempVerification.findByIdAndDelete(...) +src/services/auth/authStore.ts TempVerification findByIdAndDelete implementation +``` + +Interpretation: + +- Some of these are behind Postgres-capable store facades. +- They still show up because they preserve a Mongoose-shaped API. +- For full Mongo removal, these facades must be audited under: + - `MONGO_CONNECT_MODE=never` + - store env set to `postgres` + - fallback/mirror disabled where relevant + +Recommended migration: + +- Convert each store facade from "model-like object with Mongo fallback" to explicit repository functions. +- Add tests that set the store env to `postgres` and assert no Mongo model getter is called. + +### Payment Coordinator + +Scan hit: + +```text +src/services/payment/paymentCoordinator.ts:510 paymentRepo.deleteMany({ idIn: duplicateIds }) +``` + +Interpretation: + +- This is already repository-routed, not raw Mongoose. +- Confirm `DrizzlePaymentRepo.deleteMany` exists and works for the duplicate cleanup path. + +## Suggested Next Work Order + +1. Finish and commit this WIP after review. +2. Run the two new smoke scripts with real dev admin credentials. +3. Update `nick-doc` after pushing backend/frontend. +4. Migrate `src/services/admin/dataCleanupService.ts`. +5. Replace user/admin list/stats route `User.countDocuments` calls with explicit auth/user helper methods so scans no longer flag route-level model-shaped calls. +6. Audit `addressStore`, `reviewStore`, `levelConfigStore`, and `TempVerification` under `MONGO_CONNECT_MODE=never`. +7. Once all runtime paths are Postgres-capable, set health logic so Mongo is optional and then remove startup Mongo requirement. +8. Final pass: + - scan for non-test runtime `mongoose` / `models` imports + - run typecheck + - run focused Jest suites + - run smoke scripts against local/dev + - verify `/api/health` reports Mongo optional or absent and Postgres healthy + +## Environment Notes For Cutover Testing + +Useful envs for a no-Mongo runtime test: + +```text +MONGO_CONNECT_MODE=never +AUTH_STORE=postgres +USER_STORE=postgres +AUTH_FALLBACK_MONGO=false +AUTH_MIRROR_MONGO=false +``` + +Also ensure all existing repo/store envs that support Postgres are set to `postgres` or `pg` consistently, including marketplace/payment/dispute/release-hold/notification/blog/address/category/review/level-config/shop-settings style stores. + +Do not rely only on `rg` results: some model-shaped methods are already routed through Postgres facades. Prove no-Mongo runtime by running the backend with `MONGO_CONNECT_MODE=never` and exercising the API smoke scripts. + +## Known Caveats + +- The new smoke scripts need real admin credentials and were not executed end-to-end. +- The current WIP is not pushed and has no `nick-doc` sync yet. +- `nick-doc` and `deployment` have unrelated dirty files; do not stage them accidentally. +- The full goal is not complete. MongoDB is still a runtime dependency until the remaining service/store paths above are migrated and verified under `MONGO_CONNECT_MODE=never`.