docs: add MongoDB removal handoff document with updated versions
Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
This commit is contained in:
Siavash Sameni
454
08 - Operations/MONGODB_REMOVAL_HANDOFF_2026-06-02.md
Normal file
454
08 - Operations/MONGODB_REMOVAL_HANDOFF_2026-06-02.md
Normal file
@@ -0,0 +1,454 @@
|
||||
# MongoDB Runtime Removal Handoff
|
||||
|
||||
Date: 2026-06-02
|
||||
Workspace: `/Users/manwe/CascadeProjects/escrow`
|
||||
Goal: remove MongoDB as a runtime dependency by migrating remaining Mongo-backed backend domains and cutover paths to Postgres-compatible repositories.
|
||||
|
||||
## Current State
|
||||
|
||||
No commits or pushes were made for the current WIP. The work is local only.
|
||||
|
||||
Repo heads at handoff time:
|
||||
|
||||
| Repo | Branch | HEAD | State |
|
||||
| --- | --- | --- | --- |
|
||||
| `backend` | `integrate-main-into-development` | `cf59726` | dirty WIP |
|
||||
| `frontend` | `integrate-main-into-development` | `a2b972b` | dirty version bump only |
|
||||
| `nick-doc` | `main` | `345c585` | dirty unrelated local docs/profile files |
|
||||
| `deployment` | `main` | `8764fdf` | dirty unrelated `.env` and `docker-compose.yml` |
|
||||
|
||||
Backend/frontend package versions: backend at `2.8.79`, frontend at `2.8.94`.
|
||||
|
||||
Important repo rules:
|
||||
|
||||
- Any backend/frontend product change requires patch version bump in both repos.
|
||||
- Before any backend push, run the relevant focused tests and smoke script.
|
||||
- After every backend push, sync `nick-doc`:
|
||||
- append `09 - Audits/Activity Log.md`
|
||||
- update relevant data/architecture docs
|
||||
- commit as `docs: sync from backend <short-sha> — <summary>`
|
||||
- push `nick-doc`
|
||||
- Do not stage unrelated dirty files in `nick-doc` or `deployment`.
|
||||
|
||||
## Dirty Files
|
||||
|
||||
Backend dirty files:
|
||||
|
||||
```text
|
||||
package-lock.json
|
||||
package.json
|
||||
src/db/repositories/drizzle/DrizzleMarketplaceRepo.ts
|
||||
src/db/repositories/drizzle/DrizzlePaymentRepo.ts
|
||||
src/db/repositories/dual/DualWriteMarketplaceRepo.ts
|
||||
src/db/repositories/dual/DualWritePaymentRepo.ts
|
||||
src/db/repositories/interfaces/IMarketplaceRepo.ts
|
||||
src/db/repositories/interfaces/IPaymentRepo.ts
|
||||
src/db/repositories/mongo/MongoMarketplaceRepo.ts
|
||||
src/db/repositories/mongo/MongoPaymentRepo.ts
|
||||
src/services/auth/authStore.ts
|
||||
src/services/user/userController.ts
|
||||
__tests__/auth-store-pg-query.test.ts
|
||||
__tests__/user-dependencies-repo.test.ts
|
||||
scripts/smoke/user-admin-postgres.sh
|
||||
scripts/smoke/user-dependencies.sh
|
||||
```
|
||||
|
||||
Frontend dirty files:
|
||||
|
||||
```text
|
||||
package.json
|
||||
```
|
||||
|
||||
## What This WIP Changes
|
||||
|
||||
### 1. Admin User Dependencies Endpoint
|
||||
|
||||
Endpoint affected:
|
||||
|
||||
```text
|
||||
GET /api/user/admin/:userId/dependencies
|
||||
```
|
||||
|
||||
Before this WIP, `src/services/user/userController.ts` dynamically imported Mongo models and counted dependencies directly:
|
||||
|
||||
- `RequestTemplate.countDocuments(...)`
|
||||
- `PurchaseRequest.countDocuments(...)`
|
||||
- `Payment.countDocuments(...)`
|
||||
- `Chat.countDocuments(...)`
|
||||
|
||||
This WIP replaces that direct model access with repository calls:
|
||||
|
||||
- `getMarketplaceRepo().getUserDependencyCounts(userId)`
|
||||
- `getPaymentRepo().countByParticipant(userId)`
|
||||
- `getChatRepo().count({ 'participants.userId': userId, 'participants.isActive': true })`
|
||||
|
||||
New/extended repository contract methods:
|
||||
|
||||
- `IMarketplaceRepo.getUserDependencyCounts(userId)`
|
||||
- `IPaymentRepo.countByParticipant(userId)`
|
||||
|
||||
Implementations added:
|
||||
|
||||
- `MongoMarketplaceRepo.getUserDependencyCounts`
|
||||
- `DrizzleMarketplaceRepo.getUserDependencyCounts`
|
||||
- `DualWriteMarketplaceRepo.getUserDependencyCounts`
|
||||
- `MongoPaymentRepo.countByParticipant`
|
||||
- `DrizzlePaymentRepo.countByParticipant`
|
||||
- `DualWritePaymentRepo.countByParticipant`
|
||||
|
||||
Behavior note:
|
||||
|
||||
- Postgres counts seller-side marketplace dependencies by joining `purchase_requests.selected_offer_id` to `seller_offers.id` and checking `seller_offers.seller_id`.
|
||||
- Mongo implementation supports selected-offer-id style and also keeps compatibility with legacy embedded `selectedOffer.sellerId`.
|
||||
|
||||
New test:
|
||||
|
||||
```text
|
||||
__tests__/user-dependencies-repo.test.ts
|
||||
```
|
||||
|
||||
New smoke script:
|
||||
|
||||
```text
|
||||
scripts/smoke/user-dependencies.sh
|
||||
```
|
||||
|
||||
Smoke usage:
|
||||
|
||||
```bash
|
||||
BASE_URL=https://dev.amn.gg ADMIN_TOKEN=<admin-jwt> USER_ID=<target-user-id> scripts/smoke/user-dependencies.sh
|
||||
```
|
||||
|
||||
The smoke script checks:
|
||||
|
||||
- HTTP 200
|
||||
- `success === true`
|
||||
- `data.user` exists
|
||||
- dependency counters are non-negative numbers
|
||||
- `total` equals the sum of component counters
|
||||
|
||||
### 2. AuthUser Postgres Query Facade Hardening
|
||||
|
||||
Files affected:
|
||||
|
||||
```text
|
||||
src/services/auth/authStore.ts
|
||||
__tests__/auth-store-pg-query.test.ts
|
||||
```
|
||||
|
||||
The mounted user/admin list and stats routes already use `AuthUser` from `authStore`, not raw Mongoose imports. However, the Postgres `PgQuery` wrapper only sorted arrays by `createdAt`; admin list accepts arbitrary `sortBy`.
|
||||
|
||||
This WIP hardens the Postgres query wrapper so `AuthUser.find(...).select(...).sort(...).skip(...).limit(...).lean()` behaves more like the existing Mongoose chain:
|
||||
|
||||
- generic sorting by requested field
|
||||
- nested path sorting support, e.g. `profile.avatar`
|
||||
- date, number, boolean, and string comparison
|
||||
- multi-field sort support
|
||||
- keeps existing skip/limit/select/lean chain behavior
|
||||
|
||||
It also adds alias support in `buildUserWhere`:
|
||||
|
||||
- `isActive: true` maps to `status = 'active'`
|
||||
- `isActive: false` maps to `status <> 'active'`
|
||||
- `isVerified` maps to `is_email_verified`
|
||||
|
||||
This matters because:
|
||||
|
||||
- `src/services/user/userController.ts` builds filters with `isActive` / `isVerified`
|
||||
- `src/services/user/userRoutes.ts` builds filters with `status` / `isEmailVerified`
|
||||
- both now work in Postgres auth mode
|
||||
|
||||
New test:
|
||||
|
||||
```text
|
||||
__tests__/auth-store-pg-query.test.ts
|
||||
```
|
||||
|
||||
New smoke script:
|
||||
|
||||
```text
|
||||
scripts/smoke/user-admin-postgres.sh
|
||||
```
|
||||
|
||||
Smoke usage:
|
||||
|
||||
```bash
|
||||
BASE_URL=https://dev.amn.gg ADMIN_TOKEN=<admin-jwt> scripts/smoke/user-admin-postgres.sh
|
||||
```
|
||||
|
||||
The smoke script checks:
|
||||
|
||||
- `GET /api/user/admin/list?page=1&limit=5&sortBy=firstName&sortOrder=asc`
|
||||
- `GET /api/users/admin/stats`
|
||||
- response shape and numeric stats
|
||||
|
||||
## Verification Already Run
|
||||
|
||||
Passed:
|
||||
|
||||
```bash
|
||||
npm test -- --runTestsByPath __tests__/auth-store-pg-query.test.ts __tests__/user-dependencies-repo.test.ts __tests__/repository-factory-modes.test.ts __tests__/health-check-service.test.ts __tests__/marketplace-runtime-import-surface.test.ts --runInBand
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
```text
|
||||
Test Suites: 5 passed, 5 total
|
||||
Tests: 11 passed, 11 total
|
||||
```
|
||||
|
||||
Passed:
|
||||
|
||||
```bash
|
||||
npm run typecheck
|
||||
```
|
||||
|
||||
Passed:
|
||||
|
||||
```bash
|
||||
git diff --check
|
||||
```
|
||||
|
||||
for both backend and frontend.
|
||||
|
||||
Passed syntax checks:
|
||||
|
||||
```bash
|
||||
bash -n scripts/smoke/user-admin-postgres.sh
|
||||
bash -n scripts/smoke/user-dependencies.sh
|
||||
```
|
||||
|
||||
Not run end-to-end:
|
||||
|
||||
- `scripts/smoke/user-dependencies.sh`
|
||||
- `scripts/smoke/user-admin-postgres.sh`
|
||||
|
||||
Reason: both require an `ADMIN_TOKEN`; `user-dependencies.sh` also requires `USER_ID`.
|
||||
|
||||
## How To Pick This Up
|
||||
|
||||
Start with:
|
||||
|
||||
```bash
|
||||
cd /Users/manwe/CascadeProjects/escrow/backend
|
||||
git status --short --branch
|
||||
git diff --stat
|
||||
```
|
||||
|
||||
Review the exact WIP:
|
||||
|
||||
```bash
|
||||
git diff -- src/services/user/userController.ts src/services/auth/authStore.ts
|
||||
git diff -- src/db/repositories/interfaces/IMarketplaceRepo.ts src/db/repositories/interfaces/IPaymentRepo.ts
|
||||
git diff -- src/db/repositories/mongo/MongoMarketplaceRepo.ts src/db/repositories/drizzle/DrizzleMarketplaceRepo.ts src/db/repositories/dual/DualWriteMarketplaceRepo.ts
|
||||
git diff -- src/db/repositories/mongo/MongoPaymentRepo.ts src/db/repositories/drizzle/DrizzlePaymentRepo.ts src/db/repositories/dual/DualWritePaymentRepo.ts
|
||||
git diff -- __tests__/auth-store-pg-query.test.ts __tests__/user-dependencies-repo.test.ts
|
||||
git diff -- scripts/smoke/user-admin-postgres.sh scripts/smoke/user-dependencies.sh
|
||||
```
|
||||
|
||||
Re-run local verification:
|
||||
|
||||
```bash
|
||||
npm test -- --runTestsByPath __tests__/auth-store-pg-query.test.ts __tests__/user-dependencies-repo.test.ts __tests__/repository-factory-modes.test.ts __tests__/health-check-service.test.ts __tests__/marketplace-runtime-import-surface.test.ts --runInBand
|
||||
npm run typecheck
|
||||
git diff --check
|
||||
```
|
||||
|
||||
If you have a dev admin token:
|
||||
|
||||
```bash
|
||||
BASE_URL=https://dev.amn.gg ADMIN_TOKEN=<admin-jwt> USER_ID=<target-user-id> scripts/smoke/user-dependencies.sh
|
||||
BASE_URL=https://dev.amn.gg ADMIN_TOKEN=<admin-jwt> scripts/smoke/user-admin-postgres.sh
|
||||
```
|
||||
|
||||
If you commit this WIP, suggested commit shape:
|
||||
|
||||
Backend:
|
||||
|
||||
```bash
|
||||
git add package.json package-lock.json \
|
||||
src/db/repositories/interfaces/IMarketplaceRepo.ts \
|
||||
src/db/repositories/interfaces/IPaymentRepo.ts \
|
||||
src/db/repositories/mongo/MongoMarketplaceRepo.ts \
|
||||
src/db/repositories/drizzle/DrizzleMarketplaceRepo.ts \
|
||||
src/db/repositories/dual/DualWriteMarketplaceRepo.ts \
|
||||
src/db/repositories/mongo/MongoPaymentRepo.ts \
|
||||
src/db/repositories/drizzle/DrizzlePaymentRepo.ts \
|
||||
src/db/repositories/dual/DualWritePaymentRepo.ts \
|
||||
src/services/user/userController.ts \
|
||||
src/services/auth/authStore.ts \
|
||||
__tests__/auth-store-pg-query.test.ts \
|
||||
__tests__/user-dependencies-repo.test.ts \
|
||||
scripts/smoke/user-admin-postgres.sh \
|
||||
scripts/smoke/user-dependencies.sh
|
||||
|
||||
git commit -m "fix: route admin user counts through postgres-capable stores"
|
||||
```
|
||||
|
||||
Frontend:
|
||||
|
||||
```bash
|
||||
cd ../frontend
|
||||
git add package.json
|
||||
git commit -m "chore: sync frontend version to 2.8.38"
|
||||
```
|
||||
|
||||
Do not push without doing the `nick-doc` sync afterward.
|
||||
|
||||
## Remaining Runtime Mongo Scan
|
||||
|
||||
Latest scan command:
|
||||
|
||||
```bash
|
||||
rg -n --pcre2 "^import (?!type).*from ['\"]mongoose['\"]|^import (?!type).*from ['\"][^'\"]*models/|await import\(['\"][^'\"]*models/|countDocuments\(|deleteMany\(|findByIdAndDelete\(" \
|
||||
src/services src/routes src/app.ts src/infrastructure src/db/repositories/factory.ts \
|
||||
--glob '!**/*.test.ts' \
|
||||
--glob '!src/services/marketplace/routes.ts'
|
||||
```
|
||||
|
||||
Important results and interpretation:
|
||||
|
||||
### Auth/User Routes
|
||||
|
||||
Paths still visible in scans:
|
||||
|
||||
```text
|
||||
src/app.ts:672 AuthUser.countDocuments({ role: { $ne: 'admin' } })
|
||||
src/services/user/userController.ts:306 User.countDocuments(filter)
|
||||
src/services/user/userRoutes.ts multiple User.countDocuments(...)
|
||||
src/services/auth/authStore.ts AuthUser.countDocuments implementation
|
||||
```
|
||||
|
||||
Interpretation:
|
||||
|
||||
- These are mostly through the `AuthUser` facade, not direct Mongoose imports.
|
||||
- In `AUTH_STORE=postgres` mode, `AuthUser.countDocuments` uses Postgres.
|
||||
- This WIP improved the Postgres query-chain behavior and filter aliases.
|
||||
- Future work should reduce the noisy model-shaped facade API over time, but these are not necessarily active Mongo runtime blockers when auth store is Postgres and Mongo fallback/mirroring are disabled.
|
||||
|
||||
Recommended follow-up:
|
||||
|
||||
- Add a `UserAdminRepo` or explicit auth-store helper methods for admin list/stats to replace model-shaped route code.
|
||||
- After that, route `src/services/user/userController.ts` and `src/services/user/userRoutes.ts` through helper methods and remove the remaining `User.countDocuments` call sites from route/controller code.
|
||||
|
||||
### Admin Data Cleanup Service
|
||||
|
||||
High-priority blocker:
|
||||
|
||||
```text
|
||||
src/services/admin/dataCleanupService.ts
|
||||
```
|
||||
|
||||
Scan hits include dynamic model counting/deleting:
|
||||
|
||||
```text
|
||||
Model.countDocuments(query)
|
||||
Model.deleteMany(query)
|
||||
User.countDocuments()
|
||||
PurchaseRequest.countDocuments()
|
||||
SellerOffer.countDocuments()
|
||||
Payment.countDocuments()
|
||||
Chat.countDocuments()
|
||||
Notification.countDocuments()
|
||||
RequestTemplate.countDocuments()
|
||||
Address.countDocuments()
|
||||
Category.countDocuments()
|
||||
TempVerification.countDocuments()
|
||||
User.findByIdAndDelete(userId)
|
||||
```
|
||||
|
||||
Interpretation:
|
||||
|
||||
- This is the biggest remaining direct runtime Mongo/model surface.
|
||||
- It likely imports or dynamically resolves models and is unsuitable for `MONGO_CONNECT_MODE=never`.
|
||||
|
||||
Recommended migration:
|
||||
|
||||
- Replace cleanup stats with repo-backed counts:
|
||||
- auth/user counts from AuthUser/Postgres helper
|
||||
- marketplace counts from MarketplaceRepo
|
||||
- payments from PaymentRepo
|
||||
- notifications from NotificationRepo
|
||||
- chat from ChatRepo
|
||||
- addresses/categories/reviews/temp verification through their Postgres-capable stores
|
||||
- For destructive cleanup operations, either:
|
||||
- implement explicit Postgres cleanup repo methods with strong safety guards, or
|
||||
- disable Mongo-only cleanup actions when Mongo is disabled and return a clear `501`/unsupported result.
|
||||
|
||||
### Store Facades Still Exposing Model-Style Methods
|
||||
|
||||
Scan hits:
|
||||
|
||||
```text
|
||||
src/services/points/levelConfigStore.ts deleteMany(...)
|
||||
src/services/address/addressStore.ts findByIdAndDelete(...), countDocuments(...)
|
||||
src/services/marketplace/reviewStore.ts countDocuments(...)
|
||||
src/services/auth/authController.ts TempVerification.findByIdAndDelete(...)
|
||||
src/services/auth/authStore.ts TempVerification findByIdAndDelete implementation
|
||||
```
|
||||
|
||||
Interpretation:
|
||||
|
||||
- Some of these are behind Postgres-capable store facades.
|
||||
- They still show up because they preserve a Mongoose-shaped API.
|
||||
- For full Mongo removal, these facades must be audited under:
|
||||
- `MONGO_CONNECT_MODE=never`
|
||||
- store env set to `postgres`
|
||||
- fallback/mirror disabled where relevant
|
||||
|
||||
Recommended migration:
|
||||
|
||||
- Convert each store facade from "model-like object with Mongo fallback" to explicit repository functions.
|
||||
- Add tests that set the store env to `postgres` and assert no Mongo model getter is called.
|
||||
|
||||
### Payment Coordinator
|
||||
|
||||
Scan hit:
|
||||
|
||||
```text
|
||||
src/services/payment/paymentCoordinator.ts:510 paymentRepo.deleteMany({ idIn: duplicateIds })
|
||||
```
|
||||
|
||||
Interpretation:
|
||||
|
||||
- This is already repository-routed, not raw Mongoose.
|
||||
- Confirm `DrizzlePaymentRepo.deleteMany` exists and works for the duplicate cleanup path.
|
||||
|
||||
## Suggested Next Work Order
|
||||
|
||||
1. Finish and commit this WIP after review.
|
||||
2. Run the two new smoke scripts with real dev admin credentials.
|
||||
3. Update `nick-doc` after pushing backend/frontend.
|
||||
4. Migrate `src/services/admin/dataCleanupService.ts`.
|
||||
5. Replace user/admin list/stats route `User.countDocuments` calls with explicit auth/user helper methods so scans no longer flag route-level model-shaped calls.
|
||||
6. Audit `addressStore`, `reviewStore`, `levelConfigStore`, and `TempVerification` under `MONGO_CONNECT_MODE=never`.
|
||||
7. Once all runtime paths are Postgres-capable, set health logic so Mongo is optional and then remove startup Mongo requirement.
|
||||
8. Final pass:
|
||||
- scan for non-test runtime `mongoose` / `models` imports
|
||||
- run typecheck
|
||||
- run focused Jest suites
|
||||
- run smoke scripts against local/dev
|
||||
- verify `/api/health` reports Mongo optional or absent and Postgres healthy
|
||||
|
||||
## Environment Notes For Cutover Testing
|
||||
|
||||
Useful envs for a no-Mongo runtime test:
|
||||
|
||||
```text
|
||||
MONGO_CONNECT_MODE=never
|
||||
AUTH_STORE=postgres
|
||||
USER_STORE=postgres
|
||||
AUTH_FALLBACK_MONGO=false
|
||||
AUTH_MIRROR_MONGO=false
|
||||
```
|
||||
|
||||
Also ensure all existing repo/store envs that support Postgres are set to `postgres` or `pg` consistently, including marketplace/payment/dispute/release-hold/notification/blog/address/category/review/level-config/shop-settings style stores.
|
||||
|
||||
Do not rely only on `rg` results: some model-shaped methods are already routed through Postgres facades. Prove no-Mongo runtime by running the backend with `MONGO_CONNECT_MODE=never` and exercising the API smoke scripts.
|
||||
|
||||
## Known Caveats
|
||||
|
||||
- The new smoke scripts need real admin credentials and were not executed end-to-end.
|
||||
- The current WIP is not pushed and has no `nick-doc` sync yet.
|
||||
- `nick-doc` and `deployment` have unrelated dirty files; do not stage them accidentally.
|
||||
- The full goal is not complete. MongoDB is still a runtime dependency until the remaining service/store paths above are migrated and verified under `MONGO_CONNECT_MODE=never`.
|
||||
Reference in New Issue
Block a user