Initial commit: nick docs

04 - Flows/Payment Flow - SHKeeper.md

@@ -0,0 +1,252 @@
+---
+title: Payment Flow - SHKeeper
+tags: [flow, payment, shkeeper, crypto, escrow, webhook]
+related_models: ["[[Payment]]", "[[PurchaseRequest]]", "[[SellerOffer]]"]
+related_apis: ["POST /api/payment/shkeeper/create", "POST /api/payment/shkeeper/webhook", "GET /api/payment/shkeeper/status/:id"]
+---
+
+# Payment Flow — SHKeeper (Crypto Pay-In)
+
+End-to-end **crypto pay-in** via the self-hosted [SHKeeper](https://github.com/vsys-host/shkeeper.io) gateway at `pay.amn.gg`. The buyer pays in stablecoins/crypto; SHKeeper monitors the blockchain, sends webhooks back, and the backend marks the escrow funded.
+
+## Supported assets
+
+Pulled from env: `SHKEEPER_NETWORKS` and `SHKEEPER_ALLOWED_TOKENS` (`shkeeperService.ts:97-98`).
+- **Networks**: `bsc`, `ethereum` (default in code).
+- **Tokens**: `USDT`, `USDC` (default). The endpoint URL is built as `https://pay.amn.gg/api/v1/{NETWORK_PREFIX}-{TOKEN}/payment_request` (`shkeeperService.ts:413-417`).
+  - BSC → `BNB-USDT`, `BNB-USDC` (i.e. BEP-20).
+  - Ethereum → `ETH-USDT`, `ETH-USDC` (ERC-20).
+- TRC-20 (`USDT-TRC20`) and native `BTC` are mentioned in the task brief but **not currently wired** in `shkeeperService.ts` — only BSC/ETH variants are produced from the code path. Verify SHKeeper-side configuration if those are required.
+
+## Actors
+
+- **Buyer** — pays.
+- **Seller** — passive in this flow; gets notified on success.
+- **Frontend** — checkout components under `frontend/src/sections/request/components/buyer-steps/step-3-components/` and `frontend/src/sections/payment/`.
+- **Backend** — `shkeeperService.createPayInIntent` (`backend/src/services/payment/shkeeper/shkeeperService.ts:48-533`) and `shkeeperWebhook.handleShkeeperWebhook` (`backend/src/services/payment/shkeeper/shkeeperWebhook.ts`).
+- **SHKeeper gateway** (`https://pay.amn.gg`) — issues per-payment deposit addresses, watches the chain, sends webhooks.
+- **Blockchain** — BSC / Ethereum.
+- **PaymentCoordinator** (`backend/src/services/payment/paymentCoordinator.ts`) — serialises concurrent payment-status updates from multiple sources (webhook, wallet monitor, manual confirm).
+- **MongoDB** — `payments`, `purchaserequests`, `selleroffers`, `chats`, `notifications`.
+- **Redis** — `paymentRedisService` (wallet-address cache, 2 h TTL).
+- **Socket.IO** — `payment-created`, `seller-offer-update`, `purchase-request-update`.
+
+## Preconditions
+
+- Buyer has selected an offer (or is using the template-checkout shortcut).
+- Backend env: `SHKEEPER_API_URL`, `SHKEEPER_API_KEY`, `SHKEEPER_WEBHOOK_SECRET`, `API_URL` (for the callback URL).
+- Redis is reachable (graceful degradation if not — see `app.ts:361-367`).
+- SHKeeper has wallets provisioned for each `crypto_name`.
+
+## Payment state machine
+
+```mermaid
+stateDiagram-v2
+    [*] --> pending: createPayInIntent\n(Payment.status="pending")
+    pending --> pending_partial: webhook PARTIAL\nescrowState="partial"
+    pending --> completed: webhook PAID/OVERPAID\nescrowState="funded"
+    pending --> failed: webhook EXPIRED/CANCELLED\nescrowState="cancelled"
+    pending_partial --> completed: top-up arrives, total ≥ amount
+    pending_partial --> failed: expires
+    completed --> released: admin release → seller payout\n[[Payout Flow]]
+    completed --> refunded: dispute resolution → buyer refund
+    refunded --> [*]
+    released --> [*]
+    failed --> [*]
+```
+
+## Step-by-step narrative
+
+### Phase 1 — Create intent
+
+1. Buyer clicks "Pay" on the chosen offer (`/dashboard/buyer/requests/{id}` → step-3-select-and-pay).
+2. Frontend POSTs `POST /api/payment/shkeeper/create` with `{ purchaseRequestId, sellerOfferId, amount, token?, network? }`.
+3. Backend `createPayInIntent`:
+   - Validates ObjectIds (`shkeeperService.ts:55-71`). Special path for **template checkout** (string IDs starting with `template-checkout-`).
+   - **Duplicate-guard #1** (`:75-116`): if there is already an active `Payment` (`status ∈ {pending, processing}`) for the same `{purchaseRequestId, sellerOfferId, buyerId}`, the existing record is reused — same `paymentUrl`, no new wallet allocation.
+   - **Duplicate-guard #2 (template checkout)** (`:131-198`): if a recent `completed`/`confirmed` payment exists for the same buyer + template session, reuse it; otherwise dedupe pending records within the last 5 minutes.
+   - **Upsert** (`:218-249`): atomic `Payment.findOneAndUpdate(filter, {$setOnInsert: {...}}, {upsert: true, new: true})` keyed by `{buyerId, purchaseRequestId, provider, direction:'in', status:'pending'}` — prevents race-condition duplicates.
+   - Sets `Payment.providerPaymentId = externalId`. For template checkouts, `externalId = template-{ts}-{rand}`; otherwise it's the `Payment._id`.
+   - **Wallet cache lookup** (`:421-450`): `paymentRedisService.getCachedWallet(cacheKey)` — if a wallet was allocated for the same `(amount, token, network, requestId)` within the last 2 h, reuse it. Avoids hammering SHKeeper for the same checkout.
+   - **Call SHKeeper API** (`:453-475`): `POST https://pay.amn.gg/api/v1/{cryptoName}/payment_request` with header `X-Shkeeper-Api-Key`. Body: `{ external_id, fiat: 'USD', amount: '12.34', callback_url: ${API_URL}/api/payment/shkeeper/webhook }`. The HTTP call is wrapped by `shkeeperFetch` (`shkeeperHealthCheck.ts`) which trips the breaker on repeated failures.
+   - **Persist response** (`:484-503`): updates `Payment.metadata.{shkeeperInvoiceId, shkeeperData, cryptoName, walletAddress}`; caches the wallet for 2 h; calls `walletMonitor.addWallet(...)` so the on-chain watcher can confirm independently (belt-and-braces against missed webhooks).
+4. Returns `{ paymentId, paymentUrl, shkeeperInvoiceId, walletAddress, amount, exchangeRate, displayName, cryptoName }`.
+5. Emits **`payment-created`** globally via `emitGlobalEvent` (`shkeeperService.ts:277-287`) so the admin dashboard sees the new pending payment in real time.
+
+### Phase 2 — Buyer pays
+
+6. Frontend renders a **QR code** for `${walletAddress}?amount=${amount}&token=...` and shows the exchange-rate-locked USDT amount, recalculate-after timer (`recalculate_after` from SHKeeper, typically 15 min), and a copy-to-clipboard button.
+7. Buyer scans with MetaMask / Trust Wallet / Binance App and sends the on-chain transfer.
+8. SHKeeper polls the chain, detects the deposit. When confirmations reach the threshold it marks the invoice `PAID` (or `OVERPAID` if the buyer sent extra).
+
+### Phase 3 — Webhook
+
+9. SHKeeper POSTs to `${API_URL}/api/payment/shkeeper/webhook` with the `ShkeeperWebhookPayload` shape (`shkeeperWebhook.ts:14-37`):
+   ```
+   {
+     external_id, crypto: "BNB-USDT", addr, fiat: "USD",
+     balance_fiat, balance_crypto, paid: true,
+     status: "PAID" | "PENDING" | "EXPIRED" | "CANCELLED" | "PARTIAL" | "OVERPAID",
+     transactions: [{ txid, date, amount_crypto, amount_fiat, trigger: true, ... }],
+     fee_percent, fee_fixed, fee_policy, overpaid_fiat
+   }
+   ```
+10. **Signature verification** (`shkeeperWebhook.ts:84-120`): HMAC-SHA256 of the raw body with `SHKEEPER_WEBHOOK_SECRET`, header `x-shkeeper-signature` (also accepts `x-signature`, `signature`, `x-hub-signature`, `x-hub-signature-256`). Mismatch → `401` in production, allowed in dev. Length-mismatched signature → `401` (avoids `timingSafeEqual` crash).
+11. **Fallback auth** (`:122-141`): if no signature header but env requires it, the route accepts `X-Shkeeper-Api-Key` matching `SHKEEPER_API_KEY`. Otherwise returns `202` to **stop SHKeeper retries** even if rejected (idempotency principle: always 2xx unless the request itself is mangled).
+12. **DB reconnect** (`:143-155`): if Mongoose is disconnected, attempt reconnection. On failure → `202 OK` to avoid retry loop, log for investigation.
+13. **Payment lookup**: `Payment.findOne({ providerPaymentId: payload.external_id })`. If not found and the external_id looks like a template checkout, hand off to `handleTemplateCheckoutWebhook` (`templateCheckoutWebhook.ts`). Otherwise → `202 OK` with a rate-limited log.
+14. **Duplicate-webhook detection** (`:249-296`): if `metadata.shkeeperStatus`, `balance_fiat`, `paid` are identical to the previous webhook **and** less than 10 seconds have passed → return `202` (idempotent). Logged once per minute per payment.
+15. **Map SHKeeper status → internal status** (`:387-411`):
+    | SHKeeper | Internal `status` | `escrowState` |
+    |---|---|---|
+    | `PAID` | `completed` | `funded` |
+    | `OVERPAID` | `completed` | `funded` |
+    | `EXPIRED`, `CANCELLED` | `failed` | `cancelled` |
+    | `PARTIAL` | `pending` | `partial` |
+    | `PENDING` | `pending` | — |
+16. **Extract `transactionHash`** (`:311-385`) — prefers the transaction with `trigger === true`, then falls back to the latest, then to fetching from SHKeeper's invoice endpoint if the webhook somehow arrived without transactions.
+17. **PaymentCoordinator** (`:482-507`) — `coordinatePaymentUpdate` returns false if another worker already started processing this state change, otherwise `executePaymentUpdate` writes the new status/escrowState/txHash atomically with metadata.
+18. **Cascade on PAID/OVERPAID** (`:543-714`):
+    - Load `PurchaseRequest` and `SellerOffer` for this payment.
+    - **Mark winning offer accepted**: `selectedOffer.status = 'accepted'; save()`.
+    - **Reject all other offers**: `SellerOffer.updateMany({ purchaseRequestId, _id: { $ne: sellerOfferId } }, { status: 'rejected' })`.
+    - **Promote request**: `status = 'payment'; selectedOfferId = sellerOfferId`.
+    - **Create direct chat** (`chatService.createChat`, see [[Chat Flow]]).
+    - **Notifications**: `notifyPaymentConfirmed` (to both parties), `notifyOfferAccepted` (winner), `notifyRequestStatusChanged` (`received_offers → payment`).
+    - **Socket fan-out**: `seller-offer-update` `'payment-completed'` to winner, `'offer-rejected'` to losers (each carries `offerId`, `reason`).
+19. **Cleanup**: `simpleAutoWebhook.removePayment(external_id)` stops the simple polling fallback.
+20. Always respond `202 Accepted` (SHKeeper retries on non-2xx). `200` would cause infinite retries because SHKeeper expects `202` per its convention.
+
+### Phase 4 — Frontend reaction
+
+21. The buyer's checkout page subscribes to socket events and polls `GET /api/payment/shkeeper/status/{paymentId}`. When status flips to `completed`, the UI transitions to "Payment received" and unlocks the next buyer step (await delivery).
+22. The seller's dashboard receives `seller-offer-update` `payment-completed` and surfaces the green "Order paid — start preparing" banner.
+
+## Sequence diagram
+
+```mermaid
+sequenceDiagram
+    autonumber
+    actor B as Buyer
+    participant FE as Frontend
+    participant BE as Backend
+    participant R as Redis
+    participant SK as SHKeeper (pay.amn.gg)
+    participant BC as Blockchain
+    participant DB as MongoDB
+    participant IO as Socket.IO
+    actor S as Seller
+
+    B->>FE: Choose offer, click "Pay"
+    FE->>BE: POST /api/payment/shkeeper/create
+    BE->>DB: dedupe / upsert Payment(status:"pending")
+    BE->>R: getCachedWallet(amount, token, network, requestId)
+    alt cache hit
+        R-->>BE: cached wallet
+    else cache miss
+        BE->>SK: POST /api/v1/{cryptoName}/payment_request\nX-Shkeeper-Api-Key
+        SK-->>BE: { id, wallet, amount, exchange_rate, ... }
+        BE->>R: setCachedWallet (TTL 2h)
+        BE->>BE: walletMonitor.addWallet (chain watcher)
+    end
+    BE->>IO: emit 'payment-created' (admin)
+    BE-->>FE: { paymentId, walletAddress, amount, QR-ready data }
+    FE-->>B: Render QR + countdown + copy address
+    B->>BC: Send USDT/USDC to walletAddress
+    BC-->>SK: deposit confirmed
+    SK->>BE: POST /api/payment/shkeeper/webhook\nx-shkeeper-signature
+    BE->>BE: HMAC verify
+    BE->>DB: Payment.findOne({providerPaymentId})
+    BE->>BE: duplicate-webhook check
+    BE->>BE: PaymentCoordinator.coordinate + execute
+    BE->>DB: Payment.status="completed"\nescrowState="funded"\nblockchain.transactionHash=...
+    BE->>DB: SellerOffer.status="accepted" (others "rejected")
+    BE->>DB: PurchaseRequest.status="payment", selectedOfferId
+    BE->>DB: Chat.create (buyer + winning seller)
+    BE->>IO: emit seller-{winner} 'payment-completed'
+    BE->>IO: emit seller-{loser_i} 'offer-rejected'
+    BE-->>SK: 202 OK
+    IO-->>FE: status updated
+    IO-->>S: dashboard updates
+    FE-->>B: "Payment received ✓"
+```
+
+## API calls
+
+| Method | Endpoint | Purpose | Source |
+|---|---|---|---|
+| `POST` | `/api/payment/shkeeper/create` | Create pay-in intent | `shkeeperRoutes.ts` → `shkeeperService.createPayInIntent` |
+| `POST` | `/api/payment/shkeeper/webhook` | Receive SHKeeper webhook | `shkeeperWebhook.handleShkeeperWebhook` |
+| `GET` | `/api/payment/shkeeper/status/:paymentId` | Frontend polling | `shkeeperRoutes.ts` |
+| `GET` | `/api/payment/fetch-tx/:paymentId` | Manual transaction lookup | `paymentRoutes.ts` |
+
+## Database writes
+
+- **`payments`**: insert on intent creation (`status: 'pending'`); update on each webhook (status, escrowState, blockchain.transactionHash, metadata).
+- **`payments`** (unique index `uniq_pending_shkeeper_by_buyer_session`, see `Payment.ts:181-188`): partial unique on `{buyerId, purchaseRequestId, provider:'shkeeper', direction:'in', status:'pending'}` prevents duplicate pending pay-ins.
+- **`selleroffers`**: `status` flipped (`accepted` / `rejected`) by the webhook cascade.
+- **`purchaserequests`**: `status` → `payment`, `selectedOfferId` set.
+- **`chats`**: a new `direct` chat (or reuse) — find-or-create via `ChatService.createChat`.
+- **`notifications`**: 2–N entries depending on parties.
+
+## Socket events emitted
+
+- **`payment-created`** (global) — broadcast on intent creation.
+- **`seller-offer-update`** with `eventType: 'payment-completed'` → winning seller.
+- **`seller-offer-update`** with `eventType: 'offer-rejected'` → each losing seller.
+- **`purchase-request-update`** with `eventType: 'status-changed'` (via `PurchaseRequestService`) → `request-{id}`.
+- **`new-notification`** → both buyer and seller.
+
+## Side effects
+
+- **Redis**: `walletCache:{amount}_{token}_{network}_{requestId}` set for 2 hours.
+- **Wallet monitor**: `walletMonitor.addWallet(addr, amount, paymentId, token, network)` — the on-chain watcher polls BSC/ETH RPCs directly; if SHKeeper's webhook is lost, the monitor still flips the payment to `completed`. This is the redundancy mechanism noted in the comments.
+- **simpleAutoWebhook** (`shkeeperSimpleAuto.ts`): a poll-based fallback that asks SHKeeper for invoice status; cleaned up once a real webhook arrives.
+- **Webhook stats**: `webhookStats.recordWebhook(...)` updates an in-memory ring buffer surfaced via `webhookStats.ts` admin endpoint.
+
+## Error / edge cases
+
+- **Duplicate intent submission** → reuse the existing pending payment (no new wallet). UX-safe.
+- **SHKeeper API unreachable** → `shkeeperFetch` (with circuit breaker) throws; controller returns a **demo fallback** URL (`shkeeperService.ts:520-532`). In production this is observed as a Sentry error.
+- **Webhook signature mismatch in prod** → `401`, SHKeeper retries — usually the secret has rotated; fix env and they catch up.
+- **Webhook missing both signature and API key in prod** → `202 OK` (no-op) to prevent retry storm.
+- **DB disconnected during webhook** → reconnect; on failure `202 OK` + log (consider DLQ).
+- **`PARTIAL` payment** → state held as `pending/partial`; further deposits to the same address are aggregated by SHKeeper and a new webhook arrives.
+- **`OVERPAID`** → treated as `completed/funded`; the overage stays with the platform unless an admin manually refunds (no automatic refund of overpayment today).
+- **`EXPIRED`** → `failed/cancelled`. Buyer can re-initiate; the duplicate-guard will create a fresh intent because the old one is no longer `pending`.
+- **External_id not found** → `202` with rate-limited log; common for orphaned webhooks from old tests.
+- **Webhook arrives twice within 10 s with same data** → idempotency skip → `202`.
+- **`PaymentCoordinator` deferral** → `202` with a "coordinator skipped update" log; the in-flight worker will finish the state change.
+- **Wallet address reuse** — cached for 2 h means two parallel checkouts for the same `(amount, token, network, requestId)` share one address; whichever pays first wins (acceptable since the duplicate-guard reuses the same `Payment` doc anyway).
+- **`crypto_name` mismatch** — only `BNB-*` and `ETH-*` are produced; for TRC-20, additional logic is needed in `shkeeperService.ts:415`.
+
+> [!warning] Webhook returns 202 even on errors
+> The handler always responds 2xx to avoid SHKeeper's retry storm — even for unknown payments, signature failures (in non-production paths), DB errors, and unexpected exceptions. Operationally this means failed-to-process webhooks are silently swallowed unless someone tails the logs. Hook the catch-all into Sentry severity = `error` and alert on `webhookStats.errorCount`.
+
+> [!tip] Manual reconciliation
+> Use `fix-transaction-hashes.js` at repo root to backfill `blockchain.transactionHash` for payments where the webhook arrived without transactions. See [[Payout Flow]] for the parallel payout-side script usage.
+
+## Linked flows
+
+- [[Purchase Request Flow]] — supplies the request being paid for.
+- [[Seller Offer Flow]] — supplies the offer being accepted.
+- [[Payment Flow - DePay & Web3]] — alternative direct-wallet route.
+- [[Escrow Flow]] — what `escrowState=funded` means downstream.
+- [[Chat Flow]] — auto-created on success.
+- [[Notification Flow]] — both parties pinged.
+- [[Payout Flow]] — pays the seller from the funded escrow.
+- [[Dispute Flow]] — escape if the order goes wrong.
+
+## Source files
+
+- Backend: `backend/src/services/payment/shkeeper/shkeeperService.ts` (intent creation, ~650 lines)
+- Backend: `backend/src/services/payment/shkeeper/shkeeperWebhook.ts` (webhook handler, ~750 lines)
+- Backend: `backend/src/services/payment/shkeeper/shkeeperHealthCheck.ts` (circuit breaker)
+- Backend: `backend/src/services/payment/shkeeper/shkeeperRoutes.ts`
+- Backend: `backend/src/services/payment/paymentCoordinator.ts`
+- Backend: `backend/src/services/payment/cleanupPendingPayments.ts` (periodic GC)
+- Backend: `backend/src/services/blockchain/walletMonitor.ts` (chain watcher)
+- Backend: `backend/src/services/redis/paymentRedisService.ts` (wallet cache)
+- Backend: `backend/src/models/Payment.ts`
+- Frontend: `frontend/src/sections/request/components/buyer-steps/step-3-components/step-3-payment.tsx`
+- Frontend: `frontend/src/sections/payment/`