docs: sync from backend ca2b1c4 — marketplace e2e smoke runner
This commit is contained in:
Siavash Sameni
185
11 - Testing/Marketplace E2E Smoke Runner.md
Normal file
185
11 - Testing/Marketplace E2E Smoke Runner.md
Normal file
@@ -0,0 +1,185 @@
|
||||
---
|
||||
title: Marketplace E2E Smoke Runner
|
||||
tags: [testing, smoke, e2e, notifications, performance]
|
||||
created: 2026-06-06
|
||||
---
|
||||
|
||||
# Marketplace E2E Smoke Runner
|
||||
|
||||
The backend repo has a committed marketplace runner:
|
||||
|
||||
```bash
|
||||
backend/scripts/smoke/marketplace-e2e-notifications.sh
|
||||
```
|
||||
|
||||
It creates isolated test actors, runs buyer/seller marketplace flows, checks
|
||||
persisted notifications after every business mutation, and writes JSON/Markdown
|
||||
reports under ignored `backend/tmp/e2e-reports/`.
|
||||
|
||||
## Required Environment
|
||||
|
||||
Do not put secrets in command history or docs. Load these from an ignored local
|
||||
env file or a secure shell session:
|
||||
|
||||
| Variable | Purpose |
|
||||
|---|---|
|
||||
| `BASE_URL` | Target API, for example `https://dev.amn.gg`. Defaults to local `http://127.0.0.1:5001`. |
|
||||
| `ADMIN_EMAIL` | Admin account email used only to create test users. |
|
||||
| `ADMIN_PASSWORD` | Admin account password, never committed or printed. |
|
||||
| `PAYMENT_MODE` | `status`, `record`, or `live`. Defaults to `status`. |
|
||||
| `CONCURRENCY_LEVELS` | Comma-separated worker ramp, for example `1,2,4,8`. |
|
||||
| `ROUNDS` | Rounds per worker. Use `2` for the basic two-round requirement. |
|
||||
| `SELLERS_PER_WORKER` | Sellers per flow. Minimum enforced by runner is `2`; default is `3`. |
|
||||
| `SKIP_WRONG_RECIPIENT_CHECK` | Optional escape hatch for rate-limited environments. Defaults to `false`. |
|
||||
|
||||
## Payment Modes
|
||||
|
||||
| Mode | Use | Notes |
|
||||
|---|---|---|
|
||||
| `status` | Cheap full-flow smoke without wallet funds. | Moves the request to `payment` through the status route, then exercises seller delivery and buyer confirmation. |
|
||||
| `record` | Tests legacy `POST /api/marketplace/payments`. | Current dev run shows this route returns HTTP 500 with generated buyer/seller ids. Keep this mode for regression coverage. |
|
||||
| `live` | Tests BSC Testnet tUSDT direct-balance scanner rail. | Requires `E2E_PAYMENT_PRIVATE_KEY` or `E2E_PAYMENT_MNEMONIC`, gas, and tUSDT. Sends ERC-20 transfer and calls direct-balance check. |
|
||||
|
||||
Live mode also reads:
|
||||
|
||||
| Variable | Default |
|
||||
|---|---|
|
||||
| `E2E_NETWORK` | `bsc-testnet` |
|
||||
| `E2E_CHAIN_ID` | `97` |
|
||||
| `E2E_TOKEN_SYMBOL` | `USDT` |
|
||||
| `RPC_URL_CHAIN_97` / `BSC_TESTNET_RPC_URL` | public BSC Testnet RPC fallback |
|
||||
|
||||
## Example Commands
|
||||
|
||||
Two-round full-flow smoke without wallet funds:
|
||||
|
||||
```bash
|
||||
BASE_URL=https://dev.amn.gg \
|
||||
PAYMENT_MODE=status \
|
||||
CONCURRENCY_LEVELS=1 \
|
||||
ROUNDS=2 \
|
||||
bash scripts/smoke/marketplace-e2e-notifications.sh
|
||||
```
|
||||
|
||||
Concurrency profile starting at one worker:
|
||||
|
||||
```bash
|
||||
BASE_URL=https://dev.amn.gg \
|
||||
PAYMENT_MODE=status \
|
||||
CONCURRENCY_LEVELS=1,2,4,8,16,32 \
|
||||
ROUNDS=1 \
|
||||
bash scripts/smoke/marketplace-e2e-notifications.sh
|
||||
```
|
||||
|
||||
Live BSC Testnet tUSDT smoke:
|
||||
|
||||
```bash
|
||||
BASE_URL=https://dev.amn.gg \
|
||||
PAYMENT_MODE=live \
|
||||
CONCURRENCY_LEVELS=1 \
|
||||
ROUNDS=1 \
|
||||
bash scripts/smoke/marketplace-e2e-notifications.sh
|
||||
```
|
||||
|
||||
## Notification Behavior
|
||||
|
||||
The runner treats each state-changing business step as a notification checkpoint.
|
||||
Each checkpoint records:
|
||||
|
||||
- actor and role;
|
||||
- expected related id;
|
||||
- whether a persisted notification was found;
|
||||
- whether the missing notification is a known gap;
|
||||
- unread count from the notification list response.
|
||||
|
||||
By default, known gaps do not abort the flow. Set `STRICT_NOTIFICATIONS=true`
|
||||
to make known gaps fail the run. Set `ABORT_ON_NOTIFICATION_FAILURE=true` to
|
||||
stop at the first missing notification.
|
||||
|
||||
## Output
|
||||
|
||||
Reports are ignored by Git:
|
||||
|
||||
```text
|
||||
backend/tmp/e2e-reports/marketplace-e2e-<runId>.json
|
||||
backend/tmp/e2e-reports/marketplace-e2e-<runId>.md
|
||||
```
|
||||
|
||||
The report includes:
|
||||
|
||||
- flow pass/fail counts;
|
||||
- notification found/missing/known-gap counts;
|
||||
- wrong-recipient notification count;
|
||||
- API p50/p95/p99/max timings;
|
||||
- request, offer, payment, token, and transaction ids where applicable.
|
||||
|
||||
Actor tokens and admin credentials are never written to the report.
|
||||
|
||||
## Reference Dev Run - 2026-06-06
|
||||
|
||||
Command shape:
|
||||
|
||||
```bash
|
||||
BASE_URL=https://dev.amn.gg PAYMENT_MODE=record CONCURRENCY_LEVELS=1 ROUNDS=2 ...
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
| Metric | Value |
|
||||
|---|---:|
|
||||
| flows passed | `0/2` |
|
||||
| request-created notifications found | `8/8` |
|
||||
| buyer new-offer notifications found | `6/6` |
|
||||
| rejected-seller notifications found | `4/4` |
|
||||
| selected-seller accepted notifications by related id | `0/2` |
|
||||
| wrong-recipient notifications | `0` |
|
||||
| legacy marketplace payment records | `0/2`, HTTP 500 |
|
||||
|
||||
Observed defects:
|
||||
|
||||
1. Selected-seller accepted notification is not discoverable by the selected
|
||||
offer `relatedId`. The service path appears to create the accepted-offer
|
||||
notification with an incomplete related id.
|
||||
2. `POST /api/marketplace/payments` returned HTTP 500 after offer selection
|
||||
with generated test actors. This blocks `PAYMENT_MODE=record`; use
|
||||
`PAYMENT_MODE=status` for flow-only smoke or `PAYMENT_MODE=live` for scanner
|
||||
verification.
|
||||
3. Dev API rate limiting is `100` requests per `900s`; concurrency runs must
|
||||
use a fresh window or a non-rate-limited test environment.
|
||||
|
||||
## Reference Direct-Backend Run - 2026-06-06
|
||||
|
||||
Command shape:
|
||||
|
||||
```bash
|
||||
BASE_URL=http://127.0.0.1:15001 PAYMENT_MODE=status CONCURRENCY_LEVELS=1 ROUNDS=1 ...
|
||||
```
|
||||
|
||||
This used a temporary SSH tunnel to the dev backend container to avoid the
|
||||
public edge rate-limit window.
|
||||
|
||||
Result:
|
||||
|
||||
| Metric | Value |
|
||||
|---|---:|
|
||||
| flows passed | `1/1` |
|
||||
| final request status | `delivered` |
|
||||
| wrong-recipient notifications | `0` |
|
||||
| required notification missing | `1` |
|
||||
| known notification gaps | `2` |
|
||||
|
||||
Key ids:
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| purchaseRequestId | `dc6ec076-9c15-46a1-ab6b-1d9d17604614` |
|
||||
| selectedOfferId | `9f56dd00-8b08-4c9f-b91c-5875e1949113` |
|
||||
|
||||
Observed notification result:
|
||||
|
||||
- request-created notifications passed for buyer and three targeted sellers;
|
||||
- buyer new-offer notifications passed for all three seller bids;
|
||||
- rejected-seller notifications passed;
|
||||
- selected-seller accepted notification still failed by selected offer `relatedId`;
|
||||
- payment-confirmed notification was found in `PAYMENT_MODE=status`;
|
||||
- seller delivery and buyer delivery confirmation notifications are known gaps.
|
||||
Reference in New Issue
Block a user