nick/nick-doc
3
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs: add sub-project service docs + sync vault 2026-06-08

Browse Source 
Add 10 - Services/ docs for all sub-projects: backend, frontend, scanner,
deployment (new), update amanat-assist. Update Scanner Architecture,
Telegram Mini App flow, and Activity Log. Add payment safety edge cases.
This commit is contained in:
Siavash Sameni
2026-06-08 16:22:52 +04:00
parent 181e8e9c2f
commit 67244223ec
13 changed files with 2734 additions and 311 deletions
Download Patch File Download Diff File Expand all files Collapse all files

295
01 - Architecture/Scanner Architecture.md
View File

@@ -2,31 +2,95 @@
title: Scanner Architecture
tags: [architecture, scanner, payment]
created: 2026-05-30
updated: 2026-06-08
---
# Scanner Architecture
AMN Pay Scanner is a standalone Go microservice that watches on-chain payment events and notifies the backend via webhook when a payment is confirmed. It replaces the Request Network integration with an in-house polling scanner that supports EVM chains, Tron, and TON.
AMN Pay Scanner is a standalone Go microservice that watches on-chain payment events and notifies the backend via signed webhook when a payment is confirmed. It replaces the Request Network integration with an in-house polling scanner that supports EVM chains, Tron, and TON.
> [!info]
> Repo: `scanner/` within the escrow monorepo. Binary: `scanner`. Written in Go 1.25. SQLite (WAL mode) for state. No external dependencies beyond the chain APIs. Scanner `0.1.8` also supports direct-address EVM ERC-20 balance reads and balance watches for non-smart-contract payment rails.
> Repo: `scanner/` within the escrow monorepo. Binary: `scanner`. Written in Go 1.25. SQLite (WAL mode) for state. No external dependencies beyond the chain APIs.
>
> For operational how-it-works detail (API, webhook payloads, config vars, direct balance checks) see [[scanner]] in 10 - Services.
---
## 1. Responsibilities
- Accept payment **intents** from the backend (POST /intents)
- Accept payment **intents** from the backend (`POST /intents`)
- Watch the relevant chain for matching on-chain transfers
- Track confirmation depth (EVM) or rely on finality from the chain API (Tron, TON)
- Track confirmation depth (EVM) or rely on API-reported finality (Tron, TON)
- Deliver a signed webhook to the backend callback URL when confirmed
- Retry failed webhook deliveries
- Retry failed webhook deliveries with exponential back-off
- Expire stale pending intents on a configurable TTL
- Read an EVM ERC-20 balance on demand for a backend-supplied address/token
- Watch an EVM address/token pair for balance changes, decay polling cadence over time, and stop after backend cancellation or 7-day expiry
- Read an EVM ERC-20 balance on demand (`POST /balances/check`)
- Watch an EVM address/token pair for balance changes with age-decayed polling cadence (`POST /balance-watches`)
---
## 2. Component map
## 2. Supported chains
Chains are defined in `supported-chains.json`. A worker is spawned only for chains with `"verified": true` (or listed in `SCANNER_ENABLED_CHAINS`).
| Chain | Chain ID | Type | Proxy address | Conf. threshold | Active by default |
|---|---|---|---|---|---|
| BNB Smart Chain | 56 | EVM | `0x0DfbEe143b42B41eFC5A6F87bFD1fFC78c2f0aC9` | 200 blocks | Yes |
| Ethereum Mainnet | 1 | EVM | `0x370DE27fdb7D1Ff1e1BaA7D11c5820a324Cf623C` | 50 blocks | Yes |
| BSC Testnet | 97 | EVM | `0x0DfbEe143b42B41eFC5A6F87bFD1fFC78c2f0aC9` | 5 blocks | Yes (testnet) |
| Arbitrum One | 42161 | EVM | `0x0DfbEe143b42B41eFC5A6F87bFD1fFC78c2f0aC9` | 2400 blocks | No |
| Polygon | 137 | EVM | `0x0DfbEe143b42B41eFC5A6F87bFD1fFC78c2f0aC9` | 300 blocks | No |
| Base | 8453 | EVM | `0x1892196E80C4c17ea5100Da765Ab48c1fE2Fb814` | 300 blocks | No |
| Tron Mainnet | 728126428 | Tron | `TR7NHqjeKQxGTCi8q8ZY4pL8otSzgjLj6t` (USDT TRC20) | 200 (API-confirmed) | No |
| TON Mainnet | 1100 | TON | `EQCxE6mUtQJKFnGfaROTKOt1lZbDiiX1kCixRv7Nw2Id_sDs` (USDT Jetton master) | 120 (API-finalized) | No |
> [!note] Proxy address variations
> Ethereum mainnet uses a v0.1.0 proxy (`0x370DE...`). Base uses a non-canonical CREATE2 address (`0x189219...`). All other EVM chains use the canonical v0.2.0 address (`0x0DfbEe...`). The memory note [[RN proxy addresses per chain]] has background on why CREATE2 canonical-address claims should not be trusted without verification.
To enable a disabled chain without a rebuild: set `SCANNER_ENABLED_CHAINS=56,1,42161` (overrides the JSON `verified` flags).
---
## 3. Architecture decisions
### Why a standalone Go service
The scanner runs a tight polling loop that needs to hold open TCP connections to multiple RPC endpoints, manage per-chain checkpoints, and retry webhook delivery independently of backend restarts. A dedicated process with its own SQLite state is simpler and more reliable than embedding this into the Node.js backend.
### Why SQLite
Single-node deployment. WAL mode gives concurrent reads during writes. The state set is small (one row per intent, one checkpoint per chain). No operational overhead of a separate DB process inside the container.
### Two payment rails
The scanner supports two fundamentally different payment models:
1. **Proxy-contract rail (EVM)**: funds flow through `ERC20FeeProxy`; the scanner matches by `paymentReference` embedded in the contract event. No unique destination address required; the reference is the discriminator.
2. **Direct-address rail (Tron, TON, and EVM balance-watch)**: each payment gets a unique HD-derived destination address. The scanner matches by `to` address and validates amount. This is the only model available on Tron and TON because no fee-proxy contract exists there.
### Confirmation thresholds
EVM confirmation depths are conservative to handle reorgs:
- **BSC (200)**: BSC has had historical reorg incidents; 200 blocks (~10 min) provides a practical safety margin.
- **ETH (50)**: ~10 min at 12 s/block; Ethereum finality is probabilistic post-merge but 50 blocks is well past economic finality.
- **Arbitrum (2400)**: Arbitrum uses optimistic rollup; 2400 blocks (~54 min) covers the challenge window.
- **Polygon (300)**: polygon reorgs have occurred at depth >100; 300 blocks gives headroom.
- **Base (300)**: Base is an OP Stack chain; same rationale as Polygon.
Tron and TON do not use block-depth confirmation — TronGrid and TonCenter only surface confirmed/finalized transactions, so status goes directly to `confirmed`. The scanner reports the chain's acceptance floor (200 / 120) in the webhook for backend use.
### Reorg protection (EVM)
The EVM worker re-scans `3 × confirmationThreshold` blocks before the checkpoint (clamped 20–500) on every tick. This `ReorgBuffer()` ensures that a log in a block that was reorganised off the canonical chain will be re-evaluated when the chain reorganises. The window is wide enough to cover any realistic reorg depth for the chains the scanner targets.
### Startup reconciliation
On startup, `confirmed` intents with `webhook_delivered_at IS NULL` created within the last 7 days have their webhook re-delivered. This recovers from a crash between `finalizeIntent` and `deliverWebhook` without requiring a manual retry trigger.
---
## 4. Component map
```
┌─────────────────────────────────────────────────────────┐
@@ -46,91 +110,38 @@ AMN Pay Scanner is a standalone Go microservice that watches on-chain payment ev
│ ├── webhook retry loop main.go + webhook.go │
│ └── BalanceWatchScheduler balance_watch.go │
│ │
│ reference.go — payment reference / topic hash math │
│ webhook.go — delivery, HMAC signing, retry │
│ balance.go — EVM ERC-20 balanceOf(address) reads │
│ reference.go — payment reference / topic hash │
│ webhook.go — delivery, HMAC signing, retry │
│ balance.go — EVM ERC-20 balanceOf reads │
│ balance_watch.go — balance_watches state + webhooks │
└─────────────────────────────────────────────────────────┘
```
One worker goroutine is spawned per active chain. All three chain types implement a common `Worker` interface (`start()`, `stop()`, `getHead()`). Workers poll on `POLL_INTERVAL_SEC` (default 15 s).
---
## 3. Chain worker model
## 5. Backend integration points
All three chain types implement the `Worker` interface:
```go
type Worker interface {
start()
stop()
getHead(ctx context.Context) (int64, error)
}
```
One worker goroutine is spawned per chain marked `"verified": true` in `supported-chains.json`. Workers are selected by `chainType`:
| chainType | Worker struct | API used |
| Direction | Endpoint | When |
|---|---|---|
| `evm` (default) | `ChainWorker` | JSON-RPC 2.0 (`eth_getLogs`, `eth_blockNumber`) |
| `tron` | `TronChainWorker` | TronGrid REST (`/v1/contracts/{contract}/events`) |
| `ton` | `TonChainWorker` | TonCenter v3 REST (`/jetton/transfers`) |
| Backend → Scanner | `POST /intents` | New payment initiated; returns `checkoutBlock` with `paymentReference` and proxy address |
| Backend → Scanner | `GET /intents/{id}` | Poll intent status (optional; webhook is primary) |
| Scanner → Backend | `POST <callbackUrl>` | Payment confirmed; signed with `X-AMN-Signature` HMAC-SHA256 |
| Backend → Scanner | `POST /balances/check` | Synchronous ERC-20 balance read (direct-address rail) |
| Backend → Scanner | `POST /balance-watches` | Start async balance watch (direct-address rail) |
| Scanner → Backend | `POST <callbackUrl>` | Balance changed; `X-AMN-Event-Type: balance_changed` |
| Backend → Scanner | `DELETE /balance-watches/{id}` | Stop watch after payment accepted or cancelled |
| Backend → Scanner | `GET /scanner/status` | Chain lag + pending counts (ops/monitoring) |
| Backend → Scanner | `POST /admin/webhooks/retry` | Force re-delivery of `webhook_failed` intents |
Workers poll on `POLL_INTERVAL_SEC` (default 15 s). On first run, each worker starts scanning from the current chain head minus a small buffer (10 blocks for EVM, 24 h for Tron/TON).
All non-health endpoints require `Authorization: Bearer <SCANNER_API_KEY>`. Webhooks are HMAC-SHA256 signed; backend must verify `X-AMN-Signature` before crediting any payment.
The `amn.scanner` backend provider wires intent creation, webhook receipt, and balance-watch lifecycle. See memory note [[amn scanner pay-in wiring + env]] for the 6 required env vars and the dispatcher registration.
---
## 4. EVM scanning detail
```
for each tick:
head = eth_blockNumber
from = max(checkpoint − ReorgBuffer(), 0)
chunks = split [from..head] into 2000-block ranges
for each chunk:
logs = eth_getLogs(proxyAddress, EventTopic, from, to)
for each log:
topicRef = Topics[1] (keccak256 of paymentReference — pre-indexed)
intent = DB lookup by topicRef WHERE status='pending'
validate(log.Data, intent) ← token + destination + amount check
confirmIntentPending() ← status → 'confirming'
saveCheckpoint(to)
checkConfirmations():
for each confirming intent:
confs = head - blockNumber + 1
if confs >= required: finalizeIntent(capped at required) + deliverWebhook()
```
**Reorg protection**: `ReorgBuffer()` re-scans `3 × confirmationThreshold` blocks before the checkpoint (clamped 20–500). This catches any log that appeared in a block that was later reorganised off the canonical chain.
**Event signature**: `TransferWithReferenceAndFee` keccak256 = `0xbc8b749850bdc0c5e4a49d50f87ec40dca2acdb70a84e7c6823ccdc7b3b3d4b3`
---
## 5. Tron scanning detail
TronGrid does not expose a fee-proxy contract. Each intent is assigned a unique HD-derived destination address. The scanner watches TRC20 `Transfer` events on the USDT contract and matches by `to` address.
- Checkpoint: block timestamp in milliseconds (`last_scanned_block` column)
- TronGrid addresses arrive as `41xxxx` hex (21 bytes); normalized to `0x` (20 bytes EVM style)
- Tron transactions reported by TronGrid are already confirmed; status goes directly to `confirmed` (no multi-block wait)
- Pagination follows `meta.links.next` until empty
---
## 6. TON scanning detail
TON uses TonCenter v3. Per-intent polling: for each pending TON intent, a separate HTTP call fetches incoming Jetton transfers to that destination since the checkpoint.
- Checkpoint: Unix timestamp in seconds
- TON addresses are base64url (`EQ…`/`UQ…`) — case-sensitive, never lowercased
- `proxyAddress` = USDT Jetton master address (`EQCxE6mUtQJKFnGfaROTKOt1lZbDiiX1kCixRv7Nw2Id_sDs`)
- TonCenter returns only finalized transactions; status goes directly to `confirmed`
- Lag is reported in seconds, not blocks
- Known scaling limitation: O(pending intents) API calls per scan cycle
---
## 7. Intent lifecycle
## 6. Intent lifecycle
```
pending ──(tx seen)──► confirming ──(enough blocks)──► confirmed ──(webhook ok)──► [done]
@@ -139,112 +150,28 @@ pending ──(tx seen)──► confirming ──(enough blocks)──► confi
└───────────────────────┴──────────► expired webhook_failed
```
- **Tron / TON** skip `confirming` and jump directly to `confirmed`.
- **Tron / TON** skip `confirming` and go directly to `confirmed` (API only surfaces finalized txns).
- `webhook_failed` intents are retried every `WEBHOOK_RETRY_HOURS` (default 6 h) and on `POST /admin/webhooks/retry`.
- **Startup reconciliation**: on startup, `confirmed` intents with `webhook_delivered_at IS NULL` and created in the last 7 days have their webhook re-delivered. This recovers from a crash between `finalizeIntent` and `deliverWebhook`.
- Retry schedule on first delivery attempt: 5 s → 30 s → 2 min → 10 min → 1 h → `webhook_failed`.
---
## 8. Direct balance watch model
Direct-address payments are a separate rail from scanner intents. They do not require `ERC20FeeProxy`, `paymentReference`, or a smart-contract event. The backend gives the scanner a public address, token, callback URL, and HMAC secret.
Two backend usage modes are supported:
- **Synchronous check mode**: backend calls `POST /balances/check` when an address is allocated to the buyer and again when the buyer clicks "I paid". The backend compares the current base-unit balance to its stored baseline and target amount.
- **Watch mode**: backend calls `POST /balance-watches`; the scanner stores a row in `balance_watches`, reads the initial balance, and checks the same address/token periodically. If the balance changes, scanner sends a signed `balance_changed` webhook. Backend stops the watch with `DELETE /balance-watches/{watchId}` once the payment is accepted, cancelled, or otherwise resolved.
Cadence is age-based:
| Watch age | Next check interval |
|---|---|
| 0–24 h | 5 min |
| 24–48 h | 10 min |
| 48–72 h | 20 min |
| 72 h–7 d | 40 min |
| 7 d+ | status becomes `expired` |
The scanner only advances `current_balance` after a changed-balance webhook is delivered successfully. If the backend is down or returns non-2xx, the same change is retried on the next scheduled due check.
Current implementation scope: EVM ERC-20 `balanceOf(address)` via JSON-RPC `eth_call`. Tron/TON balance reads are future scope.
---
## 9. Payment reference math (EVM)
```
paymentReference = last8Bytes(keccak256(lower(intentId + salt + destination)))
topicRef (index) = keccak256(paymentReferenceBytes)
```
The ERC20FeeProxy indexes `paymentReference` so `Topics[1]` in the log is `topicRef`, not the raw reference. The DB stores `topic_ref` pre-computed per intent so the scan loop is a single indexed SQL lookup instead of O(n) hashing.
---
## 10. Database schema (SQLite WAL)
Three main tables:
**`intents`** — one row per payment intent
| Column | Type | Notes |
|---|---|---|
| `intent_id` | TEXT PK | caller-supplied UUID |
| `chain_id` | INTEGER | numeric chain ID |
| `chain_type` | TEXT | `evm` / `tron` / `ton` |
| `token_address` | TEXT | EVM/Tron: lowercase 0x hex; TON: base64url |
| `destination` | TEXT | receiving address |
| `amount` | TEXT | base-10 wei / token smallest unit |
| `payment_reference` | TEXT | 8-byte hex (EVM only) |
| `topic_ref` | TEXT | keccak256 of paymentReference (EVM index) |
| `status` | TEXT | `pending` / `confirming` / `confirmed` / `expired` / `webhook_failed` |
| `callback_url` | TEXT | backend webhook endpoint |
| `callback_secret` | TEXT | HMAC key (not returned in GET) |
| `confirmations_required` | INTEGER | from chain config or caller override, floored at the chain acceptance threshold |
| `tx_hash` | TEXT NULL | transaction hash once seen |
| `log_index` | INTEGER NULL | log position within tx (EVM) |
| `block_number` | INTEGER NULL | block / timestamp when seen |
| `confirmations` | INTEGER | current depth while confirming; capped at the accepted threshold after confirmation |
| `salt` | TEXT | 32-byte random hex for reference derivation |
| `webhook_delivered_at` | TEXT NULL | RFC3339 timestamp of successful delivery |
| `created_at` / `updated_at` | DATETIME | |
Unique index on `(tx_hash, log_index)` prevents duplicate intent confirmation.
**`checkpoints`** — one row per chain, tracks scan progress
| Column | Notes |
|---|---|
| `chain_id` | PK |
| `last_scanned_block` | block number (EVM), ms timestamp (Tron), unix seconds (TON) |
**`balance_watches`** — one row per direct-address balance watch
| Column | Type | Notes |
|---|---|---|
| `watch_id` | TEXT PK | caller-supplied or scanner-generated idempotency key |
| `chain_id` / `chain_type` | INTEGER / TEXT | currently EVM only for direct balance reads |
| `token_address` / `token_symbol` | TEXT | ERC-20 contract and optional registry symbol |
| `decimals` | INTEGER | registry decimals for display/metadata |
| `address` | TEXT | watched holder address |
| `baseline_balance` | TEXT | base-unit balance at backend baseline |
| `current_balance` | TEXT | last delivered/current scanner balance |
| `status` | TEXT | `watching` / `stopped` / `expired` |
| `callback_url` / `callback_secret` | TEXT | signed webhook destination |
| `last_checked_at` / `next_check_at` | DATETIME | scheduler state |
| `change_count` / `last_notified_at` | INTEGER / DATETIME | notification audit fields |
| `expires_at` | DATETIME | hard stop after 7 days |
| `created_at` / `updated_at` | DATETIME | UTC timestamps |
Indexes: `(status, next_check_at)` for due scans and `(chain_id, status)` for status reporting.
---
## 11. Security model
## 7. Security model
- All non-health endpoints require `Authorization: Bearer <SCANNER_API_KEY>` (constant-time compare).
- If `SCANNER_API_KEY` is unset the server logs a warning and allows all requests — intended for local dev only.
- Webhooks are signed with HMAC-SHA256: `X-AMN-Signature: hex(hmac(body, callbackSecret))`.
- The `callbackSecret` is stored in the DB but excluded from all JSON responses (`json:"-"` tag).
- Request bodies are limited to 64 KB.
- Balance-watch callbacks use the same HMAC header plus `X-AMN-Delivery-ID=<watchId>` and `X-AMN-Event-Type=balance_changed`.
- Unset `SCANNER_API_KEY` logs a warning and allows all requests — local dev only.
- Webhooks signed with HMAC-SHA256: `X-AMN-Signature: hex(hmac(body, callbackSecret))`.
- `callbackSecret` stored in DB but excluded from all JSON responses (`json:"-"`).
- Request bodies limited to 64 KB.
- `SCANNER_CALLBACK_ALLOWED_HOSTS` env var restricts allowed webhook target hosts (SSRF guard).
---
## 8. Known limitations and open items
| Item | Detail |
|---|---|
| TON O(n) API calls | Per-intent polling — one TonCenter call per pending TON intent per scan cycle. Fine at low volume; needs batching for scale. |
| Direct balance reads: EVM only | `POST /balances/check` and `POST /balance-watches` only support EVM ERC-20 (`eth_call` balanceOf). Tron/TON balance reads are future scope. |
| Arbitrum / Polygon / Base / Tron / TON disabled | `verified: false` in `supported-chains.json`. Enable via `SCANNER_ENABLED_CHAINS` env var without a code change. |
| Ethereum proxy version | Chain 1 uses the v0.1.0 proxy (`0x370DE...`). A v0.2.0 proxy is also deployed but checkout still uses the v0.1.0 ABI. |