nick/nick-doc

Fork 0

Files

Siavash Sameni 9dcdb420fc docs: sync from backend 22ae0bd — scanner balance watches

2026-06-03 21:23:50 +04:00

10 KiB

Raw Blame History

title, tags, created

title

Scanner Operations

Runbook for deploying, configuring, monitoring, and troubleshooting the AMN Pay Scanner microservice.

1. Configuration reference

All configuration via environment variables. See .env.example in the scanner repo.

Variable	Default	Required	Description
`PORT`	`8080`	no	HTTP listen port
`DB_PATH`	`./scanner.db`	no	SQLite database path
`CHAINS_JSON_PATH`	`./supported-chains.json`	no	Supported chains config
`TOKENS_JSON_PATH`	`./tokens.json`	no	Token registry
`SCANNER_API_KEY`	(none)	yes (prod)	Bearer token for all non-health endpoints. Generate with `openssl rand -hex 32`
`POLL_INTERVAL_SEC`	`15`	no	Chain poll interval in seconds
`INTENT_TTL_HOURS`	`24`	no	Pending/confirming intents older than this are expired (0 = disabled)
`WEBHOOK_RETRY_HOURS`	`6`	no	Interval between automatic webhook_failed re-delivery passes (0 = disabled)
`BALANCE_WATCH_TICK_SEC`	`60`	no	Scheduler tick for due direct-address balance watches
`BALANCE_WATCH_BATCH_SIZE`	`50`	no	Max due balance watches processed per scheduler tick
`TRONGRID_API_KEY`	(none)	recommended	TronGrid API key; without it rate limits are very low
`TONCENTER_API_KEY`	(none)	recommended	TonCenter API key
`RPC_BSC`	(chain config)	no	Override BSC RPC URL (chain 56)
`RPC_ARB`	(chain config)	no	Override Arbitrum RPC URL (chain 42161)
`RPC_ETH`	(chain config)	no	Override Ethereum RPC URL (chain 1)
`RPC_POLYGON`	(chain config)	no	Override Polygon RPC URL (chain 137)
`RPC_BASE`	(chain config)	no	Override Base RPC URL (chain 8453)

Warning

If SCANNER_API_KEY is not set, the scanner logs a warning and accepts all requests. Never run this way in production.

2. Docker deployment

The scanner ships as a single Docker image. The Dockerfile uses a two-stage build (Go 1.25 builder → Alpine 3.21 runtime).

Quick start (dev)

cd scanner/
cp .env.example .env
# edit .env — set SCANNER_API_KEY, RPC overrides, etc.

docker build -t amn-scanner:dev .
docker run -d \
  --name amn-scanner \
  -p 8080:8080 \
  -v $(pwd)/data:/data \
  --env-file .env \
  amn-scanner:dev

Production (via arcane-cli / Watchtower)

The scanner is deployed manually via arcane-cli (not gitops). Watchtower does NOT manage it automatically. After pushing a new image, redeploy with:

arcane-cli project redeploy --json <project-id>

The SQLite database is stored on a named Docker volume (/data). Do not recreate the volume between deploys — it holds the checkpoint and intent state.

3. Health check

curl http://localhost:8080/health
# {"status":"ok","time":"2026-05-30T12:00:00Z"}

Docker HEALTHCHECK is already configured in the Dockerfile (30 s interval, 5 s timeout, 3 retries).

4. Monitoring

Scanner status endpoint

curl -H "Authorization: Bearer $SCANNER_API_KEY" \
     http://localhost:8080/scanner/status | jq .

Check:

lag — should be near 0 for healthy chains (blocks behind for EVM, seconds for TON)
pendingIntents — number of unresolved intents per chain
activeBalanceWatches — number of direct-address watches in watching status per chain
lastScannedBlock — should advance each poll

Logs

The scanner uses Go's log/slog structured logger with level prefixes. Key log patterns:

Pattern	Meaning
`[scanner] worker started`	Worker goroutine began for this chain
`[evm] intent confirming`	EVM tx seen, waiting for confirmations
`[evm] intent confirmed`	EVM: N confirmations reached
`[tron] MATCH` / `[ton] MATCH`	Transfer matched, going to confirmed
`[webhook] delivered`	Webhook POST succeeded
`[webhook] non-2xx response`	Backend returned error (will retry)
`[webhook] all retries exhausted`	Intent moved to webhook_failed
`[scanner] reconciling confirmed intents`	Startup crash recovery in progress
`[evm] scanner lag`	Chain lag > 100 blocks (investigate RPC)
`[scanner] balance watch scheduler started`	Balance watch polling loop started
`[api] balance watch created`	Backend registered a direct-address watch
`[balance-watch] balance read error`	RPC failed while reading a watched balance
`[balance-watch-webhook] delivered`	Changed-balance webhook POST succeeded
`[balance-watch-webhook] non-2xx response`	Backend rejected changed-balance webhook; scanner will retry the change later

5. Adding / modifying chains

Edit supported-chains.json. Fields:

Field	Notes
`chainId`	Numeric EIP-155 chain ID (arbitrary int for Tron/TON)
`chainType`	`"evm"` (default) / `"tron"` / `"ton"`
`rpcUrl`	Primary RPC endpoint
`publicRpcUrl`	Fallback RPC (EVM only)
`proxyAddress`	ERC20FeeProxy address (EVM); USDT contract (Tron); USDT Jetton master (TON)
`confirmationThreshold`	Chain acceptance floor. EVM workers wait this many blocks; Tron/TON use it as the accepted confirmation count reported to backend
`verified`	`true` to activate the worker; `false` to disable without deleting

Important

Changing proxyAddress for an EVM chain only affects new scans. Existing pending intents will still be matched against the old address until they expire or are confirmed.

After editing, restart the scanner container to pick up the new config.

6. Adding tokens to the registry

Edit tokens.json. Each entry:

{ "chainId": 56, "address": "0x...", "symbol": "USDC", "decimals": 18, "name": "USD Coin" }

Token registry is used only for populating tokenSymbol and decimals in the checkoutBlock response. Omitting a token does not break scanning — it just leaves those fields empty.

7. Manual webhook retry

Force immediate re-delivery of all webhook_failed intents:

curl -X POST -H "Authorization: Bearer $SCANNER_API_KEY" \
     http://localhost:8080/admin/webhooks/retry
# {"queued": N}

8. Database inspection

The SQLite database (/data/scanner.db) can be inspected with the sqlite3 CLI inside the container:

docker exec -it amn-scanner sqlite3 /data/scanner.db

# Check stuck intents
SELECT intent_id, chain_id, status, created_at, webhook_delivered_at
FROM intents
WHERE status NOT IN ('confirmed', 'expired')
ORDER BY created_at DESC;

# Check chain checkpoints
SELECT chain_id, last_scanned_block, updated_at FROM checkpoints;

# Count by status
SELECT status, count(*) FROM intents GROUP BY status;

# Check active direct-address watches
SELECT watch_id, chain_id, token_symbol, address, current_balance, next_check_at, expires_at
FROM balance_watches
WHERE status = 'watching'
ORDER BY next_check_at ASC;

# Count watches by status
SELECT status, count(*) FROM balance_watches GROUP BY status;

9. Troubleshooting

Intent stuck in `pending`

Check /scanner/status — is the chain worker running and advancing (lag > 0 for a long time = RPC issue)?
Check that chainId and tokenAddress match exactly what is in supported-chains.json and tokens.json.
For EVM: verify the proxyAddress matches the contract the buyer is calling.
For Tron: confirm the destination address is stored in EVM-hex (0x) format in the DB.
Check scanner logs for REJECT messages around the expected tx time.

Webhook never received by backend

Check webhook_delivered_at in the DB — if not null, the scanner delivered successfully and the backend side is the issue.
If null and status is webhook_failed: check backend logs for the incoming POST; verify X-AMN-Signature validation code.
If status is confirmed but webhook_delivered_at is null: startup reconciliation may re-deliver on next restart.
Use POST /admin/webhooks/retry to trigger immediate retry.

High lag on EVM chain

Check RPC endpoint availability and rate limits.
Consider setting a RPC_* env override to a premium RPC (Alchemy, Infura, QuickNode).
The scanner falls back to publicRpcUrl if the primary fails but public nodes have lower limits.

Intent confirmed but amount looks wrong

The scanner accepts any amount >= intent.Amount. Overpayments are not flagged. Underpayments result in the intent staying pending until TTL expiry.

Direct balance watch is not firing

Confirm the target chain is EVM. Scanner 0.1.8 direct balance checks use ERC-20 balanceOf(address) and do not yet support Tron/TON balance reads.
Check /scanner/status for activeBalanceWatches on the expected chain.
Inspect balance_watches.next_check_at; if it is in the future, the scheduler is waiting according to the decay cadence.
Check logs for [balance-watch] balance read error; RPC failures reschedule the watch without notifying backend.
Confirm callbackUrl and callbackSecret match backend AMN_SCANNER_WEBHOOK_SECRET.
If [balance-watch-webhook] non-2xx response appears, inspect backend logs for the AMN scanner webhook route. The scanner keeps current_balance unchanged and retries the same balance change on the next due check.

Direct balance watch should stop

Use either stop form:

curl -X DELETE -H "Authorization: Bearer $SCANNER_API_KEY" \
     http://localhost:8080/balance-watches/<watchId>

curl -X POST -H "Authorization: Bearer $SCANNER_API_KEY" \
     http://localhost:8080/balance-watches/<watchId>/stop

Backend should stop a watch after payment acceptance, cancellation, manual resolution, or when the payment is no longer payable.

10. CI/CD notes

Woodpecker CI pipeline is in .woodpecker/.
Telegram notify steps were removed (no TG secrets configured).
Deploy step was removed — the scanner is deployed manually via arcane-cli.
The CI pipeline builds and pushes the Docker image to the Gitea registry.
Image tag format: dev-<VERSION> (from the VERSION file).

Tip

After CI completes, verify the image is in the registry before redeploying. Silent CI failures can leave a stale image tagged. Check the registry tag timestamp, not just the CI green light.

10 KiB Raw Blame History