btest-rs/docs/architecture.md

btest-rs Architecture

Overview

btest-rs is a Rust reimplementation of the MikroTik Bandwidth Test protocol. It operates in two modes: server (accepts connections from MikroTik devices) and client (connects to MikroTik btest servers).

Module Structure

graph TB
    main["main.rs<br/>CLI parsing (clap)"]
    server["server.rs<br/>Server mode"]
    client["client.rs<br/>Client mode"]
    protocol["protocol.rs<br/>Wire protocol types"]
    auth["auth.rs<br/>MD5 authentication"]
    ecsrp5["ecsrp5.rs<br/>EC-SRP5 authentication<br/>(Curve25519 Weierstrass)"]
    bandwidth["bandwidth.rs<br/>Rate control & reporting"]
    csv_output["csv_output.rs<br/>CSV result logging"]
    syslog["syslog_logger.rs<br/>Remote syslog (RFC 3164)"]
    lib["lib.rs<br/>Public API for tests"]

    main --> server
    main --> client
    main --> bandwidth
    main --> csv_output
    main --> syslog
    server --> protocol
    server --> auth
    server --> ecsrp5
    server --> bandwidth
    server --> syslog
    client --> protocol
    client --> auth
    client --> ecsrp5
    client --> bandwidth
    lib --> server
    lib --> client
    lib --> protocol
    lib --> auth
    lib --> ecsrp5
    lib --> bandwidth

Data Flow

Server Mode (MikroTik connects to us)

sequenceDiagram
    participant MK as MikroTik Client
    participant TCP as TCP Control<br/>(port 2000)
    participant SRV as btest-rs Server
    participant UDP as UDP Data<br/>(port 2001+)

    MK->>TCP: Connect
    SRV->>TCP: HELLO [01 00 00 00]
    MK->>TCP: Command [16 bytes]
    Note over SRV: Parse proto, direction,<br/>tx_size, speeds

    alt No auth configured
        SRV->>TCP: AUTH_OK [01 00 00 00]
    else MD5 auth (RouterOS < 6.43)
        SRV->>TCP: AUTH_REQUIRED [02 00 00 00]
        SRV->>TCP: Challenge [16 random bytes]
        MK->>TCP: Response [16 hash + 32 username]
        Note over SRV: Verify MD5(pass + MD5(pass + challenge))
        SRV->>TCP: AUTH_OK or AUTH_FAILED
    else EC-SRP5 auth (RouterOS >= 6.43, --ecsrp5 flag)
        SRV->>TCP: EC-SRP5 [03 00 00 00]
        MK->>TCP: [len][username\0][client_pubkey:32][parity:1]
        SRV->>TCP: [len][server_pubkey:32][parity:1][salt:16]
        MK->>TCP: [len][client_confirmation:32]
        SRV->>TCP: [len][server_confirmation:32]
        Note over SRV: Curve25519 Weierstrass EC-SRP5<br/>See docs/ecsrp5-research.md
        SRV->>TCP: AUTH_OK [01 00 00 00]
    end

    alt TCP mode
        Note over SRV,MK: Data flows on same TCP connection
        loop Every second
            SRV-->>SRV: Print bandwidth stats
        end
    else UDP mode
        SRV->>TCP: UDP port [2 bytes BE]
        Note over SRV: Bind UDP socket
        par TX Thread (if server transmits)
            loop Continuous
                SRV->>UDP: Data packets [seq + payload]
            end
        and RX Thread (if server receives)
            loop Continuous
                UDP->>SRV: Data packets [seq + payload]
            end
        and Status Loop (TCP control)
            loop Every 1 second
                MK->>TCP: Status [12 bytes]
                SRV->>TCP: Status [12 bytes]
                Note over SRV: Adjust TX speed<br/>based on client feedback
            end
        end
    end

Client Mode (we connect to MikroTik)

sequenceDiagram
    participant CLI as btest-rs Client
    participant TCP as TCP Control
    participant MK as MikroTik Server

    CLI->>TCP: Connect to MikroTik:2000
    MK->>TCP: HELLO
    CLI->>TCP: Command [16 bytes]
    Note over CLI: direction bits tell server<br/>what to do (TX/RX/BOTH)

    alt Auth response 01 (no auth)
        Note over CLI: No auth, proceed
    else Auth response 02 (MD5)
        MK->>TCP: Challenge [16 random bytes]
        CLI->>TCP: MD5 response [48 bytes]
        MK->>TCP: AUTH_OK
    else Auth response 03 (EC-SRP5)
        CLI->>TCP: [len][username\0][client_pubkey:32][parity:1]
        MK->>TCP: [len][server_pubkey:32][parity:1][salt:16]
        CLI->>TCP: [len][client_confirmation:32]
        MK->>TCP: [len][server_confirmation:32]
        MK->>TCP: AUTH_OK
    end

    Note over CLI,MK: Data transfer begins<br/>(TCP or UDP, same as server)

Threading Model

graph TB
    subgraph "Server Process"
        LISTEN["Main Loop<br/>Accept connections"]
        LISTEN -->|spawn per client| HANDLER

        subgraph "Per-Client Tasks (tokio)"
            HANDLER["Connection Handler<br/>Handshake + Auth"]
            HANDLER --> TX["TX Task<br/>Send data packets"]
            HANDLER --> RX["RX Task<br/>Receive data packets"]
            HANDLER --> STATUS["Status Loop<br/>Exchange stats every 1s"]
        end
    end

    subgraph "Shared State (Arc + Atomics)"
        STATE["BandwidthState"]
        TX_BYTES["tx_bytes: AtomicU64"]
        RX_BYTES["rx_bytes: AtomicU64"]
        TX_SPEED["tx_speed: AtomicU32"]
        RUNNING["running: AtomicBool"]
    end

    TX --> TX_BYTES
    RX --> RX_BYTES
    STATUS --> TX_BYTES
    STATUS --> RX_BYTES
    STATUS --> TX_SPEED
    TX --> TX_SPEED
    TX --> RUNNING
    RX --> RUNNING
    STATUS --> RUNNING

Key Design Decisions

1. Tokio async runtime

All I/O is async via tokio. Each client connection spawns independent tasks for TX, RX, and status exchange. This allows handling hundreds of concurrent connections on a single thread pool.

2. Lock-free shared state

TX/RX threads and the status loop share bandwidth counters via AtomicU64. No mutexes needed -- swap(0) atomically reads and resets counters each interval.

3. Sequential status loop (matching C pselect)

The UDP status exchange uses a sequential timeout-read-then-send pattern rather than tokio::select!. This ensures our status messages are sent exactly every 1 second, preventing MikroTik's speed adaptation from seeing irregular feedback.

4. Direction bits from server perspective

The direction byte in the protocol means what the server should do:

• 0x01 (CMD_DIR_RX) = server receives
• 0x02 (CMD_DIR_TX) = server transmits
• 0x03 (CMD_DIR_BOTH) = bidirectional

The client inverts before sending: client "transmit" sends CMD_DIR_RX (telling server to receive).

5. TCP socket half keepalive

When only one direction is active (e.g., TX only), the unused socket half is kept alive. Dropping OwnedWriteHalf sends a TCP FIN, which MikroTik interprets as disconnection.

6. Static musl binary

Release builds use musl for a fully static binary with zero runtime dependencies. The binary is approximately 2 MB and runs on any Linux distribution.

7. EC-SRP5 with big integer arithmetic

The EC-SRP5 implementation uses num-bigint for Curve25519 Weierstrass-form elliptic curve arithmetic. MikroTik's authentication uses the Weierstrass form (not the more common Montgomery or Edwards forms), requiring direct field arithmetic over the prime 2^255 - 19. The implementation includes point multiplication, lift_x, redp1 (hash-to-curve), and Montgomery coordinate conversion.

8. Global singletons for syslog and CSV

The syslog and CSV modules use Mutex<Option<...>> global statics. This avoids threading state through every function call while remaining safe. Both modules are initialized once at startup and used from any async task via their public API functions.

9. Shared BandwidthState for client duration timeout

When running with --duration, the tokio timeout cancels the client future. To preserve stats accumulated during the test, BandwidthState is created in main() and passed as an Arc into run_client(). The state survives cancellation because main() holds a reference. The record_interval() method accumulates totals that summary() returns.

10. IPv6 socket handling

IPv6 requires special handling on macOS:

• UDP sockets bind to [::] for IPv6 peers, 0.0.0.0 for IPv4
• Socket send/receive buffers set to 4MB via socket2 before wrapping with tokio
• SocketAddr::new() used instead of string formatting (avoids [addr]:port parsing issues)
• Connected sockets preferred for single-connection (avoids ENOBUFS on send_to())
• NDP probe packet sent before data blast to populate neighbor cache
• Adaptive backoff on ENOBUFS (200μs→10ms, resets on success)

11. CPU usage monitoring

A background OS thread samples system CPU every 1 second via:

• macOS: host_statistics(HOST_CPU_LOAD_INFO) — returns user/system/idle/nice ticks
• Linux: /proc/stat — reads aggregate CPU line

The percentage is stored in a global AtomicU8 and included in every status message at byte 1 using MikroTik's encoding: 0x80 | percentage. On receive, the remote CPU is decoded with byte & 0x7F and capped at 100%. Both local and remote CPU are displayed per interval and logged to CSV/syslog.

File Layout

btest-rs/
├── src/
│   ├── main.rs              # CLI entry point, argument parsing (clap)
│   ├── lib.rs               # Public API (used by integration tests)
│   ├── protocol.rs          # Wire format: Command, StatusMessage, constants
│   ├── auth.rs              # MD5 challenge-response authentication
│   ├── ecsrp5.rs            # EC-SRP5 authentication (Curve25519 Weierstrass)
│   ├── server.rs            # Server mode: listener, TCP/UDP handlers
│   ├── client.rs            # Client mode: connector, TCP/UDP handlers
│   ├── bandwidth.rs         # Rate limiting, formatting, shared state
│   ├── cpu.rs               # CPU usage sampler (macOS + Linux)
│   ├── csv_output.rs        # CSV result logging (append-mode, auto-header)
│   └── syslog_logger.rs     # Remote syslog sender (RFC 3164 / BSD format)
├── tests/
│   └── integration_test.rs  # End-to-end server/client tests
├── scripts/
│   ├── build-linux.sh           # Cross-compile for x86_64 Linux (musl)
│   ├── build-macos-release.sh   # macOS release build
│   ├── install-service.sh       # systemd service installer
│   ├── push-docker.sh           # Push Docker image to registry
│   ├── test-local.sh            # Loopback self-test
│   ├── test-mikrotik.sh         # Test against MikroTik device
│   ├── test-docker.sh           # Docker container test
│   └── debug-capture.sh         # Packet capture for debugging
├── docs/
│   ├── architecture.md          # This file
│   ├── protocol.md              # Protocol specification
│   ├── user-guide.md            # Usage documentation
│   ├── docker.md                # Docker & deployment guide
│   ├── ecsrp5-research.md       # EC-SRP5 reverse-engineering notes
│   └── man/
│       └── btest.1              # Unix manual page (troff format)
├── tests/
│   ├── integration_test.rs      # Basic server/client handshake tests
│   ├── ecsrp5_test.rs           # EC-SRP5 authentication tests
│   └── full_integration_test.rs # Comprehensive: all protocols, IPv4/6, CSV, syslog
├── deploy/
│   └── syslog-ng-btest.conf    # syslog-ng configuration for btest events
├── proto-test/                  # Python EC-SRP5 prototype (research branch)
│   ├── btest_ecsrp5_client.py  # Working Python btest EC-SRP5 client
│   ├── btest_mitm.py           # MITM proxy for protocol analysis
│   └── elliptic_curves.py      # Curve25519 Weierstrass (MarginResearch)
├── KNOWN_ISSUES.md              # Known bugs and platform limitations
├── Dockerfile                   # Production Docker image (multi-stage)
├── Dockerfile.cross             # Cross-compilation for Linux x86_64
├── docker-compose.yml           # Docker Compose configuration
├── Cargo.toml                   # Rust package manifest
├── Cargo.lock                   # Dependency lock file
├── LICENSE                      # MIT License
└── btest-opensource/            # Original C implementation (git submodule)