Testing strategy
Unit, property-based, integration, replay and safety tests — the system must survive its own failure modes.
A trading system has to survive its own failure modes. Testing spans five layers — from pure formulas to full-system replays of historical market events.
The layers
| Layer | Covers |
|---|---|
| Unit | Scoring formulas, indicators, position sizing, risk rules, exchange normalization, memory ops |
| Property-based (Hypothesis) | Unexpected values, boundary conditions, invariants, order-sizing, score ranges |
| Integration | Convex↔FastAPI, agent↔memory, collector events, execution adapters, Redis Streams, Compose |
| Replay | Historical events through the whole pipeline — normalization → features → scoring → decision → sim |
| Safety | Failure modes (below) |
Safety tests
The system is deliberately broken to prove it fails safely:
| Scenario | Expected behavior |
|---|---|
| Duplicate orders | Idempotency key prevents a second fill |
| Stale signals / broken feeds | Decisions blocked, alert raised |
| Unavailable exchange | Graceful degradation, no phantom orders |
| Extreme slippage / insufficient balance | Order rejected deterministically |
| LLM malformed output | Schema validation rejects it |
| Redis outage / restart mid-execution | Reconciliation on boot, no duplicates |
Replay is the integration backbone
Replaying real historical market events through the complete system validates collector normalization, feature computation, scoring, agent reasoning, decision output and execution simulation in one pass — the closest thing to a dress rehearsal before capital.