Observability
Prometheus metrics, Grafana dashboards, structured JSON logs in Loki, the trading-specific health checks, and tracing later.
Three pillars: Prometheus for metrics, Grafana for dashboards, Loki for structured logs — with OpenTelemetry tracing as a later addition.
Metrics (Prometheus)
- collector status
- event throughput
- API latency
- queue depth
- agent workflow duration
- LLM calls & token usage
- order latency
- execution failures
- slippage
- data freshness
- memory DB size
- CPU / RAM / disk
Dashboards (Grafana)
| Dashboard | Shows |
|---|---|
| Infrastructure | Container health, CPU/RAM/disk |
| Data sources | Collector & feed health, freshness |
| Strategy & execution | Live execution health, financial metrics |
| Models & agents | Model usage, agent performance |
Health checks & alerts (Uptime Kuma)
Beyond raw metrics, a small set of conditions is monitored for liveness and paged on failure — the ones that silently break trading if missed:
- frontend · Convex · API
- worker heartbeats
- CCXT exchange connectivity
- RPC endpoints
- provider error rates
- queue depth
- stale market data
- order-reconciliation delay
- wallet-balance mismatch
- daily budget consumption
- backup completion
Structured logging (Loki)
Every log is JSON and carries, where applicable:
log fields
timestamp · service · environment · trace_id · workflow_id
opportunity_id · strategy_version · trade_id · exchange · severity · eventTracing later
OpenTelemetry can add distributed tracing across collectors, FastAPI, agents, quant workers and the execution engine once the core is stable.