Data layers & boundaries
The four storage layers — Convex, SQLite, Parquet/DuckDB and Redis — the rule that decides which is which, and the retention policy per data type.
Clawlas runs four data layers on purpose. Conflating them is the easy mistake; keeping them separate is what makes the dashboard honest, the agents fast, the analytics scalable and the events disposable.
The four layers
| Layer | Responsibility |
|---|---|
| Convex | Current operational state — everything the dashboard reads reactively. |
| SQLite | The agents' persistent knowledge and memory, local to the runtime. |
| Parquet / DuckDB | Historical and analytical data — high-volume market history, backtesting datasets. |
| Redis / Valkey Streams | Temporary event transport between services. Never a system of record. |
Where does this datum live?
Read reactively by the dashboard → Convex. The agents’ private cognition → SQLite. High-volume history & analytics → Parquet/DuckDB. In-flight coordination → Redis/Valkey. The same raw data is not duplicated across all four.
The split in practice
| Data | Source of truth | Why |
|---|---|---|
| Opportunities, positions, orders, approvals | Convex | Operational truth, read live by the UI |
| Agent reports & trade journal | Convex | The dashboard must display them live |
| Long-term lessons & decisions | SQLite | Private agent cognition, high frequency |
| Raw market trades, order books, on-chain history | Parquet | Too voluminous for Convex |
| Backtesting datasets & analytical queries | DuckDB | Fast scans over Parquet |
| In-flight events (market, research, execution) | Redis / Valkey | Ephemeral transport |
| Secrets | Docker secrets / encrypted files | Never in Convex, SQLite or prompts |
Retention policy
Hot stores stay small and fast; volume ages into the lake or compresses into summaries. Nothing high-frequency lives in Convex, and nothing is kept raw forever just because it might be queried later.
| Data type | Hot retention | Long-term form |
|---|---|---|
| Raw social posts | 7–30 days | Deduplicated summary / compressed archive |
| Raw high-frequency trades | 1–7 days (selected assets) | Aggregated bars / Parquet |
| Order-book snapshots | Hours–days | Features & selected event windows |
| OHLCV | Long term | Parquet / DuckDB |
| Decisions & trade logs | Permanent | Convex + backup |
| LLM prompts / responses | Configurable, privacy-aware | Structured summaries & hashes |
| API usage logs | 12–24 months | Daily / monthly aggregates |
| System logs | 7–30 days locally | Important incidents archived |
The kill switch crosses the boundary: a flag the web raises in Convex, read by the Python core on its next loop. One store the UI controls, one process that obeys.