Data collection

Market data collection

Independent collectors per source, a provider catalogue with limits, a common event schema, and data-quality tracking on every point.

Each data source has an independent collector. A failure in one source must never stop the others — so collectors do not share a process.

The collectors

collector-binance
collector-coinbase
collector-bybit
collector-okx
collector-dexscreener
collector-solana
collector-ethereum
collector-social
collector-listing-announcements

Collection methods (in order of preference)

1
WebSocket streams
Lowest latency, push-based — the default.
2
Official APIs
When no stream exists.
3
RSS / announcement feeds
For listing announcements.
4
Polling
Only when nothing better is available.
5
Browser automation
Last resort only.

Discovery source catalogue

Deterministic filters reduce each stream to a small set of candidates before any AI call. Paid providers are added only when a measured strategy shows the data pays for itself — start free and upgrade on evidence:

Raw event firehose

Every source, continuously

Dedup · rate-limit · normalize

Common schema + data quality

Deterministic filters & rules

Indicators, anomalies, thresholds

Candidate events

Cheap AI runs only here

Source	Role	Notes / limits
CCXT loadMarkets() deltas	New-market detection on CEX/DEX	Free; runtime version-aware
DEX Screener	Pool/pair discovery & liquidity	~60 rpm profiles, ~300 rpm pairs/search; price ≠ sellability; cache & batch
CoinGecko / GeckoTerminal	Metadata, reconciliation, on-chain pools	Free Demo API (quota) + MCP; not the sole realtime source
Birdeye	Faster on-chain discovery, security, WS	Free → Lite $39 → Starter $99 → Premium $199 → Business $499 (recheck)
X API	Early social signal (curated accounts)	Pay-per-use; last-seen id; no replies/reposts; daily/monthly budget with auto-stop
SearXNG (self-hosted)	Project research, fallback web search	No licence fee; upstream engines may rate-limit

Social & narrative sources are supporting evidence only — never the sole trigger.

Event normalization

Every external event becomes one common internal schema:

event schema

event_id · event_type · source · source_event_id · token_address
symbol · chain · exchange · timestamp · received_at
payload · data_quality · collector_version

Data quality is tracked, not assumed

Every point carries source, ingestion/event timestamps, freshness, completeness, confidence and validation status — so the system can detect stale feeds, delayed or duplicate events, inconsistent symbols, broken WebSockets and anomalous values.