LLM & model layer

Every agent reaches a model through one internal adapter — never a concrete provider SDK. Requirement: change model or provider by configuration, with zero code change.

Provider abstraction

Multiple providers behind a single interface:

• Anthropic, OpenAI, Google, plus cost-efficient open-weight providers — DeepSeek, Alibaba Qwen, Moonshot (Kimi) — that often lead on quality-per-dollar;
• locally hosted models via Ollama or vLLM where privacy or cost demands it;
• structured outputs validated against a schema, so the engine consumes a stable format.

The model router

Routing happens at the task node, not once per workflow, and follows a fixed order — the cheapest sufficient path wins, and the router never burns an expensive model just to decide which model to call:

After a hard-constraint filter (availability, context window, schema reliability, tool support, privacy, region, max cost), the survivors are ranked by a configurable score whose weights vary by workflow:

route_score =
    quality_weight     × task_quality_score
  + reliability_weight × historical_success_rate
  + latency_weight     × normalized_latency_score
  + privacy_weight     × privacy_score
  - cost_weight        × normalized_expected_cost
  - failure_weight     × recent_error_rate

Routing policy by task

Model capability registry

Every usable model is registered with measurable characteristics, not chosen by brand. The router reads these fields plus per-task benchmarks (quality, schema-success, p50/p95 latency, cost-per-valid-result):

• provider · model_id · version
• hosting_type (cloud_api / local_cpu / local_gpu / remote_gpu)
• supported_tasks · supports_tools · supports_structured_output
• context_window
• input / cached / output price
• estimated local cost per hour
• privacy_class · fallback_group
• task-specific quality score

Open-source progression

Open source is introduced gradually and is not assumed free — GPU, electricity, utilization, latency and maintenance are all costed. A local model serves production traffic only after it passes task-specific acceptance thresholds:

1. 1
   Paid APIs for fast iteration & benchmark creation
2. 2
   Local small model in shadow mode — cannot affect decisions
3. 3
   Local-first for low-risk tasks (extract, classify, summarize, compress)
4. 4
   Cloud fallback + one bounded escalation for high-value ambiguity
5. 5
   Dedicated GPU only when measured savings justify it

The frontier open-weight families — DeepSeek, Qwen and Kimi — are the natural candidates down this ladder: cheap as a hosted API today, and the exact same weights can later be pulled in-house, so a shadow benchmark on the hosted version directly predicts the self-hosted one.

Serving engines stay replaceable behind one internal OpenAI-compatible gateway: Ollama / llama.cpp for local and quantized models, vLLM for future dedicated GPU. LiteLLM may be evaluated for unified access and fallbacks, but Clawlas owns the routing policy — never delegated to a third-party router.

Evaluation-driven routing

The router learns from Clawlas-specific evaluations, not vendor benchmarks:

• a versioned golden dataset per task — easy, ambiguous and adversarial cases;
• expected structured outputs recorded; models compared on accuracy, schema validity, latency and cost;
• shadow-test → canary traffic → full change, with instant rollback;
• a model never promotes itself to a higher-trust role.

Agent-loop cost control

A single analysis can fan out across many agents; if each gets the same large context, tokens multiply. The loop is bounded:

• cap agent steps and tool calls; one shared structured state, not repeated prose;
• at most one premium-model call per normal opportunity; compress long histories;
• retrieve only relevant memory; prefer a deterministic validator over a second LLM;
• stop when evidence is insufficient rather than research indefinitely.

Prompt versioning

Every prompt has an identifier, version, purpose, expected schema, model compatibility, evaluation results and status. Prompt changes are tracked like code changes.