The bill that started this was $312 for a single afternoon. One developer, one Octomind session, one model doing every job: reading files, running greps, summarizing diffs, deciding when to compress the context, and doing the actual hard reasoning. All of it on a frontier model, because that's what you reach for when you want the agent to be good.

Then OpenRouter started returning 429s in the middle of a refactor, and I watched the agent stall on a rate limit while doing something a model a tenth the price could have done perfectly. That's when it clicked: most of what an agent does is not hard.

You don't need your best model for any of that. You need it for maybe ten percent of the calls — the review, the tricky design decision, the diff that touches auth. The other ninety percent is grunt work, and grunt work is exactly where cheap and local models shine.

This is the story of splitting one Octomind agent across many models, with the real config keys it ships with. By the end you'll have a working multi-model setup, a model-selection cheat sheet, and a clear answer to the question that matters: what do you put where.

One string, any backend

Octomind talks to every provider through one library — octolib, which I wrote about earlier. What makes all of this possible is that every model is named the same way: a provider:model string.

model = "anthropic:claude-sonnet-5"
model = "openai:gpt-5.5"
model = "deepseek:deepseek-chat"
model = "openrouter:google/gemini-3.5-flash"
model = "ollama:glm-5"

Same field, same shape, anywhere a model is configured. Octolib parses the prefix, routes to the right backend, and normalizes the response — token usage, cost, structured output — so the rest of Octomind doesn't care which vendor answered.

The provider prefixes octolib recognizes, as of the current release, are:

openrouter  openai      anthropic   google      amazon
cloudflare  deepseek    cerebras    groq        together
fireworks   nvidia      minimax     moonshot    zai
byteplus    featherless octohub     ollama      local

Twenty network providers, plus a special cli meta-provider for local CLI-backed agents (more on that below). moonshot and kimi are aliases for the same provider. New providers land in octolib and become available in Octomind automatically — no Octomind release required.

API keys are environment-only, by the way. You can't put them in the config file; that was removed for security. You export OPENROUTER_API_KEY=... (or drop a .env in the working directory), and octomind config --show tells you which keys it detected and where they came from. One OpenRouter key gets you most of the catalog through a single prefix; specific provider keys get you the rest.

Where a model gets chosen

Before splitting work across models, you need to know the four places a model can be set, and which one wins when more than one applies. Octomind resolves the effective model in this order, highest priority first:

CLI --model  >  role.model  >  config.model

Plus a fourth seam for tap agents: a [taps] entry overrides config.model for a specific tap-agent tag.

  • config.model — the root default. In a freshly generated config it's a Sonnet-class Claude routed through OpenRouter — today you'd set openrouter:anthropic/claude-sonnet-5. Everything falls back to this when nothing more specific is set.
  • role.model — a per-role override. This is the main lever. Each role can name its own model, and a role's model is honored directly over the root default.
  • [taps] override — for tap agents (the category:variant agents you install from a registry), you can pin a model per tag without touching the agent's manifest:
[taps]
"developer:general" = "anthropic:claude-sonnet-5"
"octomind:assistant" = "openai:gpt-5.5"
  • CLI --model — wins over everything for one run: octomind run -m anthropic:claude-sonnet-5. And mid-session you can swap live with /model deepseek:deepseek-chat, or nudge reasoning effort with /effort high.

So the strategy is straightforward: pick a sensible default in config.model, then override per role where a specific job deserves a different model. Most of the savings come from the role layer.

Roles: the unit of model selection

A role in Octomind is a named bundle of behavior: a system prompt, a temperature, a set of MCP servers and tool permissions — and, optionally, a model. Every session runs with a role. The role decides what the agent can do and which model does it.

The shipped config already does this. It defines four plain roles, and three of them deliberately run on cheaper models than the main one — shown here with the model strings bumped to the current generation:

# Lightweight query refinement — make a vague task clearer. No tools.
[[roles]]
name = "task_refiner"
model = "openrouter:openai/gpt-5.4-mini"
# ...

# Reconnaissance — gather context, read files, report findings.
[[roles]]
name = "task_researcher"
model = "openrouter:google/gemini-3.5-flash"
# ...

# Session-history compression — summarize and reduce.
[[roles]]
name = "reduce"
model = "openrouter:openai/gpt-5-mini"
# ...

Read those model choices as a design statement. Refining a vague request is a job for a small, fast model (gpt-5.4-mini). Research that reads a lot of files wants a big, cheap context window (gemini-3.5-flash). Compressing history is a structured summarization task that a small reasoning model handles fine (gpt-5-mini). None of these need a frontier model, and the default config doesn't waste one on them.

You define your own roles the same way. Every field except model is mandatory — system, welcome, temperature, top_p, and top_k all have to be present or the config won't parse:

# Cheap, broad context for sweeping the codebase
[[roles]]
name = "researcher"
model = "openrouter:google/gemini-3.5-flash"
temperature = 0.3
top_p = 0.7
top_k = 20
system = "You gather context and report findings. You do not modify files."
welcome = ""
[roles.mcp]
server_refs = ["filesystem"]
allowed_tools = ["view"]

# Precision where it counts — the review and the hard calls
[[roles]]
name = "reviewer"
model = "anthropic:claude-opus-4-8"
temperature = 0.2
top_p = 0.7
top_k = 20
system = "You review changes for correctness, security, and edge cases."
welcome = "Reviewer ready."
[roles.mcp]
server_refs = ["filesystem"]
allowed_tools = ["view", "ast_grep"]

Now octomind run researcher sweeps the repo on a cheap large-context model, and octomind run reviewer brings the expensive model out only for the work that earns it. Switch between them mid-session with /role reviewer. Same agent, same session, two models — chosen per task.

Layers: chaining models in one pass

Roles let you pick a model per session. Layers let the system chain models automatically inside a single flow.

A layer is an ACP-invocable stage that references a role via its command field. The role carries the model, system prompt, and tools; the layer says how the conversation flows into and out of that stage. The fields are small and all the mode fields are mandatory:

[[layers]]
name = "analysis"
description = "Detailed analysis of code, systems, or requirements"
command = "octomind acp analysis"   # spawns the `analysis` role over ACP
input_mode = "last"                 # last | all | summary
output_mode = "append"              # none | append | replace | last | restart
output_role = "assistant"

Because the layer points at a role and the role carries the model, each layer in a chain can run on a different model. A cheap model gathers context, hands its output to the next stage, a stronger model reasons over it. The same LayerConfig struct backs [[commands]] — layers you trigger interactively with /run <name> — so the shipped reduce command is itself a layer that runs the cheap reduce role to compress history on demand.

The mental model: roles tell you "which model for this session," layers tell you "which model for this stage." Together they let you route a single request through a cheap-then-expensive pipeline without writing orchestration code — it's all TOML.

Note: a [[layers]] entry needs a matching role in [[roles]]. The model lives in the role, never in the layer. If you find yourself wanting to set a model on a layer, you're looking for the role.

Compression has its own model — and it should be cheap

There's one more model selection that's easy to miss and pays for itself immediately. Octomind compresses long sessions automatically, and the decision-plus-summary call that drives compression runs on its own configured model, separate from your main one:

[compression.decision]
model = "openai:gpt-5-mini"   # cheap by design
max_tokens = 16000
temperature = 0.3

The shipped default is openai:gpt-5-mini for a reason: this call fires repeatedly across a long session, and you do not want a frontier model deciding whether to compress. anthropic:claude-haiku-4-5 is a fine alternative if you're already on Anthropic. The point is the same as everywhere else in this post — a routine, structural job gets a routine, cheap model, and the savings compound because it runs often.

The compression engine is itself cost-aware: before compressing, it computes the net benefit using per-model pricing fetched from the provider, and skips compression when it would cost more than it saves. For a free/zero-priced session model — a local Ollama model, say — it always compresses, because cost is irrelevant and you only care about keeping context manageable. Which is a nice segue.

Local models for the bulk work

Not every call needs to leave your machine. Octomind speaks to local models two ways.

Ollama and OpenAI-compatible endpoints via the ollama: and local: prefixes:

model = "ollama:glm-5"
model = "local:my-finetune"   # set LOCAL_API_URL to your endpoint

ollama: points at your local Ollama daemon; local: points at any OpenAI-compatible server via LOCAL_API_URL. Either one is the same provider:model string the rest of the config uses — drop it into a role's model field and that role runs locally.

Local CLI agents via the special cli: meta-provider, which runs a command-line agent on your machine instead of calling a network API:

model = "cli:codex/gpt-5.2-codex"

The format is cli:<backend>/<model>, where the backend is one of codex, claude, cursor, gemini, or a generic command. Because it runs through a local CLI, no API key is required — Octomind skips credential validation entirely for cli: models. Behavior is tuned with backend-specific environment variables (for the codex backend, CODEX_COMMAND, CODEX_REASONING_EFFORT, and friends).

Where local models earn their keep: high-volume, low-stakes work where latency and privacy matter more than the last few points of quality. Bulk summarization, classification, first-pass research, anything you're running thousands of times. Put a local model on a researcher role and the reconnaissance phase costs you nothing per token. Keep the cloud model on the reviewer role where the quality gap actually shows.

OctoHub: routing and fallback underneath

So far the routing is per-role and per-layer, decided in your Octomind config. But there's a layer below that — a real proxy — and it's where fallback, load-balancing, and centralized accounting live.

OctoHub is our open-source LLM proxy. To Octomind it's just another provider: you set a model like octohub:my-model, point Octomind at the OctoHub endpoint, and from Octomind's side it's one more provider:model string. What OctoHub does behind that string is where it gets useful. Its config maps a short model name to a list of fully-qualified provider:model targets, and picks one at random per request:

[models]
# One alias, several real backends — random pick per request = simple load balancing
"fast" = ["minimax:minimax-m3", "ollama:minimax-m3"]

That single mapping gives you load-balancing and a soft form of failover: if one backend is rate-limited or down, the others in the list absorb the traffic. OctoHub also does full request/response logging, multi-tenant API keys (so each developer or service gets its own key and its usage is tracked separately), and aggregated usage analytics by key and time bucket. For a team running many agents, that's the difference between "we spent $4k on LLMs last month" and "the review pipeline on the auth service spent $4k last month, here's the breakdown by key."

Here's how the pieces split:

  • octolib is the library that knows how to talk to each provider — request shape, response parsing, cost tables.
  • OctoHub is the proxy that sits in front of providers and decides which one a request hits, logs it, and meters it per key.
  • Octomind is the agent that decides, per role and per layer, which model string to send in the first place.

You can run Octomind with no proxy at all and route purely through roles and OpenRouter. You add OctoHub when you want centralized control: one place to rotate keys, rebalance traffic, and see where the money went.

The model-selection cheat sheet

Here's where I'd actually put each model class. Treat the specific model names as current examples, not commandments — the right model this quarter won't be the right model next quarter, but the shape of the decision holds.

Job in the agent Model class Concrete example Why
Final review, hard design calls, auth-adjacent diffs Frontier anthropic:claude-opus-4-8 Quality is worth the price on the 10% that matters
Main development loop Strong, cached anthropic:claude-sonnet-5 Good coding + prompt caching keeps cost sane
Research / context sweeps Large-context, cheap openrouter:google/gemini-3.5-flash Reads a lot of files, doesn't need to be brilliant
Task refinement Small, fast openrouter:openai/gpt-5.4-mini Clean up a vague request — trivial work
Compression decision Small, cheap openai:gpt-5-mini Fires constantly; never use a frontier model here
Session reduction / summaries Small reasoning openrouter:openai/gpt-5-mini Structured summarization, runs often
Bulk / high-volume / private Local ollama:glm-5, cli:codex/gpt-5.2-codex Latency + privacy + zero per-token cost
Cost-effective cloud fallback Cheap native deepseek:deepseek-chat Lowest cost when you still need a real API

The rule underneath the table: spend on calls in proportion to how much the output quality changes the outcome. The review either catches the bug or it doesn't — spend there. The file listing comes back the same from any model — don't.

A complete multi-model config

Putting it together, here's a config that splits work across four models — a cheap default, a strong reviewer, a local researcher, and a cheap compression decision — plus hard spending caps so a runaway loop can't surprise you:

# Cheap, capable default routed through one OpenRouter key
model = "openrouter:anthropic/claude-sonnet-5"

# Hard caps — enforced, not advisory
max_request_spending_threshold = 0.50   # USD per request, stops execution
max_session_spending_threshold = 5.00   # USD per session, prompts/halts

# Compression runs on a cheap model, separate from the main one
[compression.decision]
model = "openai:gpt-5-mini"
max_tokens = 16000
temperature = 0.3

# Local model for the high-volume reconnaissance phase
[[roles]]
name = "researcher"
model = "ollama:glm-5"
temperature = 0.3
top_p = 0.7
top_k = 20
system = "You gather context and report findings. You do not modify files."
welcome = ""
[roles.mcp]
server_refs = ["filesystem"]
allowed_tools = ["view"]

# Frontier model for the work that earns it
[[roles]]
name = "reviewer"
model = "anthropic:claude-opus-4-8"
temperature = 0.2
top_p = 0.7
top_k = 20
system = "You review changes for correctness, security, and edge cases."
welcome = "Reviewer ready."
[roles.mcp]
server_refs = ["filesystem"]
allowed_tools = ["view", "ast_grep"]

Validate it before you trust it:

octomind config --validate
octomind config --show     # confirms which provider keys were detected

Then run the cheap path by default and call the expensive one only when you mean to:

octomind run researcher              # local model, free per token
octomind run reviewer                # frontier model, on purpose
octomind run -m deepseek:deepseek-chat   # one-off override for a session

Watch what it actually costs with /info for the session overview and /report for the per-request breakdown. The first time you run a real task this way and see the research phase cost nothing while the review cost a few cents, the split stops being theoretical.

What changed after we split

The $312 afternoon doesn't happen anymore, and not because we use the agent less. The research and refinement traffic moved to cheap and local models, the compression decision stopped riding a frontier model, and the expensive model now only shows up for review and the genuinely hard calls. The frontier model's share of total calls dropped to roughly the ten percent it was always supposed to be.

The rate-limit stalls went away too, which I didn't expect. Spreading traffic across providers — some through OpenRouter, some direct, some local, some behind OctoHub's random-pick mapping — means no single provider's 429 stops the whole session. The agent that used to freeze on one upstream's rate limit now just routes around it.

None of this required a framework, a router library, or orchestration code. It's provider:model strings in TOML, a few extra roles, and one cheap compression model. The hard part wasn't the wiring — octolib and Octomind already had all of it. The hard part was the mindset shift: stop paying frontier prices for grunt work, and put your best model only where the output quality actually changes what happens next.

If you're wrapping project-specific tools for these agents too, the custom-MCPs-in-your-repo pattern pairs cleanly with multi-model roles — narrow tools and right-sized models pull in the same direction.

FAQ

Do I need OpenRouter, or can I use providers directly?

Either. One OPENROUTER_API_KEY gets you most of the catalog through the openrouter: prefix, which is the easiest start. Or set individual keys (ANTHROPIC_API_KEY, DEEPSEEK_API_KEY, …) and use the native prefixes. You can mix both in one config — different roles can hit different providers entirely.

How do I switch models without editing config?

Per run: octomind run -m anthropic:claude-sonnet-5. Mid-session: /model deepseek:deepseek-chat. The CLI flag wins over role and config models; the slash command swaps the live session and saves it.

Where exactly does a model get decided?

Resolution order is CLI --model > role.model > config.model, with a [taps] override slotting in at the config.model tier for tap agents. Set a sensible config.model default, then override per role for the jobs that deserve a different model.

Can different steps of one task use different models?

Yes — that's what layers are for. A layer references a role, the role carries the model, so a chain of layers runs a chain of models. Cheap model gathers context, strong model reasons over it, all in one flow.

Will local models break cost tracking?

No. A free/zero-priced model is handled explicitly: cost tracking reports zero, and the compression engine always compresses for such models (since the cost gate is moot) to keep context manageable.

What does OctoHub add that roles don't?

Roles route in your config; OctoHub routes at the proxy. It maps one alias to several real backends with random load-balancing, logs every request, issues per-key API keys for multi-tenant usage tracking, and aggregates analytics. You want it when a team needs one place to rebalance traffic and account for spend.

The point was never "use a cheaper model." It's "use the right model for each call, and let the routing be config, not code." Octomind hands you the seams — root default, per-role, per-layer, per-run, plus a proxy underneath — and one provider:model string ties them all together.

Spend your best model where the answer actually matters. Let everything else run cheap.

— Don

Octomind is open source under Apache-2.0, and so are octolib and OctoHub. Multi-model routing is config, not a plugin — if a provider you need is missing, it lands in octolib and shows up in Octomind automatically. Open an issue if something here doesn't hold.