Introducing OctoHub: One Front Door for Every LLM Your Agents Call

Your agent just ran a task. It made forty model calls. Some went to a local Ollama box, some to Anthropic, one to OpenAI because the local model choked on a long context. The task finished. Now answer me three questions: what exactly did it send, what did it cost, and which upstream actually answered.

If you're like we were, the honest answer is "I'd have to add print statements and run it again." Provider dashboards show you their slice. Your agent logs show you its slice. Nobody shows you the whole request as it crossed the wire, tagged with which key issued it and which provider picked it up. The single most useful view — every request through one pane — is the one nobody gives you, because there's no single place the requests pass through.

OctoHub is that single place. It's a self-hosted LLM proxy you run in front of your agents. They talk to one endpoint; OctoHub talks to whoever you tell it to, and it writes down everything in between. Today we're open-sourcing it.

What it is, in one diagram

            ┌──────────────────┐
 agent   →  │  OctoHub proxy   │  →  openai
            │  (Rust / hyper)  │  →  anthropic
            │                  │  →  ollama (your GPU)
            └──────────────────┘  →  openrouter
                     │
                     └─→ your DB (SQLite / MySQL / PostgreSQL)
                         (api_keys, completions, embeddings)

A single Rust binary built on hyper. Clients hit one HTTP endpoint. The proxy resolves the model name, picks an upstream, forwards the call through octolib — our LLM client library, the same one every Muvon tool uses to talk to models — and persists the request, the response, the token counts, and the latency. The proxy itself is stateless across requests. State lives in two places: octohub.toml for configuration, and a database for keys, logs, and usage.

It is deliberately not a few things. It's not a chat UI — point your application at it. It's not a semantic router that picks the "best" model for a prompt; model choice is the caller's job. It's not a vector store; embeddings pass through and get logged, but OctoHub doesn't index them. It adds auth, logging, load balancing, and a stable interface. It doesn't try to be clever about what the providers return — every meaningful field comes back to the client verbatim.

Why we built it

We didn't set out to build a proxy. We were running Octomind, our agent runtime, against a mix of models — a self-hosted fleet on our own GPUs for the cheap high-volume work, frontier APIs for the hard stuff. The setup worked. What didn't work was seeing it.

The moment you have more than one model behind one application, you lose the single pane. Each provider has its own dashboard, its own token accounting, its own idea of what a "request" is. Your agent has its own logs, which tell you what it thought it sent. When a run costs more than expected, or a model starts returning garbage, you're stitching together three incomplete stories and guessing at the seams.

We wrote more about that specific pain — wanting to see what an agent actually did, not what it claimed — in the observability we wanted. OctoHub is the infrastructure answer to it. Put one proxy in the path and the single pane exists by construction: there's literally one place every request goes through, so there's one place to record it.

The second reason was load balancing. We were already running one agent across many models by hand — config swaps, environment variables, a model name change per run. We wanted a model alias that meant "any of these upstreams, your pick," and a concurrency limit so a burst of agent calls wouldn't melt the local GPU box. Both of those belong in a proxy, not scattered across every client.

How a request flows through it

Take a POST /v1/completions. Here's what OctoHub does with it:

  1. Authenticate. The Authorization: Bearer token is a client API key from the api_keys table. OctoHub looks it up, checks it's active, and tags the request with the key ID. (If you start the server with no master key, only the admin API is disabled — a warning prints at startup; completion and embedding calls still require a valid client key.)
  2. Check the allow-list. Client keys can be restricted to a set of models. A key scoped to ["gpt", "anthropic:claude-haiku-4-5"] asking for anything else gets a 403.
  3. Resolve the model. If the model field is an alias from [models], OctoHub expands it to a provider:model string. If it's already a bare provider:model, it passes straight through. Aliases that map to a list start from a random entry and take the first provider that can admit — that's the load balancing.
  4. Acquire a provider permit. If you've set a concurrency limit for that provider, the request waits for a free slot. No slot, no 429 — the HTTP connection just stays open until one frees up — up to the queue timeout (60 s by default), after which you get a 503. Intentional throttling, recorded as queue-wait time.
  5. Call the upstream through octolib, with a configurable operation deadline.
  6. Persist and respond. The full request, the response, token counts, cost, resolved provider, and latency go into the completions table. The client gets the upstream's answer verbatim, plus an X-Request-Id header.

That X-Request-Id is the join key. It's either a value you passed in (validated, echoed back) or a fresh ULID. It shows up as req_id on every log line for that request. Take an error response, grab the header, grep your logs — you have the whole story.

Model aliases and load balancing

The [models] section is where one name fans out to many upstreams:

[models]
# One alias, one upstream
"sonnet"  = ["anthropic:claude-sonnet-5"]

# One alias, several upstreams — OctoHub picks one at random per request
"workhorse" = ["ollama:kimi-k2.6", "ollama:minimax-m3", "openrouter:google/gemini-3.1-pro-preview"]

[embedding_models]
"voyage" = ["voyage:voyage-4"]

A client asking for workhorse gets one of the three — OctoHub starts from a random entry and takes the first provider whose rate windows admit the request. That's the whole load-balancing model — simple, predictable, and exactly enough to spread agent traffic across a fleet or across keys without a separate router. There's no weighted routing today; providers on an error cooldown get deprioritized behind healthy ones, and chain requests stick to the provider that served the previous turn. We'd rather ship the honest version of that than imply a smarter scheduler than the one that exists.

Clients can also bypass aliases entirely and send a raw provider:model like openai:gpt-5.5. The alias table is a convenience, not a gate — the gate is the per-key allow-list.

The auto model: say why, not which

Since 0.6.0 there's a second way to pick a model, and it's the one our own agents use most. Instead of naming a model at all, the client sends "model": "auto" plus an X-Model-Purpose header — any string you like — and OctoHub resolves the purpose to an alias:

[auto]
default = "workhorse"
compression = "cheap"
supervisor = "sonnet"

Purposes are hierarchical, split on -: supervisor-gate falls back to supervisor, then to default. One supervisor row covers every supervisor-* purpose until you pin a specific one — you define exactly as many rows as you have opinions. A missing or typo'd purpose degrades to default, never fails.

Why bother? Because the caller usually knows what kind of call it's making — a compression pass, a gate check, the main loop — and the operator knows which tier that kind of call deserves. Purpose routing puts that decision in the proxy config (or in a per-owner override map via PUT /v1/admin/owners/:owner/auto, which beats the config floor entirely) instead of hardcoding model names into the agent. Octomind sends main, compression, and the supervisor-* family out of the box.

Per-provider concurrency

Your frontier API can take thirty-two parallel requests without blinking. Your single GPU box running Ollama cannot. So OctoHub caps in-flight requests per provider:

[providers.ollama]
concurrency = 5

[providers.openai]
concurrency = 32

Requests beyond the limit queue inside the OctoHub process — the client connection blocks until a slot opens. Providers you don't list run unthrottled. The limiter is process-local and counts completions and embeddings together, since both flow through the same upstream connection. It's a semaphore per provider, nothing exotic, but it means an agent that fans out forty calls won't knock over the model server those calls land on.

Concurrency isn't the only knob. Each provider also takes fixed-window rate limits — requests_per_minute, tokens_per_minute, requests_per_day, tokens_per_day — and a multi-upstream alias rotates to the next candidate when a window is full. That's the "rate windows" the alias picker checks: one provider hitting its daily token budget doesn't fail the request, it just shifts traffic to the next upstream that still has headroom.

The observability you actually get

This is the part we built OctoHub for, so it gets two surfaces.

Structured logs on stdout — pretty if you're at a TTY, JSON otherwise. Every completed request emits one line:

{
	"level": "INFO",
	"message": "request completed",
	"req_id": "01HMQGSB3R",
	"route": "/v1/completions",
	"status": 200,
	"dur_ms": 1523,
	"api_key_id": 1,
	"model": "workhorse",
	"provider": "ollama",
	"queued_ms": 0,
	"tok_in": 56,
	"tok_out": 120
}

You see the key that issued it, the model name the client asked for, the provider that actually answered, how long it queued, how long it took, and the tokens in and out. The provider field is the answer to "which upstream did the random pick land on" — without re-running anything.

Prometheus metrics on a separate port (127.0.0.1:9090 by default, GET /metrics). Everything is prefixed octohub_:

Metric What it tells you
octohub_requests_total request volume by route, method, status
octohub_request_duration_seconds end-to-end latency histogram
octohub_completions_total completion volume by model, provider, status
octohub_completion_tokens_total tokens in/out by model and provider
octohub_provider_queue_wait_seconds time spent waiting for a concurrency permit
octohub_provider_in_flight active requests at each provider right now

The queue-wait histogram is the leading indicator of saturation — when P99 climbs, your fleet is the bottleneck before any request times out. A couple of PromQL lines give you error rate by model and output tokens/sec by provider. Turn on per_key = true and completion metrics get an api_key_id label, so you can bill or attribute cost per client. (Mind the cardinality if you issue thousands of keys.)

For the full record — not aggregates, the actual prompt and response bytes — the admin API serves raw completion history straight from the database.

Multi-tenant keys and the admin API

OctoHub has two auth layers. The master key (set in octohub.toml) protects the admin API. Client keys — issued through that admin API, stored in the database — authenticate the completion and embedding endpoints. Every completion is tagged with the issuing key, which is what makes per-tenant usage tracking work.

There's a shell wrapper, octohub-admin.sh, for daily operations:

export OCTOHUB_MASTER_KEY=your-master-secret

# Issue a client key, restricted to two models
./octohub-admin.sh keys create ci-pipeline --allowed-models gpt,anthropic:claude-haiku-4-5

# Daily usage for keys 1 and 2
./octohub-admin.sh usage --bucket day --key 1,2

# Pull the last 20 raw completions — full input and output
./octohub-admin.sh completions --limit 20

Keys are revoked, never deleted — usage records are linked to the key ID, so the history survives. Usage rolls up by hour, day, week, or month, filterable by key and time range. The same data is available as plain HTTP if you'd rather not use the script.

Two more operational surfaces worth knowing about. GET /v1/admin/status reports per-model health observed from real traffic — not a synthetic probe, so a single hiccup doesn't paint a red light on your dashboard. And sending the process a SIGHUP reloads octohub.toml in place — new aliases, new limits, no restart.

Run it in 5 minutes

OctoHub is a single binary. Build it, write a config, start it, issue a key.

# 1. Build from source
git clone https://github.com/Muvon/octohub
cd octohub && cargo build --release

Write a minimal octohub.toml:

[server]
host = "127.0.0.1"
port = 8080
api_key = "your-master-secret"        # enables auth + the admin API
db_url = "sqlite://octohub.db"         # schema auto-created on first run

[models]
"workhorse" = ["ollama:kimi-k2.6", "openrouter:google/gemini-3.1-pro-preview"]

[metrics]
enabled = true
bind = "127.0.0.1:9090"

[providers.ollama]
concurrency = 5

Start it and create a client key:

# 2. Start the server (point at the config with -c if it isn't in the cwd)
./target/release/octohub

# 3. Issue a client key
curl -X POST http://127.0.0.1:8080/v1/admin/keys \
  -H "Authorization: Bearer your-master-secret" \
  -H "Content-Type: application/json" \
  -d '{"name": "my-app"}'
# → {"id": 1, "key": "abc...xyz", ...}   ← save this, it's shown once

# 4. Make a completion through the proxy
curl -X POST http://127.0.0.1:8080/v1/completions \
  -H "Authorization: Bearer abc...xyz" \
  -H "Content-Type: application/json" \
  -d '{"model": "workhorse", "input": "Explain Rust in one sentence."}'

The database starts as SQLite — no setup. Point db_url at MySQL or PostgreSQL when you outgrow it; the schema is created automatically on first connection. Provider API keys (OPENAI_API_KEY, ANTHROPIC_API_KEY, and friends) live in the environment, read by octolib exactly as the providers expect.

OctoHub also speaks classic OpenAI Chat Completions at POST /v1/chat/completions, so any OpenAI-compatible SDK or tool points at it as a drop-in base URL. (One honest caveat: streaming isn't implemented — a request with "stream": true gets a 501.)

Pointing Octomind at it

OctoHub and Octomind were built for each other, and octolib ships a native octohub: provider — no OpenAI-compat shim, it speaks OctoHub's Responses API directly. Two environment variables wire them together:

export OCTOHUB_API_URL=http://127.0.0.1:8080   # your OctoHub server
export OCTOHUB_API_KEY=abc...xyz               # a client key you issued

Now any Octomind model reference of the form octohub:<alias> routes through the proxy:

octomind run --model octohub:workhorse developer:general

The agent thinks it's talking to one provider. Behind the proxy, workhorse fans out across your fleet, every call is logged with the cost and the upstream that answered, and the concurrency limit keeps the GPU box upright. The agent stays simple; the visibility lives where the requests actually cross the wire.

And this isn't a lab setup. Octomind Cloud — our managed agent runtime — runs every customer agent through OctoHub in production. Calls arrive tagged with purposes (main, compression, supervisor-gate), the auto model routes each one to the right tier, and every request lands in the log with a cost and the upstream that answered. The same binary you can clone is the one sitting in front of our fleet.

Open source, Rust, Apache-2.0

OctoHub is on GitHub under Apache-2.0. It's a single hyper-based Rust binary — small, fast, and dependency-light. SQLite by default so there's nothing to stand up, MySQL and PostgreSQL when you need them. Configuration is one TOML file plus a handful of OCTOHUB_* environment overrides for the things you tweak per deployment (OCTOHUB_DB_URL, OCTOHUB_LOG_FORMAT, OCTOHUB_METRICS_BIND).

It's early — version 0.6.4, the honest number. It does what it says: one front door, model aliases, purpose-based auto routing, load balancing with failover and cooldown, per-provider concurrency and rate limits, modality checks that skip providers which can't handle your images or video, multi-tenant keys, full request logging, and a Prometheus endpoint. Provider support keeps growing — Google Studio landed in 0.6.2 alongside the existing twenty-plus. It does not yet do streaming or weighted routing, and I'd rather tell you that than let you find out in production.

If you run more than one model behind one application — and if you're building agents, you do — OctoHub gives you the single pane that the rest of the stack quietly assumes exists. Clone it, point an agent at it, and watch a run go by with the cost attached.

— Don

OctoHub is open source under Apache-2.0, developed by Muvon Un Limited. Get it on GitHub — issues and pull requests welcome.