# Lessons From Building a Unified LLM Provider Layer in Rust > What we learned shipping octolib, the Rust LLM library behind our AI stack: how to abstract multiple LLM providers, normalize tool calls and token usage, and survive the day a model's thinking format returned a 400. The bug report was three words: "Opus is broken." It wasn't. What had broken was an assumption. We had one code path that set `thinking.type: "enabled"` on every Claude model that supported extended reasoning. Anthropic shipped Opus 4.7, which rejects that exact field with a `400`. It only accepts `thinking.type: "adaptive"`. One model, one renamed enum value, and every request routed to it died at the wire. That's the whole job of a unified LLM provider layer, compressed into one incident. You are not writing an HTTP client. You are maintaining a running model of how twenty-one different providers disagree about the same four concepts — messages, tools, tokens, and errors — and absorbing every change they make so the code above you never has to care. This is the engineering deep-dive on [octolib](https://github.com/Muvon/octolib), the open-source Rust library that does that absorbing for [Octocode](/blog/octolib-the-engine-behind-our-ai-stack), Octomind, and Octobrain. I'm going to walk through the lessons that cost us the most, each tied to a real piece of the code. If you've read [the introduction to octolib](/blog/octolib-the-engine-behind-our-ai-stack) or the sibling piece on [running one AI agent across many models](/blog/running-one-ai-agent-across-many-models), this is the layer underneath both. --- ## The shape of the abstraction The contract is one trait. Everything below it is a provider that implements it. ```rust #[async_trait::async_trait] pub trait AiProvider: Send + Sync { fn name(&self) -> &str; fn supports_model(&self, model: &str) -> bool; async fn chat_completion(&self, params: ChatCompletionParams) -> Result; fn get_api_key(&self) -> Result; // capability probes with defaults: supports_caching, supports_vision, // supports_structured_output, supported_sampling_params, get_model_pricing... } ``` A caller never constructs a provider directly. It hands a `provider:model` string to the factory: ```rust let (provider, model) = ProviderFactory::get_provider_for_model("anthropic:claude-opus-4-8")?; let response = provider.chat_completion(params).await?; ``` `ProviderFactory::parse_model` splits on the first colon — provider prefix is **required**, by design. There is no "default provider" guessing. `anthropic:claude-opus-4-8`, `deepseek:deepseek-chat`, `openrouter:anthropic/claude-opus-4.8`, `cli:claude/...` — the prefix is the routing key and the model name is everything after it. Twenty-one providers register in one `match`: ```rust match provider_name.to_lowercase().as_str() { "openai" => Ok(Box::new(OpenAiProvider::new())), "anthropic" => Ok(Box::new(AnthropicProvider::new())), "deepseek" => Ok(Box::new(DeepSeekProvider::new())), "moonshot" | "kimi" => Ok(Box::new(MoonshotProvider::new())), "octohub" => Ok(Box::new(OctoHubProvider::new())), // ... cerebras, groq, together, nvidia, cloudflare, fireworks, // featherless, minimax, zai, byteplus, amazon, google, // ollama, local, openrouter _ => Err(anyhow::anyhow!("Unsupported provider: {}", provider_name)), } ``` The default trait methods are the unsung half of this. A provider that does nothing special inherits sane behavior: `supports_caching` returns `false`, `supported_sampling_params` returns `SamplingSupport::ALL`, pricing falls back to a reference table. A provider that _is_ special overrides exactly the methods where it diverges and nothing else. That's the whole design philosophy — make the common case free and the weird case local. **Lesson zero, the one everything else rests on:** the abstraction is not "an LLM API." It's the set of capability questions you'll need to ask before you build a request. We didn't get that list right on day one. We grew it every time a provider surprised us, and the trait is the scar tissue. --- ## Lesson 1: Tool calls are the same idea in two incompatible shapes If you abstract one thing carefully, make it tool calling. It's where providers diverge most and where a silent mismatch produces the worst failure mode: the model asked to call a tool, your code didn't notice, the agent stalls. Two vendors, same feature, different wire format: - **Anthropic** returns a `tool_use` content block. Arguments live in an `input` field that is **already a JSON object**. - **OpenAI and everyone OpenAI-shaped** return a `tool_calls` array where `function.arguments` is a **JSON string you have to parse yourself** — and which, when a model truncates or hallucinates, is not always valid JSON. octolib models this divergence explicitly instead of pretending it away. The internal type is a tagged enum: ```rust #[serde(tag = "provider")] pub enum ProviderToolCalls { Anthropic { content: Vec }, // input: serde_json::Value OpenAI { tool_calls: Vec }, // function.arguments: String OpenRouter { tool_calls: Vec }, DeepSeek { tool_calls: Vec }, Generic { calls: Vec }, } ``` `extract_from_exchange` dispatches on the provider name, and `to_tool_calls()` collapses all five variants into one flat `Vec` for the caller. The OpenAI branch is where the lesson lives: ```rust let arguments: serde_json::Value = if call.function.arguments.is_empty() { serde_json::Value::Object(serde_json::Map::new()) } else { serde_json::from_str(&call.function.arguments) .map_err(ToolCallError::InvalidArguments)? }; ``` An empty argument string is legal and means "no arguments" — not an error. A malformed one is a real error. We had to distinguish those two cases the hard way, after a model emitted `""` for a zero-arg tool and an early version treated it as a parse failure and dropped the call. There's a second, subtler trap: **round-tripping**. When you send the conversation back for the next turn, the assistant's previous tool calls have to be re-encoded in _that provider's_ shape. So octolib stores tool calls in one canonical `GenericToolCall` form in history, and each provider's `convert_messages` rebuilds its native blocks on the way out — Anthropic reconstructs `tool_use` blocks with the object intact; the OpenAI-compat path re-serializes arguments _back_ into a string. The tolerant fallback matters here too: ```rust serde_json::from_str(&call.function.arguments).unwrap_or_else(|_| { serde_json::json!({"raw_arguments": call.function.arguments}) }) ``` Rather than lose a malformed tool call, we preserve the raw string under a `raw_arguments` key. The model can sometimes recover from seeing its own broken output. It can never recover from a call that silently vanished. --- ## Lesson 2: Token usage is where money quietly leaks Every provider reports usage. No two report it the same way. If you want per-request cost — and at thousands of requests across a dozen providers, you do — you have to normalize a genuinely messy input into one struct: ```rust pub struct TokenUsage { pub input_tokens: u64, // CLEAN — never includes cache tokens pub cache_read_tokens: u64, pub cache_write_tokens: u64, pub output_tokens: u64, pub reasoning_tokens: u64, pub total_tokens: u64, pub cost: Option, pub request_time_ms: Option, } ``` The deserialization struct for OpenAI-compatible providers tells the real story. A single `usage` object has to tolerate every dialect at once: ```rust struct OpenAiCompatUsage { input_tokens: Option, // some providers prompt_tokens: Option, // OpenAI classic completion_tokens: Option, output_tokens: Option, // others total_tokens: Option, reasoning_tokens: Option, completion_tokens_details: Option, // reasoning here... prompt_tokens_details: Option, // cached_tokens here total_cost: Option, cost: Option, // or here prompt_cost: Option, completion_cost: Option, // ...or split across these } ``` The extraction is a chain of `.or()` fallbacks: input is `input_tokens` **or** `prompt_tokens` **or** Ollama's `prompt_eval_count` **or** zero. Reasoning tokens hide in a top-level field _or_ nested under `completion_tokens_details`. Cost is `total_cost` _or_ `cost` _or_ `prompt_cost + completion_cost`, and only if the provider config opts into trusting upstream cost at all. When the API returns no cost field, octolib computes it from a per-model pricing table. Two things bit us specifically here, both encoded in comments in the source now because we never want to relearn them: - **`input_tokens` must be "clean."** Some APIs fold cached tokens into the input count; some don't. We subtract `cache_read_tokens` ourselves so the field always means the same thing, or cost math double-counts the cheap cached tokens at the expensive rate. - **Anthropic's cache TTL split.** Anthropic bills 5-minute cache writes at 1.25× input and 1-hour writes at 2×. It returns a per-TTL `cache_creation` breakdown — _but only sometimes_. When that nested object is absent, the safe assumption is 5m, not 1h. Guess 1h and you over-charge your own cost accounting by ~60% on the most common usage. That's a one-line default with a four-line comment explaining why. The boring lesson: cost tracking is not a feature you bolt on. It's a normalization problem disguised as arithmetic, and the arithmetic is the easy part. --- ## Lesson 3: We don't stream, and that was the right call This one is contrarian, so let me be precise about it. Across the providers that take a `stream` flag, octolib sends `stream: false`. The comment in the DeepSeek adapter is blunt: `// We don't support streaming in octolib yet`. We buffer the entire response before returning it. For a library whose job is agent loops and structured output, this is a feature, not a gap. The reason is in the shared HTTP helper: ```rust pub(super) async fn send_and_read( request: reqwest::RequestBuilder, timeout: Option, ) -> anyhow::Result { let response = apply_request_timeout(request, timeout).send().await?; let status = response.status(); let headers = response.headers().clone(); let body = response.text().await?; // body read INSIDE the retried unit Ok(CapturedResponse { status, headers, body }) } ``` Both the `send()` and the full body read happen inside the same awaited unit that the retry loop wraps. This matters because the nastiest transport failure is a mid-stream RST — the connection dies _after_ headers arrive but _while_ the body is still coming. A firewall inspecting the payload, an overloaded load balancer, a NAT timeout on a long generation. If you stream, that failure surfaces deep in your token handler, half a response already emitted, with no clean way to retry. Because we read the body inside the retried unit, `is_connection_error` classifies the failure (`is_connect() || is_request() || is_body()`), the loop refreshes the HTTP client, and the _whole request_ retries cleanly. The tradeoff is honest: you don't get first-token latency, and you can't render output progressively. For a chat UI, stream. For an agent that's going to parse the whole response into tool calls or a JSON schema before doing anything anyway, buffering is simpler, more correct under flaky networks, and removes an entire class of partial-state bugs. We chose correctness. Streaming is on the list; it has never been urgent. --- ## Lesson 4: Retries are easy. Knowing what's retryable is not. The backoff itself is unremarkable — `base_timeout * 2^attempt`, capped at five minutes, cancellable via a `watch` token: ```rust pub fn is_retryable_status(status: u16) -> bool { status == 429 || status >= 500 } ``` Retry `429` and `5xx`. Never retry a `4xx` — a bad request stays bad. That part is twelve lines. The hard-won part is the connection-error path. A pooled HTTP connection that the upstream silently half-closed produces a `reqwest` error that is `is_request() = true, is_connect() = false`. If you only check `is_connect()`, you never refresh the pool, and _every retry reuses the same dead socket_ — so all your retries fail identically and you conclude the provider is down when it isn't. We learned this from CN-hosted endpoints (DeepSeek, Moonshot) that drop idle keep-alive connections aggressively. The fix is to treat `is_connect()`, `is_request()`, and `is_body()` all as connection errors, and to atomically swap in a fresh client before the next attempt: ```rust if on_connection_error(&e) { crate::llm::providers::shared::refresh_http_client(); // new pool, no stale sockets } ``` The client itself is tuned for exactly this: 20s connect timeout (bounding handshake, _not_ generation, which can take minutes), TCP keepalive, HTTP/2 PING frames, and a 30s pool idle timeout to evict connections before the upstream edge kills them. None of that is glamorous. All of it is the difference between a library that survives a bad network afternoon and one that pages you. --- ## Lesson 5: Per-model quirks belong in data, not in `if` statements scattered everywhere Back to the opening incident. The fix wasn't a hack — it was admitting that "which sampling and thinking knobs does this model accept" is _data about models_, and putting it where data goes. In the Anthropic adapter, that's a handful of `const` slices: ```rust const NO_SAMPLING_MODELS: &[&str] = &["fable-5", "mythos-5", "opus-4-8", "opus-4-7"]; const ADAPTIVE_ONLY_MODELS: &[&str] = &["fable-5", "mythos-5", "opus-4-7"]; // manual thinking → 400 const NO_TOP_P_MODELS: &[&str] = &["opus-4-1", "opus-4-7", "sonnet-4-5", /* ... */]; const EFFORT_PARAM_MODELS: &[&str] = &["fable-5", "mythos-5", "opus-4-7", "opus-4-6", /* ... */]; ``` The generic mechanism that makes this composable is `SamplingSupport` — a tiny boolean mask a provider returns per model, declaring which of `temperature`, `top_p`, `top_k` the model will even accept: ```rust pub struct SamplingSupport { pub temperature: bool, pub top_p: bool, pub top_k: bool } // SamplingSupport::ALL, ::NONE, ::TEMPERATURE_ONLY, ::TEMPERATURE_AND_TOP_P ``` The trait merges the user's requested values with this mask in `effective_sampling_params`. A model that rejects `top_p` simply never has `top_p` in its request body — no caller anywhere writes `if model == ...`. There's a related subtlety we only found by getting a 400: some Anthropic models reject _any_ non-default sampling value once thinking is on, so the adapter decides `thinking_enabled` up front and clamps sampling accordingly, before building the request. The lesson generalizes past Anthropic: the moment you have two `if model.contains(...)` checks for the same concept, you have a data table trying to be born. Let it. --- ## Where OctoHub fits: the proxy on top of the layer The natural question once you have a unified client: what if the unification lived on the network, so non-Rust clients got it too? That's [OctoHub](https://github.com/Muvon/octohub) — a high-performance LLM proxy server that is, quite literally, octolib with an HTTP face and a database. OctoHub's proxy engine doesn't reimplement any provider. It depends on octolib and calls the same factory: ```rust let (provider_name, model) = self.config.resolve_model(&req.model)?; let provider = ProviderFactory::create_provider(&provider_name)?; let response = provider.chat_completion(params).await?; ``` What OctoHub adds is the operational layer around that call: per-tenant API keys, full request/response logging to SQLite/MySQL/Postgres, usage analytics, model-name mapping (a short alias resolves to a list of `provider:model` strings, one chosen at random for crude load balancing), and per-provider concurrency limits with upstream timeouts. It speaks a Responses-API shape (`/v1/completions`, `previous_completion_id` for multi-turn) and an OpenAI-compatible `/v1/chat/completions` for clients that already speak that. And then it closes the loop: octolib _also ships an `octohub:` provider_. So a Rust app using octolib can route through an OctoHub instance — which routes back out through octolib to the real upstream — getting centralized keys, logging, and cost tracking for free, with the same `provider:model` string everywhere. The `octohub` provider advertises every capability as supported (`supports_caching`, `supports_vision`, `supports_structured_output` all `true`) precisely because _it can't know_ what's behind the proxy; the real upstream returns an explicit error if the underlying model can't honor a request. The proxy is deliberately honest about its own ignorance. This is the payoff of getting the trait boundary right. The same abstraction serves an in-process library call and a network hop, and they compose without either side knowing which it's talking to. --- ## What I'd tell someone starting this today | Lesson | The concrete thing | | ------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | Abstract capabilities, not endpoints | The trait is a list of questions (`supports_caching`, `supported_sampling_params`), not just `chat_completion` | | Model tool-call divergence explicitly | A tagged enum per provider shape beats one lossy "universal" struct | | Normalize usage before you trust cost | `.or()` chains across `prompt_tokens`/`input_tokens`/`prompt_eval_count`; keep `input_tokens` cache-clean | | Buffer if you're parsing anyway | Reading the body inside the retried unit makes mid-stream RSTs retryable | | Refresh the pool on connection errors | `is_request()` and `is_body()` are connection errors too, not just `is_connect()` | | Quirks are data | `const` model slices and a `SamplingSupport` mask, never scattered `if model.contains` | None of this is novel. All of it is the kind of thing you only write down after it's bitten you, which is why most of these lessons live as four-line comments above one-line defaults in the actual source. The hardest part of a unified LLM provider layer isn't the abstraction. It's staying honest about how much the things underneath it actually differ — and refusing to paper over a real difference with a hopeful default. ## FAQ **Is octolib a Rust LLM library I can use standalone?** Yes. It's open source under Apache-2.0, compiles with `cargo check`, and the examples run with just an API key. The `provider:model` string is the entire interface. It does LLM inference, embeddings, and reranking — nothing else, no agents or chains. **Which providers does the unified layer support?** Twenty-one LLM providers register in the factory, including OpenAI, Anthropic, Google, DeepSeek, Moonshot, MiniMax, Z.ai, Groq, Cerebras, Together, NVIDIA, Cloudflare, Fireworks, Featherless, BytePlus, Amazon Bedrock, OpenRouter, OctoHub, Ollama, local endpoints, and a `cli:` backend. Native providers get bespoke adapters; the rest share an OpenAI-compatible path. **Does it stream tool calls?** Not today — octolib buffers the full response (`stream: false`) so mid-stream connection resets become cleanly retryable, which matters more for agent loops and structured output than first-token latency. Streaming is on the roadmap. **How is OctoHub different from octolib?** octolib is the in-process Rust library. OctoHub is a proxy _server_ built on top of it — same provider routing, plus multi-tenant keys, request logging, usage analytics, and load balancing. octolib can even call OctoHub via its `octohub:` provider. — Don --- _[octolib](https://github.com/Muvon/octolib) and [OctoHub](https://github.com/Muvon/octohub) are open source under Apache-2.0. If a provider you need is missing, the template lives in `src/llm/providers/openai.rs` — [open an issue or a PR](https://github.com/Muvon/octolib/issues)._