The bug report was three words: "Opus is broken." It wasn't. What had broken was an assumption. We had one code path that set thinking.type: "enabled" on every Claude model that supported extended reasoning. Anthropic shipped Opus 4.7, which rejects that exact field with a 400. It only accepts thinking.type: "adaptive". One model, one renamed enum value, and every request routed to it died at the wire.

That's the whole job of a unified LLM provider layer, compressed into one incident. You are not writing an HTTP client. You are maintaining a running model of how twenty-one different providers disagree about the same four concepts — messages, tools, tokens, and errors — and absorbing every change they make so the code above you never has to care.

This is the engineering deep-dive on octolib, the open-source Rust library that does that absorbing for Octocode, Octomind, and Octobrain. I'm going to walk through the lessons that cost us the most, each tied to a real piece of the code. If you've read the introduction to octolib or the sibling piece on running one AI agent across many models, this is the layer underneath both.

The shape of the abstraction

The contract is one trait. Everything below it is a provider that implements it.

#[async_trait::async_trait]
pub trait AiProvider: Send + Sync {
    fn name(&self) -> &str;
    fn supports_model(&self, model: &str) -> bool;
    async fn chat_completion(&self, params: ChatCompletionParams) -> Result<ProviderResponse>;
    fn get_api_key(&self) -> Result<String>;
    // capability probes with defaults: supports_caching, supports_vision,
    // supports_structured_output, supported_sampling_params, get_model_pricing...
}

A caller never constructs a provider directly. It hands a provider:model string to the factory:

let (provider, model) = ProviderFactory::get_provider_for_model("anthropic:claude-opus-4-8")?;
let response = provider.chat_completion(params).await?;

ProviderFactory::parse_model splits on the first colon — provider prefix is required, by design. There is no "default provider" guessing. anthropic:claude-opus-4-8, deepseek:deepseek-chat, openrouter:anthropic/claude-opus-4.8, cli:claude/... — the prefix is the routing key and the model name is everything after it. Twenty-one providers register in one match:

match provider_name.to_lowercase().as_str() {
    "openai"    => Ok(Box::new(OpenAiProvider::new())),
    "anthropic" => Ok(Box::new(AnthropicProvider::new())),
    "deepseek"  => Ok(Box::new(DeepSeekProvider::new())),
    "moonshot" | "kimi" => Ok(Box::new(MoonshotProvider::new())),
    "octohub"   => Ok(Box::new(OctoHubProvider::new())),
    // ... cerebras, groq, together, nvidia, cloudflare, fireworks,
    //     featherless, minimax, zai, byteplus, amazon, google,
    //     ollama, local, openrouter
    _ => Err(anyhow::anyhow!("Unsupported provider: {}", provider_name)),
}

The default trait methods are the unsung half of this. A provider that does nothing special inherits sane behavior: supports_caching returns false, supported_sampling_params returns SamplingSupport::ALL, pricing falls back to a reference table. A provider that is special overrides exactly the methods where it diverges and nothing else. That's the whole design philosophy — make the common case free and the weird case local.

Lesson zero, the one everything else rests on: the abstraction is not "an LLM API." It's the set of capability questions you'll need to ask before you build a request. We didn't get that list right on day one. We grew it every time a provider surprised us, and the trait is the scar tissue.

Lesson 1: Tool calls are the same idea in two incompatible shapes

If you abstract one thing carefully, make it tool calling. It's where providers diverge most and where a silent mismatch produces the worst failure mode: the model asked to call a tool, your code didn't notice, the agent stalls.

Two vendors, same feature, different wire format:

  • Anthropic returns a tool_use content block. Arguments live in an input field that is already a JSON object.
  • OpenAI and everyone OpenAI-shaped return a tool_calls array where function.arguments is a JSON string you have to parse yourself — and which, when a model truncates or hallucinates, is not always valid JSON.

octolib models this divergence explicitly instead of pretending it away. The internal type is a tagged enum:

#[serde(tag = "provider")]
pub enum ProviderToolCalls {
    Anthropic  { content: Vec<AnthropicToolUse> },   // input: serde_json::Value
    OpenAI     { tool_calls: Vec<OpenAIToolCall> },   // function.arguments: String
    OpenRouter { tool_calls: Vec<OpenAIToolCall> },
    DeepSeek   { tool_calls: Vec<OpenAIToolCall> },
    Generic    { calls: Vec<GenericToolCall> },
}

extract_from_exchange dispatches on the provider name, and to_tool_calls() collapses all five variants into one flat Vec<ToolCall> for the caller. The OpenAI branch is where the lesson lives:

let arguments: serde_json::Value = if call.function.arguments.is_empty() {
    serde_json::Value::Object(serde_json::Map::new())
} else {
    serde_json::from_str(&call.function.arguments)
        .map_err(ToolCallError::InvalidArguments)?
};

An empty argument string is legal and means "no arguments" — not an error. A malformed one is a real error. We had to distinguish those two cases the hard way, after a model emitted "" for a zero-arg tool and an early version treated it as a parse failure and dropped the call.

There's a second, subtler trap: round-tripping. When you send the conversation back for the next turn, the assistant's previous tool calls have to be re-encoded in that provider's shape. So octolib stores tool calls in one canonical GenericToolCall form in history, and each provider's convert_messages rebuilds its native blocks on the way out — Anthropic reconstructs tool_use blocks with the object intact; the OpenAI-compat path re-serializes arguments back into a string. The tolerant fallback matters here too:

serde_json::from_str(&call.function.arguments).unwrap_or_else(|_| {
    serde_json::json!({"raw_arguments": call.function.arguments})
})

Rather than lose a malformed tool call, we preserve the raw string under a raw_arguments key. The model can sometimes recover from seeing its own broken output. It can never recover from a call that silently vanished.

Lesson 2: Token usage is where money quietly leaks

Every provider reports usage. No two report it the same way. If you want per-request cost — and at thousands of requests across a dozen providers, you do — you have to normalize a genuinely messy input into one struct:

pub struct TokenUsage {
    pub input_tokens: u64,        // CLEAN — never includes cache tokens
    pub cache_read_tokens: u64,
    pub cache_write_tokens: u64,
    pub output_tokens: u64,
    pub reasoning_tokens: u64,
    pub total_tokens: u64,
    pub cost: Option<f64>,
    pub request_time_ms: Option<u64>,
}

The deserialization struct for OpenAI-compatible providers tells the real story. A single usage object has to tolerate every dialect at once:

struct OpenAiCompatUsage {
    input_tokens: Option<u64>,       // some providers
    prompt_tokens: Option<u64>,      // OpenAI classic
    completion_tokens: Option<u64>,
    output_tokens: Option<u64>,      // others
    total_tokens: Option<u64>,
    reasoning_tokens: Option<u64>,
    completion_tokens_details: Option<CompletionTokensDetails>,  // reasoning here...
    prompt_tokens_details: Option<PromptTokensDetails>,          // cached_tokens here
    total_cost: Option<f64>, cost: Option<f64>,                  // or here
    prompt_cost: Option<f64>, completion_cost: Option<f64>,      // ...or split across these
}

The extraction is a chain of .or() fallbacks: input is input_tokens or prompt_tokens or Ollama's prompt_eval_count or zero. Reasoning tokens hide in a top-level field or nested under completion_tokens_details. Cost is total_cost or cost or prompt_cost + completion_cost, and only if the provider config opts into trusting upstream cost at all. When the API returns no cost field, octolib computes it from a per-model pricing table.

Two things bit us specifically here, both encoded in comments in the source now because we never want to relearn them:

  • input_tokens must be "clean." Some APIs fold cached tokens into the input count; some don't. We subtract cache_read_tokens ourselves so the field always means the same thing, or cost math double-counts the cheap cached tokens at the expensive rate.
  • Anthropic's cache TTL split. Anthropic bills 5-minute cache writes at 1.25× input and 1-hour writes at 2×. It returns a per-TTL cache_creation breakdown — but only sometimes. When that nested object is absent, the safe assumption is 5m, not 1h. Guess 1h and you over-charge your own cost accounting by ~60% on the most common usage. That's a one-line default with a four-line comment explaining why.

The boring lesson: cost tracking is not a feature you bolt on. It's a normalization problem disguised as arithmetic, and the arithmetic is the easy part.

Lesson 3: We don't stream, and that was the right call

This one is contrarian, so let me be precise about it. Across the providers that take a stream flag, octolib sends stream: false. The comment in the DeepSeek adapter is blunt: // We don't support streaming in octolib yet. We buffer the entire response before returning it.

For a library whose job is agent loops and structured output, this is a feature, not a gap. The reason is in the shared HTTP helper:

pub(super) async fn send_and_read(
    request: reqwest::RequestBuilder,
    timeout: Option<Duration>,
) -> anyhow::Result<CapturedResponse> {
    let response = apply_request_timeout(request, timeout).send().await?;
    let status = response.status();
    let headers = response.headers().clone();
    let body = response.text().await?;   // body read INSIDE the retried unit
    Ok(CapturedResponse { status, headers, body })
}

Both the send() and the full body read happen inside the same awaited unit that the retry loop wraps. This matters because the nastiest transport failure is a mid-stream RST — the connection dies after headers arrive but while the body is still coming. A firewall inspecting the payload, an overloaded load balancer, a NAT timeout on a long generation.

If you stream, that failure surfaces deep in your token handler, half a response already emitted, with no clean way to retry. Because we read the body inside the retried unit, is_connection_error classifies the failure (is_connect() || is_request() || is_body()), the loop refreshes the HTTP client, and the whole request retries cleanly.

The tradeoff is honest: you don't get first-token latency, and you can't render output progressively. For a chat UI, stream. For an agent that's going to parse the whole response into tool calls or a JSON schema before doing anything anyway, buffering is simpler, more correct under flaky networks, and removes an entire class of partial-state bugs. We chose correctness. Streaming is on the list; it has never been urgent.

Lesson 4: Retries are easy. Knowing what's retryable is not.

The backoff itself is unremarkable — base_timeout * 2^attempt, capped at five minutes, cancellable via a watch token:

pub fn is_retryable_status(status: u16) -> bool {
    status == 429 || status >= 500
}

Retry 429 and 5xx. Never retry a 4xx — a bad request stays bad. That part is twelve lines.

The hard-won part is the connection-error path. A pooled HTTP connection that the upstream silently half-closed produces a reqwest error that is is_request() = true, is_connect() = false. If you only check is_connect(), you never refresh the pool, and every retry reuses the same dead socket — so all your retries fail identically and you conclude the provider is down when it isn't. We learned this from CN-hosted endpoints (DeepSeek, Moonshot) that drop idle keep-alive connections aggressively. The fix is to treat is_connect(), is_request(), and is_body() all as connection errors, and to atomically swap in a fresh client before the next attempt:

if on_connection_error(&e) {
    crate::llm::providers::shared::refresh_http_client();  // new pool, no stale sockets
}

The client itself is tuned for exactly this: 20s connect timeout (bounding handshake, not generation, which can take minutes), TCP keepalive, HTTP/2 PING frames, and a 30s pool idle timeout to evict connections before the upstream edge kills them. None of that is glamorous. All of it is the difference between a library that survives a bad network afternoon and one that pages you.

Lesson 5: Per-model quirks belong in data, not in if statements scattered everywhere

Back to the opening incident. The fix wasn't a hack — it was admitting that "which sampling and thinking knobs does this model accept" is data about models, and putting it where data goes. In the Anthropic adapter, that's a handful of const slices:

const NO_SAMPLING_MODELS: &[&str]     = &["fable-5", "mythos-5", "opus-4-8", "opus-4-7"];
const ADAPTIVE_ONLY_MODELS: &[&str]   = &["fable-5", "mythos-5", "opus-4-7"];   // manual thinking → 400
const NO_TOP_P_MODELS: &[&str]        = &["opus-4-1", "opus-4-7", "sonnet-4-5", /* ... */];
const EFFORT_PARAM_MODELS: &[&str]    = &["fable-5", "mythos-5", "opus-4-7", "opus-4-6", /* ... */];

The generic mechanism that makes this composable is SamplingSupport — a tiny boolean mask a provider returns per model, declaring which of temperature, top_p, top_k the model will even accept:

pub struct SamplingSupport { pub temperature: bool, pub top_p: bool, pub top_k: bool }
// SamplingSupport::ALL, ::NONE, ::TEMPERATURE_ONLY, ::TEMPERATURE_AND_TOP_P

The trait merges the user's requested values with this mask in effective_sampling_params. A model that rejects top_p simply never has top_p in its request body — no caller anywhere writes if model == .... There's a related subtlety we only found by getting a 400: some Anthropic models reject any non-default sampling value once thinking is on, so the adapter decides thinking_enabled up front and clamps sampling accordingly, before building the request. The lesson generalizes past Anthropic: the moment you have two if model.contains(...) checks for the same concept, you have a data table trying to be born. Let it.

Where OctoHub fits: the proxy on top of the layer

The natural question once you have a unified client: what if the unification lived on the network, so non-Rust clients got it too? That's OctoHub — a high-performance LLM proxy server that is, quite literally, octolib with an HTTP face and a database.

OctoHub's proxy engine doesn't reimplement any provider. It depends on octolib and calls the same factory:

let (provider_name, model) = self.config.resolve_model(&req.model)?;
let provider = ProviderFactory::create_provider(&provider_name)?;
let response = provider.chat_completion(params).await?;

What OctoHub adds is the operational layer around that call: per-tenant API keys, full request/response logging to SQLite/MySQL/Postgres, usage analytics, model-name mapping (a short alias resolves to a list of provider:model strings, one chosen at random for crude load balancing), and per-provider concurrency limits with upstream timeouts. It speaks a Responses-API shape (/v1/completions, previous_completion_id for multi-turn) and an OpenAI-compatible /v1/chat/completions for clients that already speak that.

And then it closes the loop: octolib also ships an octohub: provider. So a Rust app using octolib can route through an OctoHub instance — which routes back out through octolib to the real upstream — getting centralized keys, logging, and cost tracking for free, with the same provider:model string everywhere.

The octohub provider advertises every capability as supported (supports_caching, supports_vision, supports_structured_output all true) precisely because it can't know what's behind the proxy; the real upstream returns an explicit error if the underlying model can't honor a request. The proxy is deliberately honest about its own ignorance.

This is the payoff of getting the trait boundary right. The same abstraction serves an in-process library call and a network hop, and they compose without either side knowing which it's talking to.

What I'd tell someone starting this today

Lesson The concrete thing
Abstract capabilities, not endpoints The trait is a list of questions (supports_caching, supported_sampling_params), not just chat_completion
Model tool-call divergence explicitly A tagged enum per provider shape beats one lossy "universal" struct
Normalize usage before you trust cost .or() chains across prompt_tokens/input_tokens/prompt_eval_count; keep input_tokens cache-clean
Buffer if you're parsing anyway Reading the body inside the retried unit makes mid-stream RSTs retryable
Refresh the pool on connection errors is_request() and is_body() are connection errors too, not just is_connect()
Quirks are data const model slices and a SamplingSupport mask, never scattered if model.contains

None of this is novel. All of it is the kind of thing you only write down after it's bitten you, which is why most of these lessons live as four-line comments above one-line defaults in the actual source. The hardest part of a unified LLM provider layer isn't the abstraction. It's staying honest about how much the things underneath it actually differ — and refusing to paper over a real difference with a hopeful default.

FAQ

Is octolib a Rust LLM library I can use standalone?

Yes. It's open source under Apache-2.0, compiles with cargo check, and the examples run with just an API key. The provider:model string is the entire interface. It does LLM inference, embeddings, and reranking — nothing else, no agents or chains.

Which providers does the unified layer support?

Twenty-one LLM providers register in the factory, including OpenAI, Anthropic, Google, DeepSeek, Moonshot, MiniMax, Z.ai, Groq, Cerebras, Together, NVIDIA, Cloudflare, Fireworks, Featherless, BytePlus, Amazon Bedrock, OpenRouter, OctoHub, Ollama, local endpoints, and a cli: backend. Native providers get bespoke adapters; the rest share an OpenAI-compatible path.

Does it stream tool calls?

Not today — octolib buffers the full response (stream: false) so mid-stream connection resets become cleanly retryable, which matters more for agent loops and structured output than first-token latency. Streaming is on the roadmap.

How is OctoHub different from octolib?

octolib is the in-process Rust library. OctoHub is a proxy server built on top of it — same provider routing, plus multi-tenant keys, request logging, usage analytics, and load balancing. octolib can even call OctoHub via its octohub: provider.

— Don

octolib and OctoHub are open source under Apache-2.0. If a provider you need is missing, the template lives in src/llm/providers/openai.rs — open an issue or a PR.