Octolib 0.39.0: A Model That Cannot Write a Sentence Just Joined the Stack

Since the day we extracted it from Octocode, every model call in our stack has been one of three things. Generate text. Turn text into a vector. Rank vectors against a query. Octolib — the Rust library under Octomind, OctoHub, Octocode and Octobrain — had a module for each: llm, embeddings, reranker. Then it grew media.

Today it grows a fourth, and this one isn't like the others. octolib::evaluation talks to a model that can't generate a single token. You hand it a state and a set of typed questions, and it hands back probabilities. Yes or no, one of N, a position on a scale. In 70 to 500 milliseconds. For $0.042 per million input tokens, with output free — because there's no output to bill.

The model is Jev, from TypeSafe AI, which came out of stealth on September 15. TypeSafe calls the category System One models. We think it's the most interesting new primitive to land in the AI stack this year, and Octolib 0.39.0 is our bet on it: one provider:model string, three ways to reach it, and the same cost accounting we apply to everything else.

That's the headline. The release also turns Cloudflare Workers AI into a full media provider, adds a local ONNX embedding backend, brings in two new LLM providers, and finally makes structured output validate exactly once. Everything since 0.36.1 is below.

Why an agent needs a model that only decides

Strip a coding agent down and there's one big model doing the work. Around it sits a control plane making small, bounded judgments about what that model is doing. Should this shell command run? Which of forty skills fits this request, if any? Is this recalled memory actually relevant? Does the agent need all of this 40 KB tool result, or none of it?

Every serious coding agent has that second layer. Ours is Octomind's supervisor. Anthropic's is the auto-mode classifier in Claude Code. And in both cases, a general-purpose model makes those judgments with its defining capability — writing a reasoned explanation — switched off for exactly these calls. A frontier round-trip, resent per decision, to get back one word.

Jev is built for those decisions and nothing else. Three primitives:

  • Noul — a yes/no question, answered as the probability that the answer is yes.
  • Choice — one option from a set you define, with a probability for every option and a confidence that says how peaked the distribution is.
  • Score — a position on an ordered rubric of levels described in words, which may land between levels.

It ingests the state once and evaluates every question against it in a single parallel pass, so the tenth question is nearly free. It's trained for calibration, not preference, so a 90% answer should be right about 90% of the time across many predictions. And the answer space is closed by construction: it can pick the wrong option, but it can't invent one, so there's no JSON to repair and no enum to fuzzy-match.

We wrote the long version — what it is, where it breaks, what people built with it in its first 72 hours, and every control-plane decision inside an agent that it could take over, with live numbers — over on the Octomind blog: Jev Explained: TypeSafe's System One Model and the Decisions Inside Every AI Agent. This post is about the library.

The API

The module is evaluation, on by default, and it compiles alone if that's all you need:

octolib = { version = "0.39", default-features = false, features = ["evaluation"] }

The shape follows the rest of Octolib. One request type, one high-level helper, one provider:model string:

use octolib::{evaluate, Answer, EvaluationRequest, Question};

async fn triage() -> octolib::EvaluationResult<()> {
    // Requires TYPESAFE_API_KEY; use "cloudflare:typesafe/jev" to bill AI Gateway credits instead.
    let request = EvaluationRequest::new("Help! My payouts have been failing for 3 days.")
        .with_question("is_urgent", Question::noul("Does this convey urgency?"))
        .with_question(
            "department",
            Question::choice(
                "Which team should handle this?",
                [("billing", "Payments, refunds"), ("technical", "Bugs, outages")],
            ),
        )
        .with_question(
            "frustration",
            Question::score("How frustrated is the customer?", ["Calm", "Frustrated", "Very angry"]),
        );
    let response = evaluate("typesafe:jev-latest", request).await?;
    if let Answer::Noul { noul } = response.answers["is_urgent"] {
        println!("urgent with p={noul:.2}, cost {:?}", response.usage.cost);
    }
    Ok(())
}

The state can be a string, a JSON object or an array. Answers come back typed under the same ids as the questions — Answer::Noul, Answer::Choice with its probability map and confidence, Answer::Score with a legend — so you branch on numbers in Rust, not on parsed text.

Three routes to the same model:

Model string Reaches Jev through Keys
typesafe:jev-latest TypeSafe's API directly (early access) TYPESAFE_API_KEY
cloudflare:typesafe/jev Cloudflare AI, billed via AI Gateway credits CLOUDFLARE_API_KEY, CLOUDFLARE_ACCOUNT_ID
octohub:jev Your own OctoHub proxy, with hub-reported cost OCTOHUB_API_KEY, OCTOHUB_API_URL

A few details that only matter once you run it hard — which is the part we care about:

  • Retries are safe here. An evaluation has no side effects, so a replay can't duplicate work. The client retries on 429, 529 and 5xx with backoff, and honors Retry-After up to a 30-second cap. Generation POSTs elsewhere in Octolib are deliberately not replayed; this one is.
  • Cost is computed from the published rate, $0.042 per million input tokens, and lands in usage.cost like every other call. Through OctoHub the hub's reported cost wins.
  • The response's model field reports the versioned model that answered. jev-latest moves when TypeSafe ships a new version; if you tune thresholds, log it and pin.
  • Model ids are validated before the request leaves. openai:jev-latest is an unsupported provider; cloudflare:@cf/zai-org/glm-5.3 is a chat model, not an evaluation model. Both fail at construction, not at the wire.

Jev has a 32k context, and TypeSafe's own docs say accuracy falls as state fills with material a question doesn't need. Trim the state. Ask many questions; send little.

Where it plugs in: Octomind and OctoHub

We don't add modules to Octolib for fun. Every module exists because a product needed it, and this one was needed twice.

Octomind's supervisor already runs on the invariant that free signals gate the model and model calls are rare. On main today, [supervisor.evaluate] adds a calibrated third tier between "free" and "frontier" at three seams: a relevance filter after memory recall, a roster Choice when every skill-activation rule abstained, and a pre-screen that decides whether a tool batch needs to wake the authorizer model at all. Each seam is one switch, off by default. One attempt per call, a five-second timeout, no retries; any failure keeps the pre-change behavior for that turn. And one rule we'd put at the top of any Jev integration: the state never carries tool results or assistant messages, because the reviewer must never read the thing that might be arguing.

OctoHub gets POST /v1/evaluations. Configure an alias once, in your own proxy, with your own keys:

[evaluation_models]
"jev" = ["typesafe:jev-latest", "cloudflare:typesafe/jev"]

and every client in every language — a Python hook, a TypeScript worker, a shell script — calls octohub:jev and gets the same typed answers, the same per-request cost row, and the same attribution as its chat completions. That's why octohub is the third evaluation provider in Octolib: the library calls the proxy, the proxy calls Octolib.

Both land in the next Octomind and OctoHub releases; the library shipped first because the library always ships first.

Cloudflare Workers AI is now a media provider

Last round, Octolib's media stack covered image, video, speech and transcription across OpenRouter, Replicate, fal, ElevenLabs and Runway. 0.39.0 adds Cloudflare Workers AI for image, speech and transcription, through the same synchronous /ai/run endpoint the chat provider uses.

The registered models, with capabilities, options and reference pricing:

  • Image: FLUX.1 schnell, FLUX.2 dev, Leonardo Lucid Origin and Phoenix 1.0. Priced per output tile and step.
  • Speech: Deepgram Aura 1 and Aura 2 (English and Spanish), MeloTTS. Priced per character.
  • Transcription: Whisper, Whisper large-v3-turbo, Deepgram Nova 3. Priced per audio minute, with segments, words, languages and metadata parsed into the typed transcript.

The catalog loads lazily and caches pages, with a fallback to metadata for models Cloudflare hasn't listed yet. Where upstream reports neuron costs, they're preserved as reported rather than replaced with a duration estimate. Image seeds only go to models that accept them; speech types are preserved instead of coerced.

The chat side of the Cloudflare provider got the same treatment: it picks and normalizes reasoning levels per model, enforces schemas only for the OpenAI-shaped models that actually honor them, resolves overlapping model ids by longest match, and falls back to reference capabilities for unknown models instead of failing.

Everything else since 0.36.1

Ten releases in twelve days. The condensed version.

Two new LLM providers. Inception Labs serves the Mercury diffusion LLM family: inception:mercury-2.5, with Mercury 2 alongside, JSON schema output and tool calling. Tinker, from Thinking Machines, serves its Inkling family plus open-weight models (Nemotron, GLM, Kimi, Qwen, GPT-OSS, DeepSeek) and sampler checkpoints, with colon-containing model ids handled and short names like tinker:inkling resolved to serverless inference ids. Both come with examples that take the model spec from the command line.

ONNX embeddings, locally. A new onnx embedding provider loads HuggingFace graphs and tokenizers into ONNX Runtime — onnx:<org>/<repo>, with an optional #path-to.onnx for a specific graph — preferring quantized graphs, honoring pooling metadata and static dimensions, batching with token usage tracked. It sits next to FastEmbed and the Candle-based HuggingFace backend as the third way to embed with no API key. This is the one breaking change of the round: EmbeddingProviderType gained an Onnx variant, so exhaustive matches need a new arm.

Structured output validates once. The old flow could retry a failed schema validation and aggregate usage across attempts. That was well-meaning and wrong: it hid provider behavior behind a loop. Now there's exactly one upstream schema call, one centralized extraction and validation step, and a parsing or validation error comes back with the offending response logged and no retry. Client tool calls survive final-output validation. Two providers stopped over-promising in the same change: Ollama and Cloudflare no longer claim guaranteed schema enforcement they can't deliver, and OpenAI fails closed when schema guidance is ignored instead of returning something that merely looks right.

Reasoning effort tiers map per provider. Alibaba and Z.ai GLM models are mapped to the effort tiers they support; Fireworks preserves the maximum tier for GLM 5.2. Z.ai now accepts preserved-thinking messages and keeps thinking-only assistant turns instead of dropping them.

The roster moved. Cerebras retired models are replaced with qwen-3.8-27b, with video support and a 128K input limit. DeepSeek's retired V4 routes are replaced by deepseek-flash, treated as multimodal, with V4.1 Flash pricing corrected and added on Alibaba. Hetzner's withdrawn models are swapped for Qwen. BytePlus cache and third-party pricing refreshed. OctoHub now probes embedding dimensions at construction and caches them per endpoint and model, which made OctoHub provider construction async — the second small API change worth knowing about. The provider mapper validates model aliases, preserves quantized ones, and rejects unmapped models at construction rather than at the first request.

Small but real. fal audio formats map to values fal accepts (MP3 and PCM; unsupported formats fall through to the model default). Empty message content is allowed again. rustls is at 0.23.45, dirs at 7.0.

Upgrading

  • cargo add [email protected] or bump the version. Every feature is on by default; the new evaluation feature pulls in nothing heavy.
  • If you match on EmbeddingProviderType, add the Onnx arm.
  • If you construct the OctoHub embedding provider directly, it's now async.
  • If you relied on structured output retrying a failed validation, it doesn't anymore. The error carries what the provider returned; decide in your own code.
  • Set TYPESAFE_API_KEY, or CLOUDFLARE_API_KEY plus CLOUDFLARE_ACCOUNT_ID, and try the example above. It runs a few hundred input tokens: under two thousandths of a cent.

FAQ

What is an evaluation model? A model that answers typed questions about a state with calibrated probabilities instead of generating text. TypeSafe's Jev, the first one, returns a yes/no probability (Noul), one option from a set (Choice), or a position on a rubric (Score), in 70 to 500 ms. It can't write code or prose. Its job is the decisions around your LLM: routing, admission, relevance filtering, skill selection.

Do I need a TypeSafe account to use it? No. Direct access is waitlisted, but cloudflare:typesafe/jev works today with a Cloudflare account and AI Gateway credits loaded, and octohub:jev works through your own OctoHub with either key behind it.

Does Octolib 0.39.0 break anything? One enum variant (EmbeddingProviderType::Onnx), one constructor turned async (OctoHub embeddings), and structured output no longer retries validation failures. Chat, media, reranking and every model string you already use are unchanged.

Is Jev already inside Octomind? The evaluation gates are on Octomind's main behind [supervisor.evaluate], off by default, and ship in the next release. This Octolib release is what they are built on.

Which products run on Octolib? Octomind (every chat call, embedding and evaluation gate), OctoHub (chat, embeddings, media and the new /v1/evaluations route), Octocode (chat, embeddings, reranking) and Octobrain (embeddings, reranking). When a provider changes its API or its pricing, the fix lives in one place and every product gets it.

The point

Octolib's job has always been to make the next model a one-line change for everything we ship. Most of the time that means a new provider or a corrected price. Once in a while it means a new kind of model, and the library has to decide how it fits.

Evaluation models fit as a fourth module because they are a different function: not "write me an answer" but "here is a state, here are my questions, tell me what you believe and how strongly." Agents have needed that primitive since the first pre-tool hook. We've been faking it with a single token from a reasoning model. Now it's a typed call, priced honestly, one string away.

github.com/muvon/octolib · Apache-2.0 · crates.io · the Jev deep dive on the Octomind blog.