OctoHub 0.8.0: The Proxy Grew a Unified Media API

A month ago the question that made us build OctoHub was: your agent made forty model calls — what did it send, what did it cost, and which upstream answered?

Here is the same question with the money moved. Your agent generated eleven images while iterating on a design, then a nine-second video, then read the summary aloud in a synthetic voice. Four providers were involved. Three of them bill in units that aren't tokens — GPU-seconds, video-seconds, characters. One of them didn't tell you what it cost at all. Now: which customer do you charge?

For chat completions we'd answered that. For everything else, the answer was still a spreadsheet.

OctoHub 0.8.0 closes the gap. Image generation, video generation, speech synthesis and transcription now run through the same proxy, the same client keys, the same allow-lists, the same request log and the same cost column as /v1/completions. Five providers — fal, ElevenLabs, Replicate, Runway and OpenRouter — behind one JSON envelope.

The division of labor that made it small

This landed on top of octolib 0.36.1, tagged the same afternoon, which is where the media stack actually lives: typed request structs for the four tasks, the provider adapters, the job lifecycle, the capability descriptors, and a reference rate table for pricing what providers won't price themselves. We covered that release in the early-September round-up.

The OctoHub side follows from one rule: anything about models belongs in octolib, anything about tenants belongs here. Routing grammar, adapters, job handles and pricing are octolib's. Keys, quotas, persistence, metrics and the wire API are OctoHub's.

That rule is why the config gained no new syntax. A media alias is a model alias:

[media_models]
"flux" = ["fal:fal-ai/flux/dev", "replicate:black-forest-labs/flux-1.1-pro"]
"veo"  = ["openrouter:google/veo-3.1"]
"tts"  = ["elevenlabs:eleven_flash_v2_5"]

[providers.fal]        # concurrency and rate windows, unchanged
concurrency = 8
requests_per_minute = 60

Same provider:model grammar, same list-means-load-balance semantics, same per-provider limiter. Media never feeds tokens_per_minute — media providers report no tokens — but the windows are keyed by provider name and shared with completions, so on a provider you use for both, a token budget your chat traffic already spent will turn a media request away.

Bad configuration fails at boot rather than on the first paid request: an unknown provider, a malformed provider:model, an empty mirror list, or an alias that collides with [models] all refuse to start.

Four tasks, one envelope

POST /v1/images/generations     generate | edit | inpaint | variation
POST /v1/videos                 text_to_video | image_to_video | reference_to_video | extend | edit
POST /v1/audio/speech
POST /v1/audio/transcriptions
GET  /v1/media/{id}             fetch or advance a job
POST /v1/media/{id}/cancel
GET  /v1/media/models           capabilities, parameters, reference price

They're client endpoints, authenticated exactly like completions — same bearer key, same per-key model allow-list, same X-Request-Id correlation:

curl -sX POST http://127.0.0.1:8080/v1/images/generations \
  -H "Authorization: Bearer <client-key>" \
  -d '{"model":"flux","prompt":"a red panda astronaut","count":2,"size":"1024x1024"}'

Every response — image, video, voice, transcript, finished or still running — is the same object:

{
  "id": "med_9f3c1e0b…", "object": "media", "task": "text_to_image",
  "status": "succeeded", "model": "fal-ai/flux/dev", "provider": "fal",
  "progress": 1.0,
  "artifacts": [ { "kind": "image", "media_type": "image/png",
                   "source": { "type": "url", "value": "https://…" },
                   "size_bytes": 812345, "expires_at": 1767225600 } ],
  "usage": { "cost": 0.08, "cost_source": "provider", "currency": "USD", … },
  "warnings": [], "safety": { "status": "passed", … }, "error": null
}

One shape means one code path for persistence, one row format for the log, and one thing for your client to parse. Transcription is the only task whose payload isn't an artifact, so it adds a result object with text, language, segments and words.

Two deliberate deviations from OpenAI's API, stated up front rather than discovered: everything is JSON, never multipart/form-data — binary inputs are {"type":"url"…} or {"type":"base64"…} objects — and image edit, inpaint and variation are a mode field rather than separate paths. /v1/images/generations borrows OpenAI's path and its model / prompt / size spellings, but not its wire format — image count is count, not n, and the reply is the envelope above rather than OpenAI's {"created", "data"}. An OpenAI SDK won't parse it; call it over plain HTTP or a thin wrapper.

A job that outlives the request that started it

This is the part that genuinely differs from proxying a chat completion, and it's where the design decisions are.

A completion is one call that returns or fails. A media job commits money upstream the moment the provider accepts it, and then may run for minutes. A video is not a slow request; it's a purchase followed by a wait. Everything else follows from taking that seriously:

The row is written before the wait, not after. As soon as a queue-based provider accepts — fal, Replicate, Runway, OpenRouter video — OctoHub persists the record and the credential-free JobHandle, and only then starts waiting. A restart, a timeout, a client that hangs up: none of them can orphan a job you already paid for. The handle is in your database and the job is resumable from it. (ElevenLabs and OpenRouter's synchronous endpoints have no queue to hand back a handle for — they do the whole job inside the submit call, so there is no window to be interrupted in and nothing to resume.)

202 is not a failure. Send wait: false and you get the id as soon as the provider accepts. Send wait: true and exceed server.upstream_timeout_secs and you get the same 202 with status: "queued" or "running". The remote work continues; the id is live; nothing was lost. Poll GET /v1/media/{id} when you're ready.

Polling is what advances a job — there is no background worker. A deliberate non-goal: a worker would mean a scheduler, leases, and a second failure mode for jobs nobody is waiting on. The consequence is stated in the docs rather than hidden: a job you never poll stays queued and its cost is never recorded. Reading a job that already finished is free and never re-bills — a terminal row is served from the database with no upstream call at all.

The provider permit covers submit only. OctoHub's per-provider concurrency limiter guards the submit call, then releases. A four-minute video does not pin one of your eight fal slots for four minutes. The per-tenant slot is held for the whole request, though, so a customer's media work drains the same budget as their completions.

Failover happens at submit, where it's safe. Turn on server.failover_on_error — off by default, same as for completions — and a provider fault at submit drops that candidate and hands the request to the next mirror in the alias. The fault also counts toward that provider's failure streak: set server.provider_error_cooldown_secs (0, off, by default) and three consecutive provider-side failures put it on cooldown, which sorts it behind healthy candidates rather than blocking it. Left at the defaults, the fault goes straight back to the caller. Either way, once a job is accepted there's nothing to fail over — it's been paid for.

Records are scoped to the key that created them. Another tenant's id returns 404, not 403 — you don't get to learn that an id exists.

The parameter problem, and the honest answer to it

Every media provider has a different idea of what a request looks like. fal wants num_inference_steps and guidance_scale; some endpoints call the prompt text; Runway sells credits and thinks in its own model names. A unified API has to decide what to do about that, and there are two bad answers: expose only the intersection (useless), or invent a translation layer that pretends everything is the same (lies, expensively).

OctoHub's answer is three parts:

A portable core with one spelling everywhere — prompt, count, seed, size, duration_secs, negative_prompt, output_format. size takes "1024x1024" or "16:9"; anything else is a 400. Portable means one name, not universal support: Runway has no equivalent for count, negative_prompt or output_format, and neither does OpenRouter's video endpoint, so under the default strict policy those come back as a 400 on those providers rather than being quietly ignored — which is the third part below.

An escape hatch that passes anything through verbatim, namespaced by provider:

"provider_options": {
  "fal": { "input": { "num_inference_steps": 28, "guidance_scale": 3.5 },
           "field_map": { "prompt": "text" } }
}

field_map remaps a portable name onto whatever the endpoint actually calls it — so portable prompt keeps working against an endpoint whose field is text. Send every namespace at once when an alias spans providers; only the winning candidate's namespace is forwarded and the rest are dropped, which is what makes a multi-provider alias usable at all.

A policy for what happens when a parameter can't be honored. unsupported_parameters: "error" (the default) fails before money is spent — correct for production. "warn_and_drop" drops it and returns a warning — useful when one alias fans out across providers with uneven support. You choose per request, because only you know which one you meant.

And GET /v1/media/models tells you which is which before you spend anything: each configured candidate's execution and parameter capability flags, limits, its adapter's own provider_options JSON Schema, and the reference price. Two rough edges in 0.8.0, since you'd find them anyway: discovery probes every candidate through the image adapter, so the tasks field reads ["text_to_image"] even for a video alias, and an elevenlabs candidate has no image adapter at all — it comes back with a price and a null descriptor. Many capability fields honestly read unknown — the adapters cannot know every endpoint's schema, and saying so is better than a confident wrong answer. That's precisely why the escape hatch exists.

Cost that refuses to guess

This is the feature the release is really about, and the one place we were most stubborn.

usage.cost is the number billed. usage.cost_source says where it came from:

`cost_source`	Meaning
`provider`	The upstream returned actual dollars. OpenRouter and Replicate do this.
`estimate`	Computed locally from octolib's reference rate table.
`unavailable`	Nothing could price it — `cost` is `null`.

null is not zero. A request nothing could price is recorded as unpriced, never as free. It shows up in octohub_media_cost_unknown_total and carries a cost_unavailable warning, instead of silently dragging your spend total down and making a dashboard look better than reality.

The estimates come from octolib's reference table, which is unit-aware because the providers are: ElevenLabs bills characters, Runway sells seconds of video converted from credits, fal falls back to GPU wall-clock because that's the only quantity its queue metrics report. Where a rate would be a guess, there is no rate — Replicate community models bill GPU-seconds against an unknown GPU class, so they resolve to nothing and stay unpriced rather than get stamped with a plausible number.

The known gap, said out loud: ElevenLabs transcription is unpriced. Scribe bills input-audio duration, which isn't reported back, and no reference rate covers it. Transcription elsewhere does get a number — fal falls through to its per-GPU-second catch-all, and Replicate and OpenRouter price from whatever dollar amount the upstream reports. When we can price Scribe, we will; until then it's unavailable, not 0.00.

On the aggregate side, GET /v1/admin/usage gains media_count and total_cost — which now sums completions, embeddings and media into the one number you'd actually put on an invoice. GET /v1/admin/media lists individual records with the same filters as the other two, newest first, in-flight jobs included with a non-terminal status and a null completed_at. A job is its record at an earlier status; there's no separate queue to inspect.

Prometheus, per task, model and provider — plus an api_key_id label on the request counter when metrics.per_key is on. The cost counters stay unlabelled by key; per-tenant spend comes from GET /v1/admin/usage, where it's exact rather than sampled:

octohub_media_requests_total{task,model,provider,status}
octohub_media_duration_seconds{task,model,provider}
octohub_media_cost_microusd_total{task,model,provider,source}
octohub_media_cost_unknown_total{task,model,provider}

Costs are counted in micro-USD because a counter of dollars for images that cost $0.003 is a rounding-error generator. There's deliberately no outstanding-jobs gauge: an accurate one would have to count non-terminal rows in the database, and an in-process counter would be wrong the moment a job is polled by a different replica or survives a restart.

What we said no to

The features that aren't here are as load-bearing as the ones that are:

OctoHub does not become a blob store. There's no object store, no CDN, no artifact lifecycle to run, and it never fetches an artifact on your behalf. A provider that answers with a URL is stored as a URL — the row holds a link and metadata, nothing more, and that's most of the traffic. A provider that answers with the payload itself is the exception you should size for: ElevenLabs speech always does, and fal, Replicate or OpenRouter sometimes do. Those bytes are base64'd into the response, and the same base64 is persisted in the record's result column — because that row is exactly what a later GET /v1/media/{id} replays without touching the upstream. An MP3 in the media table costs about 4/3 of its own size, so a heavy text-to-speech deployment should plan for a table that is part ledger, part media library.

Client-supplied file paths are rejected. octolib's MediaSource supports file, provider_file and object_storage; all three return 400 here. A path in a request to a server is a request to read the server's filesystem — an SSRF/LFI hole, not a feature. Inline base64 is size-checked against media.max_source_bytes (20 MiB default) before anything reaches a provider, and the payload never lands in the database: the stored request keeps the shape and the byte count, not the bytes.

Upstream credentials stay on the server. Provider keys come from the server's environment (FAL_API_KEY, ELEVENLABS_API_KEY, and so on) and are never accepted from a client. provider_options.<provider>.cost_estimate is rejected outright — pricing is resolved server-side, and a client doesn't get to tell you what it owes you.

Also not in 0.8.0: streaming TTS, and multipart/form-data. Both are additive when someone actually needs them — though streaming will need its own answer for cost first, because octolib's streaming speech path reports no usage at all.

Upgrading

The media table is created on startup alongside the others, on SQLite, MySQL and PostgreSQL — no migration step. Existing config keeps working: if you add no [media_models], the new endpoints simply have nothing to route and the rest of the proxy behaves exactly as it did in 0.7.

# Linux x86_64, static musl build
curl -fsSL https://github.com/Muvon/octohub/releases/download/0.8.0/octohub-0.8.0-x86_64-unknown-linux-musl.tar.gz | tar xz
./octohub

Binaries ship for six targets (Linux musl, macOS, Windows — x86_64 and ARM64), or cargo build --release from source.

The release is marked breaking for one reason: the Storage trait grew media methods. That matters only if you maintain your own storage backend against OctoHub's internals; if you run the binary, there is nothing to change. One smaller thing worth knowing: since 0.7.12, OctoHub forwards attribution headers end to end, and as of this cycle it identifies itself upstream as Octohub/<version> rather than the generic octolib default — so provider-side dashboards name the proxy that made the call.

The point

The reason to put a proxy in front of your models was never the routing. It was that one place every request passes through is the only place the whole truth exists. That argument doesn't get weaker when the requests start returning pixels and audio instead of tokens — it gets stronger, because media is where the per-request cost stops being a rounding error and starts being the bill.

As of 0.8.0 the answer to "which customer generated this video, on which provider, and what did it cost" is one row in your own database, next to the chat completions, in the same currency, with a column that admits it when nobody knows.

— Don

OctoHub is open source under Apache-2.0, developed by Muvon Un Limited. Get it on GitHub — issues and pull requests welcome. Full media documentation: doc/11-media.md.