The first time I timed it, I didn't believe the number. I held the hotkey, said a sentence, released, and started a stopwatch in my head against the moment the text landed at the cursor. Then I did the same thing against a cloud dictation tool. The cloud round-trip wasn't twice as slow. On a bad-coffee-shop-Wi-Fi morning it was an order of magnitude slower, and worse, it was unpredictable — sometimes 300 ms, sometimes two seconds, occasionally a spinner that resolved into nothing.
That variance is the whole story. Dictation is a real-time UI surface, and real-time UI surfaces die on tail latency, not average latency. So when we built Vext — our closed-source macOS voice-to-text app — the decision to run everything on-device wasn't an ideological one about privacy. It started as a latency decision. The privacy argument fell out of it for free.
This post is the math and the architecture behind that call. It's written for the engineer standing at the same fork: you have an audio or AI feature, you're staring at a cloud STT API with a friendly SDK, and a voice in your head is asking whether you should just run the model locally instead. I want to give you a way to answer that question with numbers instead of vibes.
One disclaimer up front. Vext itself is closed source. Muvon's developer tooling — Octomind, Octocode, and the rest of the agent stack — is open source under permissive licenses, and we write about its internals freely. Vext is the commercial app that pays for the open work, so what follows is about the approach and the tradeoffs, the kind any engineer could reason through, not a guided tour of our source tree.
Dictation Is a Latency-Hostile Workload
Most audio features are forgiving. Transcribe a podcast overnight? Nobody is watching the clock. Generate meeting minutes after the call ends? A few seconds of processing is invisible. These are batch workloads, and for batch workloads the cloud is often the right answer — you get someone else's GPUs, someone else's ops team, and the latency simply doesn't matter because no human is waiting on the critical path.
Dictation is the opposite. A human is staring at the cursor, mid-thought, waiting for words to appear. The interaction loop is: speak, release, expect text now. Every millisecond between "release" and "text appears" is dead air the user feels in their hands. And unlike a page load, there's no spinner that makes it acceptable — the mental model is "I'm typing with my voice," and typing doesn't buffer.
The brutal part is that this is a per-utterance cost. You don't pay it once. You pay it on every single dictation, fifty or a hundred times a day for a heavy user. A 700 ms penalty that you'd never notice on a page load becomes, across a workday, the difference between a tool that disappears into the flow and a tool you fight.
So the budget is tight, and the budget is the thing to reason about first.
The Round-Trip Math
Let's build the cloud latency budget honestly, from the parts you can't avoid.
A cloud STT request has to do, at minimum, the following before a single word comes back:
- Capture and encode the audio on the device.
- Establish or reuse a connection — TLS handshake if cold, or at least the cost of a kept-alive socket.
- Upload the audio bytes across the network.
- Queue on the provider's side until a worker picks it up.
- Run inference on their hardware.
- Serialize and download the result back to you.
Here is the part people underestimate: steps 2, 3, 4, and 6 are pure overhead that has nothing to do with how good the model is. They are tax. And they are the steps with the worst variance.
Network round-trip time alone is rarely your friend. Same-region, warm connection, good network, you might see RTT in the low tens of milliseconds. But "same region" is doing heavy lifting there. A user in Bangkok hitting a us-east endpoint is paying a couple hundred milliseconds of RTT before anything useful happens, on a single round trip — and a cold TLS handshake is multiple round trips. Add congestion, add a flaky hotel network, add the provider's queue depth at peak hours, and your p99 detaches from your p50 entirely.
Here's the conceptual budget. I'm deliberately not quoting hard numbers as if they were measured in your environment — they depend on your network and your provider — but the shape is what matters:
Cloud STT latency = encode
+ connection_setup # 0 if warm, 1-3 RTT if cold
+ upload_time # audio_bytes / uplink_bandwidth
+ provider_queue_wait # the variance killer
+ inference_time
+ download_time
+ deserialize
On-device STT latency = capture_to_buffer
+ inference_time # on local Neural Engine / GPU
Look at what the on-device path doesn't have. No connection setup. No upload. No queue. No download. The entire networking column collapses to zero. What's left is capture plus inference, and on Apple Silicon the inference is fast enough that the term that dominates a cloud budget — the network — simply isn't in the equation.
This is why the comparison isn't "cloud is a bit slower." For an interactive dictation workload it's structurally different. The cloud budget is dominated by terms with high variance that you don't control. The on-device budget is dominated by one term you do control, running on hardware you can measure.
And the model itself is no slouch. The speech model we default to — NVIDIA's Parakeet, from the NeMo family — runs far faster than real time on M-series chips, in the ballpark of two orders of magnitude faster than real time when it's mapped onto the Neural Engine. When transcription is that much faster than the speech that produced it, the inference term in your budget becomes small relative to the audio you already had to wait to be spoken. The bottleneck stops being the model and becomes physics: you can't transcribe a sentence before the person finishes saying it.
Privacy Was a Side Effect, Then It Became the Point
We chose on-device for latency. Then we noticed what we'd accidentally bought.
If the audio never leaves the machine, there is no audio to leak, subpoena, breach, retain, or train on. This isn't a privacy policy — a policy is a promise that a server could be doing something else. It's a privacy architecture: the data physically does not have anywhere else to go. There's no bucket of your voice recordings on someone's infrastructure because the recording was processed and discarded a few centimeters from the microphone.
For a dictation tool this matters more than it first appears, because of what people dictate. Watch your own usage for a day. You dictate passwords you're resetting. You dictate the architecture of systems under NDA. You dictate the contents of a message to a lawyer, a doctor, a co-founder mid-acquisition. The microphone on a developer's machine is pointed at some of the most sensitive raw material in their life, and the default posture of most voice tools is to stream all of it to a third party for processing.
There's a compliance dimension too, and it's not abstract. "Audio never leaves the device" is a sentence that closes a lot of security-review tickets. No data-processing agreement to negotiate, no sub-processor list to audit, no cross-border data-transfer question, no breach-notification surface for the voice data because there is no voice data on any server to breach. For a team trying to ship a tool into a regulated environment, "it runs locally" is often the difference between approved and indefinitely-under-review.
The honest version: we wouldn't have made privacy the headline if the latency math hadn't pointed the same direction. But it did, and once you've decided the audio stays home for performance reasons, refusing to ship a cloud fallback that quietly undoes that is the easy, correct call.
Two Models, One Pipeline, Both Local
Speech-to-text gets you words. Words are not what the user wants.
What they said was "uh so basically we need to like, move the the retry logic into the the worker, you know, the one that uh handles the webhooks." What they want pasted is "Move the retry logic into the worker that handles the webhooks." Stripping the filler, fixing the false starts, and shaping the result is a different job than transcription, and it's a language job, not a speech job. So the pipeline has two stages, and the interesting architecture decision was keeping both of them on-device.
Conceptually it's a hand-off:
audio ──▶ [ speech model ] ──▶ raw transcript
(Parakeet) "uh so basically we need to..."
│
▼
[ small local LLM ] ──▶ clean text
(Gemma 3 4B, MLX) "Move the retry logic..."
│
┌─────────────┼─────────────┐
▼ ▼ ▼
cleanup translation summary
Stage one is a specialized speech model: small, fast, single-purpose, optimized for one thing. Stage two is a small general-purpose LLM — we use Gemma 3 4B by default (Qwen 3, LLaMA 3.2, and Phi-3.5 are selectable), running through Apple's MLX framework on the GPU — doing the linguistic cleanup, and the same model, with a different instruction, handles live translation and meeting summarization. One model, three jobs, swapped by prompt rather than by shipping three different binaries.
The reason this two-model split is the right shape: each stage gets to be good at its job. You don't want your speech model trying to also rewrite prose, and you don't want a general LLM trying to do acoustic modeling. Specialized-then-general is a clean separation, and it keeps each model small enough to fit in memory and run fast. A giant end-to-end model that did both would be slower, hungrier, and harder to reason about — and you'd lose the ability to swap the cleanup model independently from the recognizer.
Keeping stage two local is the part people are tempted to skip. The speech model has to be local for latency; surely the cleanup can be a quick API call to a big cloud model? It can — and then you've thrown away the entire privacy property for the stage that handles the most legible version of the user's words. The raw transcript is messy; the cleaned-up text is the clear, quotable, fully-formed thought. That's exactly the thing you least want to ship to a server. If stage two goes to the cloud, stage one being local bought you nothing. So both stay home.
Where On-Device Starts to Hurt
I'd be lying if I sold this as free. On-device inference has real costs, and pretending otherwise is how you ship something that's elegant in a demo and miserable in the field. Here's where it bites.
Model size and the download. Local models are files, and good local models are big files. The first launch has to fetch them — a multi-gigabyte download for the speech model's compiled weights, plus a comparably-sized download for the LLM — and that's a first-run experience you have to design around, because "downloading 3 GB before you can say your first word" is a brutal onboarding if you handle it badly. There's no equivalent on the cloud side; their model is already warm on their hardware.
Memory pressure. Both models, loaded and resident, cost RAM. On a 64 GB machine, who cares. On a base-model laptop that's also running a browser with ninety tabs, an IDE, and three Electron apps, you are now competing for memory with everything the user actually opened the laptop to do. You have to be deliberate about when models are loaded, when they're released, and how big a model you're willing to default to.
You inherit the hardware floor. Cloud STT runs on the same datacenter GPUs for a flagship phone and a five-year-old laptop. On-device, your performance is the user's silicon. We require Apple Silicon precisely because the Neural Engine and unified memory are what make the latency math work; on older or weaker hardware the whole argument softens. You're making a bet on a hardware floor, and you're cutting off everyone below it.
Quantization is a knob with teeth. To make models fit and run fast you quantize them — represent the weights in fewer bits, trading a little accuracy for a lot of speed and memory. Done well it's nearly free. Pushed too far it's death by a thousand small wrongnesses: a slightly-off transcription here, a subtly-mangled cleanup there. Choosing the quantization level is choosing a point on the accuracy-vs-resource curve, and it's a real engineering decision, not a checkbox.
You ship the model, so you own updates. The cloud provider improves their model and every user gets it silently. Your local model improves when you ship an app update and the user downloads new weights. You've traded "someone else's continuous improvement" for "control and predictability," which is the right trade for a tool that has to behave identically offline on a plane — but it is a trade.
None of these sink the on-device approach for an interactive dictation tool. The latency and privacy wins are too large. But every one of them is a reason a different feature might correctly choose the cloud.
A Decision Framework You Can Actually Use
So here's the thing to take to your own audio feature. The question is never "is on-device better." It's "which column of costs does my workload care about." Map your feature against this:
| Signal | Lean on-device | Lean cloud |
|---|---|---|
| Interaction model | Real-time, human waiting on the result | Batch, processed in the background |
| Latency tolerance | Tail latency is felt; p99 matters | Seconds are fine; nobody is watching |
| Data sensitivity | Audio is private, regulated, or under NDA | Public or already-shared content |
| Offline requirement | Must work on a plane / bad network | Always-online is a safe assumption |
| Per-use frequency | Many times a day, per user | Occasional |
| Target hardware | You can require a modern hardware floor | Must run anywhere, including weak devices |
| Model freshness | Predictable, identical, offline behavior wins | Continuous silent improvement wins |
Read the table top to bottom. If your answers cluster in the left column — interactive, latency-sensitive, private, frequent, on capable hardware — you have a dictation-shaped problem, and on-device is very likely the correct architecture even though it's more work. If they cluster right — batch, latency-tolerant, non-sensitive, occasional, must-run-anywhere — the cloud is doing you a favor and going local is self-inflicted pain.
The mistake I see most often is treating this as a one-time global decision for a whole product. It isn't. It's per-feature. A tool can absolutely run interactive dictation on-device for latency and still hand a two-hour archived recording to a cloud batch job overnight, because those two workloads sit in opposite columns. We default everything local in Vext because our workloads happen to live on the left — but the framework, not the conclusion, is the part worth stealing.
What I'd Tell Myself a Year Ago
If I could send one note back to the version of me staring at the friendly cloud STT SDK: the friendly SDK is selling you the inference and hiding the tax. The model quality is rarely the problem. The network column — setup, upload, queue, download, and the variance that lives in all four — is the problem, and it's a problem you pay on every single utterance, forever, in the exact moment a human is waiting on your software with their hands.
On-device removes that column wholesale and hands you a privacy architecture as a side effect. It costs you a big download, some memory discipline, a hardware floor, and a quantization decision. For an interactive, sensitive, high-frequency workload, that's a trade I'd make every time — and did.
Run the math on your own feature before you reach for the SDK. The napkin is cheaper than the rewrite.
— Don
Vext is Muvon's closed-source, on-device voice-to-text app for Apple Silicon — speech recognition and the cleanup LLM both run locally, and your audio never leaves the Mac. Muvon's developer tooling, including Octomind, is open source. The architecture and tradeoffs described here are general engineering patterns, not a walkthrough of Vext's internals.
