Hey Steph

The short version

Problem. I work in voice AI but only ever through other people's APIs, and Mac voice control was either too shallow, like Siri, or too brittle, like agents that simulate clicks.

Action. Built a native macOS assistant solo to learn it from the inside: on-device models for routing, speech and vision that I picked, fine-tuned and combined myself, controlling apps through their documented interfaces.

Result. Routing accuracy rose from 91% to 96%, command latency fell from 736ms to 288ms, and memory dropped from 14GB to 0.4GB. It is public, alpha on purpose, and free to run.

The impact

• 91% → 96% routing accuracy
• 288ms command latency
• $0 required to run

Why this exists

Voice AI is the space I actually work in, professionally, at both hiRobin and Convin. I wanted to understand it from the inside rather than only ever consuming it through someone else's API. Which local model to pick, how to actually train and fine-tune one, what a custom wake word takes to build properly, the kind of judgment you only get by doing the work yourself, not reading about it.

I'm honest about what this is and isn't. Siri already owns this space on the Mac, and it's not a small competitor. On-device OS-level automation is getting genuinely capable fast, narrowing the room for a third-party assistant further with every release. Hey Clicky is a funded, actively growing product doing something adjacent. This was never going to out-compete any of them, and it stayed Alpha on purpose, I haven't pushed it further because that was never really the goal. The actual goal was the model work itself: picking, training, and combining real voice and language models, and seeing what breaks when you do.

A concrete example: the voice pipeline. Kokoro has great voice quality and high latency. Pocket TTS is faster but weaker sounding. Rather than accept one tradeoff, I combined them, Kokoro's voice with Pocket TTS's underlying model for speed, to get a result closer to what either one could deliver alone. The routing model itself, Gemma 4B, I trained and fine-tuned using Kaggle for the compute, and "Hey Steph" itself needed its own custom wake-word training, not just a config flag.

The bet, within that honest context

Given the goal was learning, not market share, the actual design decision still mattered: don't try to control every app on the Mac. Siri handles single commands and can't chain anything; GUI-automation agents like Dottie promise to control "everything" by reading the screen and simulating clicks, which means they break every time an app changes its UI. Hey Steph's bet: control a defined set of things reliably, through each app's actual documented interface, never by guessing at pixels.

That single rule, native API over GUI automation, shows up everywhere below, including in a fair number of things deliberately not built.

Architecture

Split by what each language is actually good at. Swift/SwiftUI menu bar app handles audio capture, wake word, Accessibility and AppleScript automation, system controls, OAuth, and UI, everywhere the OS integration has to live. Local Python model servers handle routing (MLX/Gemma), TTS (Kokoro), and vision (UI-TARS), entirely on-device. Cloud LLMs and voice services are optional, bring-your-own-key upgrades, never a requirement, since the free tier has to be good on its own or the whole premise fails.

Every model, chosen the same way: investigate, verify, ship what holds up

Distillation took accuracy from 91% to 96% with zero regressions. The tool-retrieval prefilter cut average routing latency from 2.16s to 0.84s. Turn detection cut a short command's time-to-act from 736ms to 288ms.

What this project deliberately doesn't do

Never simulates a click when a real API exists, even when asked directly to build "control anything on screen like a mouse," declined outright as a direct reversal of the project's own core rule. Didn't fork Dottie, the closest open-source competitor, on the belief that "always kept" beats "broader but brittle." Kept the larger, smarter model tier off by default, since responsiveness was prioritized over a marginal quality edge for the default experience. Not on the Mac App Store, since always-on background listening and broad Accessibility permissions don't fit App Store sandboxing, the same reason comparable utilities like BoringNotch and DockDoor ship direct-download.

How it actually got built: not trusting the first plausible answer

The same MLX thread-affinity bug surfaced independently in three separate features, an MLX model's Metal command stream is tied to whichever thread first touched it. Fixed once, correctly reapplied twice more without re-debugging from scratch.

"Verify live" held as a standing rule, not a slogan: Sparkle's master branch was caught pointing at an 18-months-stale binary before it could ship silently; a pinned HTTP library's transport was caught quietly migrating and breaking error classification; Google's 7-day OAuth-token expiry was confirmed to apply to the project owner too, not just test users. None of these were assumed from memory, each was checked against live documentation the night it mattered.

"Never confidently fake a capability" was hardened twice independently, once when the model answered "close Slack" with the same tool as "open Slack" instead of declining, once when Apple's own Foundation Models framework reproduced the identical failure under a stricter constraint meant to prevent it. Later validated from the outside: both Alexa+ and Gemini for Home spent 2026 publicly getting hammered by reviewers for exactly this failure mode.

Where it stands

Public release on GitHub, a self-contained build verified from a clean download, not assumed to work because it was published. Real first-run setup downloads the routing model, the trained fine-tune, and the voice model directly from Hugging Face. 7 real OAuth integrations, each hardened against a real CSRF gap found in a security pass. ~85 voice tools, retrieval-prefiltered so the list keeps growing without slowing routing down.

Honestly still open: code signing and notarization is built and scripted, gated on a one-time credential step. "Works on any Mac" is verified by simulation, not yet tested on a second physical machine. Two OAuth scope and deadline loose ends are flagged, not yet closed. And the bigger, honest truth: this stays Alpha because the goal it was built for, understanding voice AI from the inside, was already met, not because it ran out of runway.

At a glance

What it is

Native macOS voice assistant, solo build

Why it exists

A hands-on way to actually learn voice AI from the inside, not a commercial bid

Status

Alpha, public, downloadable, in active use

Routing accuracy

91% to 96% after distillation

Command latency

736ms to 288ms after turn detection

Memory

14GB to 0.4GB after fixing a GPU cache leak

Cost to run

$0 required, cloud APIs are opt-in