The $5 Chip on My Desk Doesn't Know Anything — And That's the Design

I wanted a thing on my desk that answers questions. Not a phone app, not a browser tab — a small object with a microphone that I can ask something without touching anything.

The obvious candidate is an ESP32. A few dollars, WiFi built in, runs Linux-adjacent firmware, and there's a whole ecosystem of open-source voice assistants for it. Also — and this is the part I got wrong — I assumed the chip would be doing the thinking.

It cannot. Not even slightly. Understanding why is what this post is actually about, because the fix turned out to be a design principle I already believe in.

The chip got very good at running a language model

First, the part that genuinely surprised me. The state of on-device LLM work on microcontrollers moved a long way recently, and the number that made me sit up was this: a 28.9-million-parameter language model running on an ESP32-S3, generating text at 9.88 tokens per second, entirely on the chip, with nothing sent to a server. On an $8 board.

For context, the previous record for a chip this size was 260,000 parameters. That's roughly a hundredfold jump in about three years.

The trick is memory tiering. An ESP32-S3 gives you 512KB of fast SRAM, up to 8MB of PSRAM, and 16MB of flash. None of that is enough to hold a real model, so the trick is deciding what doesn't need to be fast:

The insight is borrowed from Google's Gemma models: most of a language model's parameters live in an embedding table that you read from rather than compute on. So you leave that table in flash and pull only the few rows each token needs — about 450 bytes. The expensive thing is never loaded; it just sits there being sampled a little at a time.

The numbers that make it work are worth knowing, because they tell you where the ceiling actually is. That flash table costs 20.3 microseconds per row lookup, six lookups per token, so about 0.12ms — roughly 0.7% of the per-token memory traffic. Free. The real bottleneck is entirely different: the output head, a matrix-vector multiply over 1.5MB of weights. At 60.7 MB/s the theoretical floor is 24.7ms, and the achieved number is 57.6ms.

Which is why the optimisation ladder looks like this:

A 17x speedup, and every step of it was about data placement, not about adding compute power. That's the whole story of embedded inference.

And it still cannot answer a single question

Here is the caveat that the press release skipped, and it's the reason this post exists. That model was trained on TinyStories. It writes short, simple, fictional stories. It will not answer questions, it will not follow instructions, it will not write your Python, and it has zero factual knowledge about the real world.

That's not a bug in the implementation. That's what 28.9 million parameters is. There is no arrangement of memory layout that turns a model trained to tell bedtime stories into something that knows what the weather is doing in Cape Town.

For completeness, the other end of the spectrum is just as interesting. There's a 118,784-parameter model that runs on a $2 board with zero PSRAM, hits 18.24 tok/s, keeps a 24.5KB KV cache in internal SRAM, and serves a chat UI over a WiFi hotspot. Faster than the 28.9M one, and knows even less. It has 128 tokens of vocabulary.

So: the honest summary of "AI in microcontrollers" in 2026 is that we got astonishingly good at generating believable text on a chip, and that has almost nothing to do with answering questions. Those are different problems wearing the same coat.

So the chip is the interface, not the brain

The build I ended up with is a split, and once you accept the split it stops being a compromise and starts being the obvious design:

The left column is a list of things a microcontroller is genuinely good at: streaming audio over I2S, holding state, driving a GPIO, and staying awake at 5mA while listening. None of that needs a transformer.

Wake word detection is the clearest example of a job that belongs on the chip. Espressif's own wake word engine runs fully offline in under 200ms, and because it's local it works with the WiFi down, on a flat battery, in a room with no signal. That last property is the entire reason it stays on the chip — a wake word that needs a network round trip is a button.

The voice stack is ordinary microcontroller work: an I2S MEMS microphone in, an I2S amplifier out, 16kHz mono, Opus-compressed over a WebSocket. The open-source xiaozhi-esp32 project (MIT, ~25k stars) has already built most of this, including device-side MCP so a model can turn "turn the light down" into a GPIO write without anyone writing voice-parsing code.

Where the time actually goes is worth a table, because it tells you which part to optimise:

The chip contributes 110ms. The network contributes 1.5 seconds. Optimising the microcontroller is not the lever here, and pretending otherwise is how people end up spending a weekend on firmware when they should have been shopping for a different model.

The public models are a rental, not a foundation

"Public AI models" is doing a lot of work in the phrase, so let me be concrete about what it actually is. Here's the state of the free tiers as I found them:

Two arithmetic details that matter more than the table. First, Groq's 1,000 requests a day sounds generous until you notice the 200,000 token ceiling lands at an average of 200 tokens per request — fine for a desk toy, brutal for anything carrying context. Second, every one of these numbers moves, and not always slowly: GitHub Models was retired entirely in July 2026, Cerebras replaced its free tier with a card-required trial, and Llama quietly left Groq's free tier in August.

That's the lesson, and it's the same one as the push URLs in the monitoring post. A free tier is a rental agreement, not a foundation. You can absolutely live on it — you just have to know it's someone else's building, and that they can demolish it on a Tuesday. So: two providers configured, fallback on a 429, and a cache in front of both.

The part that actually matters

Here's what I got wrong at the start. I designed the buddy as a thin client: hear a question, send it, play the answer. Which means when the network is down, the object on my desk is a brick that costs me money to keep alive.

And that's the exact moment you ask it what's going on.

So the same rule from the monitoring post applies, and I should have seen it coming: the witness must not live on the thing being watched. The chip cannot be a dependent of the API. Concretely, four tiers of degraded behaviour:

1. Cached answers — a local store of things I actually ask it often. A question normalised to a hash, answered once, replayed forever. This is the cheapest rate-limit relief there is, and it doubles as the offline mode.
2. Local intent — volume, lights, timers, "what time is it", "stop talking". Never leave the network for a command that didn't need it.
3. Local presence — the chip knows when someone's in the room, what the temperature is, whether it's dark. That context is on the device because it's on the device, and it doesn't need permission to leave.
4. An honest offline state — when the network is gone, it says so, in one sentence, and then still answers the things it can. A device that lies about why it can't help is worse than one that goes quiet.

The LED is the tell. Green pulse means the model answered live. Amber means cached. Steady blue means offline and it knows it. You can read the state of the whole system off one diode on your desk, which is more than most of my actual infrastructure manages.

The key is on the device

Worth stating plainly, because it's the part people skip: that API key is sitting in flash on a $5 device on your desk, and the device will eventually be lost, stolen, reflashed, or handed to someone. Provision per-device keys, not one global key. Scope them. Rotate them. Assume the one on the desk is already public, because a hardware token in an esptool image is not a secret — it's a string in a firmware binary anyone can pull over the air.

What it cost

• Money: about $35 total, one-off. The board was $5, the microphone and amplifier a few dollars, the enclosure more than the electronics.
• Tokens: free tiers, comfortably, because a person asks a desk toy maybe twenty questions a day and the cache eats most of them.
• Power: ~5mA idle listening, ~150mA awake. It's a USB-powered object, not a battery problem, and that's a decision I made on day one by not using one.
• Time: far more than the electronics. The firmware was the cheap part.

Lessons

1. Running a language model and answering questions are different problems. 28.9 million parameters generate lovely nonsense. Check what a model was trained on before you build a product on top of it.
2. On tiny hardware, data placement beats compute. 0.57 to 9.72 tokens/second came entirely from memory layout and integer maths, not from a faster chip.
3. Put the latency where the latency belongs. 110ms of the 1.66s was the microcontroller. Optimise the chip when it's on the critical path, and not before.
4. Keep the interface local and the intelligence remote. Wake word, sensors, personality and the offline path stay on the silicon because that's what silicon is for.
5. The offline path isn't a nice-to-have — it's the use case. The moment your remote brain is unavailable is the moment someone asks your device what the problem is.
6. A free tier is a rental. Keep a second provider, cache aggressively, and assume any published limit is a suggestion with a countdown.
7. Hardware tokens leak. Per-device keys, scoped, rotated, and treated as public from the day you flash them.

My desk buddy is a $5 chip that knows nothing. What it actually is, is a very good sensor with excellent manners and a good fallback — it notices I'm here, it hears me, and it either knows the answer or tells me honestly that it can't.

The chip is the part you can trust. The model is the part you're borrowing.