Isak Labuscagne — Blog

The $5 Chip on My Desk Doesn't Know Anything — And That's the Design

Sun, 27 Sep 2026 00:00:00 +0000

I wanted a thing on my desk that answers questions. Not a phone app, not a browser tab — a small object with a microphone that I can ask something without touching anything.

The obvious candidate is an ESP32. A few dollars, WiFi built in, runs Linux-adjacent firmware, and there's a whole ecosystem of open-source voice assistants for it. Also — and this is the part I got wrong — I assumed the chip would be doing the thinking.

It cannot. Not even slightly. Understanding why is what this post is actually about, because the fix turned out to be a design principle I already believe in.

The chip got very good at running a language model

First, the part that genuinely surprised me. The state of on-device LLM work on microcontrollers moved a long way recently, and the number that made me sit up was this: a 28.9-million-parameter language model running on an ESP32-S3, generating text at 9.88 tokens per second, entirely on the chip, with nothing sent to a server. On an $8 board.

For context, the previous record for a chip this size was 260,000 parameters. That's roughly a hundredfold jump in about three years.

The trick is memory tiering. An ESP32-S3 gives you 512KB of fast SRAM, up to 8MB of PSRAM, and 16MB of flash. None of that is enough to hold a real model, so the trick is deciding what doesn't need to be fast:

Tier	What lives there	Speed	Access pattern
SRAM (512KB)	Activations, norm weights — the actual thinking	Fastest	Touched every single token
PSRAM (8MB, octal)	Output head + KV cache	60.7 MB/s measured	Read once per position
Flash (16MB)	A 25M-parameter lookup table	Slow, and it doesn't matter	~6 random 512B rows per token

The insight is borrowed from Google's Gemma models: most of a language model's parameters live in an embedding table that you read from rather than compute on. So you leave that table in flash and pull only the few rows each token needs — about 450 bytes. The expensive thing is never loaded; it just sits there being sampled a little at a time.

The numbers that make it work are worth knowing, because they tell you where the ceiling actually is. That flash table costs 20.3 microseconds per row lookup, six lookups per token, so about 0.12ms — roughly 0.7% of the per-token memory traffic. Free. The real bottleneck is entirely different: the output head, a matrix-vector multiply over 1.5MB of weights. At 60.7 MB/s the theoretical floor is 24.7ms, and the achieved number is 57.6ms.

Which is why the optimisation ladder looks like this:

Implementation	Speed	What changed
First portable port	0.57 tok/s	Scalar FP32, head resident in flash
Staged head in PSRAM	4.77 tok/s	Stop bit-shifting 4-bit nibbles per token
Dual-core head	6.22 tok/s	Moved the head to core 1
Int8 staged head	9.72 tok/s	Integer dot products, no float at all

A 17x speedup, and every step of it was about data placement, not about adding compute power. That's the whole story of embedded inference.

And it still cannot answer a single question

Here is the caveat that the press release skipped, and it's the reason this post exists. That model was trained on TinyStories. It writes short, simple, fictional stories. It will not answer questions, it will not follow instructions, it will not write your Python, and it has zero factual knowledge about the real world.

That's not a bug in the implementation. That's what 28.9 million parameters is. There is no arrangement of memory layout that turns a model trained to tell bedtime stories into something that knows what the weather is doing in Cape Town.

For completeness, the other end of the spectrum is just as interesting. There's a 118,784-parameter model that runs on a $2 board with zero PSRAM, hits 18.24 tok/s, keeps a 24.5KB KV cache in internal SRAM, and serves a chat UI over a WiFi hotspot. Faster than the 28.9M one, and knows even less. It has 128 tokens of vocabulary.

So: the honest summary of "AI in microcontrollers" in 2026 is that we got astonishingly good at generating believable text on a chip, and that has almost nothing to do with answering questions. Those are different problems wearing the same coat.

So the chip is the interface, not the brain

The build I ended up with is a split, and once you accept the split it stops being a compromise and starts being the obvious design:

Stays on the ESP32-S3	Goes to a public AI model
Wake word detection	Everything requiring knowledge
Microphone capture + noise suppression	Speech to text
Speaker playback + TTS streaming	The actual answer
The personality (LED states, voice, timing)	Long-tail reasoning
Local commands, sensors, presence
The offline fallback

The left column is a list of things a microcontroller is genuinely good at: streaming audio over I2S, holding state, driving a GPIO, and staying awake at 5mA while listening. None of that needs a transformer.

Wake word detection is the clearest example of a job that belongs on the chip. Espressif's own wake word engine runs fully offline in under 200ms, and because it's local it works with the WiFi down, on a flat battery, in a room with no signal. That last property is the entire reason it stays on the chip — a wake word that needs a network round trip is a button.

The voice stack is ordinary microcontroller work: an I2S MEMS microphone in, an I2S amplifier out, 16kHz mono, Opus-compressed over a WebSocket. The open-source xiaozhi-esp32 project (MIT, ~25k stars) has already built most of this, including device-side MCP so a model can turn "turn the light down" into a GPIO write without anyone writing voice-parsing code.

Where the time actually goes is worth a table, because it tells you which part to optimise:

Stage	Time
Microphone capture	10ms
Local preprocessing	20ms
Wake word (on-chip)	80ms
Speech to text (remote)	300ms
Model inference (remote)	800ms
Text to speech (remote)	400ms
Speaker playback	50ms
Total	~1.66s

The chip contributes 110ms. The network contributes 1.5 seconds. Optimising the microcontroller is not the lever here, and pretending otherwise is how people end up spending a weekend on firmware when they should have been shopping for a different model.

The public models are a rental, not a foundation

"Public AI models" is doing a lot of work in the phrase, so let me be concrete about what it actually is. Here's the state of the free tiers as I found them:

Provider	Free allowance	The catch
Groq	30 req/min, 1,000 req/day, 200K tokens/day	Open-weight only; the token cap binds first
OpenRouter	20 req/min, 50 req/day	Roster churns constantly; 1,000/day after a one-off $10
Cloudflare Workers AI	10,000 neurons/day	Budget is in neurons, not tokens
Google Gemini	Flash tier free	Vendor states free-tier content is used to improve their products
Mistral	$10/month of credits	Training on by default in Free mode
Cohere	1,000 calls/month	Non-commercial only

Two arithmetic details that matter more than the table. First, Groq's 1,000 requests a day sounds generous until you notice the 200,000 token ceiling lands at an average of 200 tokens per request — fine for a desk toy, brutal for anything carrying context. Second, every one of these numbers moves, and not always slowly: GitHub Models was retired entirely in July 2026, Cerebras replaced its free tier with a card-required trial, and Llama quietly left Groq's free tier in August.

That's the lesson, and it's the same one as the push URLs in the monitoring post. A free tier is a rental agreement, not a foundation. You can absolutely live on it — you just have to know it's someone else's building, and that they can demolish it on a Tuesday. So: two providers configured, fallback on a 429, and a cache in front of both.

The part that actually matters

Here's what I got wrong at the start. I designed the buddy as a thin client: hear a question, send it, play the answer. Which means when the network is down, the object on my desk is a brick that costs me money to keep alive.

And that's the exact moment you ask it what's going on.

So the same rule from the monitoring post applies, and I should have seen it coming: the witness must not live on the thing being watched. The chip cannot be a dependent of the API. Concretely, four tiers of degraded behaviour:

Cached answers — a local store of things I actually ask it often. A question normalised to a hash, answered once, replayed forever. This is the cheapest rate-limit relief there is, and it doubles as the offline mode.
Local intent — volume, lights, timers, "what time is it", "stop talking". Never leave the network for a command that didn't need it.
Local presence — the chip knows when someone's in the room, what the temperature is, whether it's dark. That context is on the device because it's on the device, and it doesn't need permission to leave.
An honest offline state — when the network is gone, it says so, in one sentence, and then still answers the things it can. A device that lies about why it can't help is worse than one that goes quiet.

The LED is the tell. Green pulse means the model answered live. Amber means cached. Steady blue means offline and it knows it. You can read the state of the whole system off one diode on your desk, which is more than most of my actual infrastructure manages.

The key is on the device

Worth stating plainly, because it's the part people skip: that API key is sitting in flash on a $5 device on your desk, and the device will eventually be lost, stolen, reflashed, or handed to someone. Provision per-device keys, not one global key. Scope them. Rotate them. Assume the one on the desk is already public, because a hardware token in an esptool image is not a secret — it's a string in a firmware binary anyone can pull over the air.

What it cost

Money: about $35 total, one-off. The board was $5, the microphone and amplifier a few dollars, the enclosure more than the electronics.
Tokens: free tiers, comfortably, because a person asks a desk toy maybe twenty questions a day and the cache eats most of them.
Power: ~5mA idle listening, ~150mA awake. It's a USB-powered object, not a battery problem, and that's a decision I made on day one by not using one.
Time: far more than the electronics. The firmware was the cheap part.

Lessons

Running a language model and answering questions are different problems. 28.9 million parameters generate lovely nonsense. Check what a model was trained on before you build a product on top of it.
On tiny hardware, data placement beats compute. 0.57 to 9.72 tokens/second came entirely from memory layout and integer maths, not from a faster chip.
Put the latency where the latency belongs. 110ms of the 1.66s was the microcontroller. Optimise the chip when it's on the critical path, and not before.
Keep the interface local and the intelligence remote. Wake word, sensors, personality and the offline path stay on the silicon because that's what silicon is for.
The offline path isn't a nice-to-have — it's the use case. The moment your remote brain is unavailable is the moment someone asks your device what the problem is.
A free tier is a rental. Keep a second provider, cache aggressively, and assume any published limit is a suggestion with a countdown.
Hardware tokens leak. Per-device keys, scoped, rotated, and treated as public from the day you flash them.

My desk buddy is a $5 chip that knows nothing. What it actually is, is a very good sensor with excellent manners and a good fallback — it notices I'm here, it hears me, and it either knows the answer or tells me honestly that it can't.

The chip is the part you can trust. The model is the part you're borrowing.

If You Can't Monitor It, You Don't Own It

Tue, 22 Sep 2026 00:00:00 +0000

Server access logs tell me who visited this site. Nothing told me whether they could.

Two weeks ago the disk on this box filled to 100% and nginx started returning 502s for the site itself. I knew because I kept re-curling it between fixes. That's not monitoring — that's a habit. If I'd gone to lunch, nobody would have known until I came back.

The homelab page has had "monitoring dashboard" sitting on the backlog for months. This is that — minus the dashboard vanity. The part that actually matters is the part that wakes you up.

The trap: a monitor that shares a failure domain

The obvious move was one command right next to the site itself:

docker run -d --restart unless-stopped -p 127.0.0.1:3001:3001 \
  -v uptime-kuma:/app/data louislam/uptime-kuma:1

Same box, same Docker, one compose pull away. And completely worthless for the failures I actually fear. OOM killer, swap death, a bad kernel update, an IAM rule that locks me out of the instance — every one of those takes the monitor down with the thing it's watching. A green screenshot isn't monitoring. Monitoring is a witness that survives the crime.

Failures come in two classes, and they need different vantage points:

Service dead, box alive — the 502 case. nginx is running, the disk is full, certbot is failing quietly, and the box hums along looking fine to anything on it.
Box dead — everything stops, including any monitor unlucky enough to be on it.

One box can't reliably witness either of its own failures. So the rule became: the thing that watches must not live on the thing being watched.

The layout

Check	Runs on	What it catches
Health-gate heartbeat	cron on the EC2	Box death, cron death, broken outbound network
HTTP checks (site, n8n endpoint)	Uptime Kuma on the separate node	nginx/TLS/DNS/502s — observed from outside
Alert routing	n8n on that same node	Turns DOWN into a Telegram message that waits its turn
Outer ring: one free external monitor	SaaS, watching the status page	Death of the node that runs Kuma and n8n

The separate node is the same one from the n8n post — the whole reason n8n doesn't sit beside the site itself. Monitoring inherits that reasoning: different failure domain or it doesn't count.

The health gate: the only thing that runs on the box

The heartbeat isn't a ping. A ping says "the kernel still answers." I want "the things that rot quietly are still fine" — so it's a 20-line bash script that checks nginx, the cert expiry, the disk and the last successful deploy, then pushes one overall status to Kuma's push API:

#!/usr/bin/env bash
set -u
BASE="https://status.network-ai.co.za/api/push/${KUMA_TOKEN}"
fail=()

systemctl is-active --quiet nginx || fail+=("nginx-down")

# a 502 on the site itself is the failure that actually matters
curl -fsS -o /dev/null --max-time 10 https://network-ai.co.za/ \
  || fail+=("site-not-200")

cert=/etc/letsencrypt/live/network-ai.co.za/cert.pem
expiry=$(openssl x509 -enddate -noout -in "$cert" | cut -d= -f2)
days=$(( ( $(date -d "$expiry" +%s) - $(date +%s) ) / 86400 ))
(( days < 14 )) && fail+=("cert-expires-in-${days}d")

(( $(df -P / | awk 'NR==2{print $5}' | tr -d %) > 90 )) && fail+=("disk-over-90")

if (( ${#fail[@] == 0 )); then
  curl -fsSG --max-time 10 "$BASE" --data-urlencode "status=up" --data-urlencode "msg=ok"
else
  msg=$(IFS='|'; echo "${fail[*]}")
  curl -fsSG --max-time 10 "$BASE" --data-urlencode "status=down" --data-urlencode "msg=$msg"
fi

And the dead-man's-switch part is free: if the script dies, if cron dies, if the box dies — the push stops arriving, and Kuma marks the monitor DOWN anyway. The failure mode fails safe. That's the whole trick. A heartbeat you have to poll for is a chore; a heartbeat that expires on its own is a witness.

One cron line:

* * * * * /usr/local/bin/health-gate.sh >/dev/null 2>&1

The --max-time 10 matters more than it looks. A hung curl without a timeout stacks a new run every minute until the box is out of processes — exactly the kind of self-inflicted outage this script exists to catch.

Kuma on the other node

Uptime Kuma itself lives on the separate node, behind nginx with TLS, bound to localhost like everything else:

services:
  uptime-kuma:
    image: louislam/uptime-kuma:1
    container_name: uptime-kuma
    volumes:
      - uptime-kuma:/app/data
    restart: unless-stopped
    ports:
      - "127.0.0.1:3001:3001"
volumes:
  uptime-kuma:

Three monitors, two failure domains:

Push monitor — the health gate's heartbeat, 60-second grace.
HTTPS: network-ai.co.za — catches nginx death, expired TLS, DNS pointed somewhere stupid, and the 502 class from outside, which is the exact blind spot from the day the disk filled.
HTTPS: the n8n webhook endpoint — because an alerting pipeline nobody checks is the most optimistic kind of monitoring.

Kuma's built-in status page goes on a subdomain — status.network-ai.co.za — and deliberately lives off the broken box. When the site is down, the status page is the one thing still up telling people so. It replaces the "is it just me?" messages I used to get on WhatsApp, which is the MSP version of this same problem.

Turning DOWN into an actual page

Kuma's webhook feeds n8n — the same instance from the automation post. But a raw DOWN-to-Telegram pipeline trains you to ignore alerts the first time something flaps during a docker compose pull. So the workflow waits, re-checks, and only pages if the failure was real:

{
  "trigger": "webhook: kuma.monitor.down",
  "steps": [
    { "action": "wait", "seconds": 60 },
    { "action": "http_request", "url": "kuma /api/monitor/{id}", "expect": "status=down" },
    { "action": "if_still_down", "then": ["notify_telegram", "notify_email"] },
    { "action": "otherwise", "then": ["log_as_flap"] }
  ]
}

The DOWN message from the push monitor carries the health gate's reason string — cert-expires-in-12d or disk-over-90 — so the first message is already the diagnosis, not just "something is wrong, go look."

And per the rule from the n8n post: the whole thing ships with an Error Workflow. If n8n can't reach Telegram, it posts the failure to email instead. A silently broken alerter is worse than no alerter, because now you also trust something you can't trust.

Watching the watcher

There's one more ring out, and it's free. If the separate node dies, Kuma dies with it — and the witness to my EC2 just vanished without telling anyone. So one free-tier external monitor points at the status page's heartbeat URL and pages Telegram directly, bypassing n8n entirely.

The principle: every layer alerts through a channel that doesn't depend on the layer below it. Kuma routes through n8n because Kuma and n8n failing together is the case the outer ring exists for. Two hops to Telegram for routine alerts, zero hops for the catastrophe.

Day one

The first DOWN fired before the first real incident: a docker compose pull restarted Kuma for 40 seconds, the push timed out, and DOWN went into the queue. The 60-second re-check saw it back UP before Telegram ever fired — logged as a flap, nobody woken up. The grace step earned its keep before the feature ever had to be useful.

The motivating case would have been different. On the day the disk filled, the external check on network-ai.co.za would have caught the 502s in the first minute, with the reason attached, on my phone — while I was still re-curling it by hand.

The cert tripwire is the quiet one. certbot.timer runs every morning and renews at 30 days out — I checked. But a timer that runs isn't a renewal that succeeds, and cert expiry is the failure that looks exactly like an outage to a user: a browser scary page, no server logs to grep, nothing in error.log. Fourteen days before expiry, the health gate starts reporting DOWN with cert-expires-in-13d whether or not certbot knows it's broken.

What it cost

Money: $0. Uptime Kuma is free and self-hosted on a node I already pay for; the outer ring is a free tier doing exactly one job.
RAM on the tight box: ~0. The 908MB EC2 runs one cron line; the dashboard's ~100MB deliberately lives somewhere else — the box only has 908MB to give in the first place.
Maintenance: docker compose pull on the node, and rotating the push token if it ever leaks. Push URLs are bearer tokens — treat them like the credentials vaulted in the n8n post, not like a URL you paste into a README.

Lessons

A monitor that dies with the server is a screenshot. Put the witness in a different failure domain — a different box at minimum, a different provider if you can. One docker run on the box you're watching is motion, not monitoring.
Heartbeat beats ping. Bake the checks that rot quietly — containers, cert days, disk — into what you push. One token, whole-system truth, and a failed push fails safe to DOWN.
Grace before page. Wait, re-check, then wake a human. Flapping trains people to ignore alerts, and ignored alerts are slower than any outage.
The status page must survive the incident it describes. Host it off the broken box, then point one free external monitor at it. Every layer needs somebody watching from further out.
Cert expiry is a monitored service. A running timer is not a successful renewal. Put a day-count tripwire in the gate so expiry shows up as DOWN two weeks early instead of as a browser warning on launch day.
Alerts route through n8n with an Error Workflow. Same lesson as the automation post: the alerter itself can rot, and the one thing that must never fail silently is the thing that tells you things are failing.

The site hasn't had an outage since this went in, which is the wrong way round — monitoring earns its keep on the bad day, not the good ones. But the blind spot is closed: something that doesn't live on this box now notices, within a minute, when the box or its services stop being what they're supposed to be.

You don't own the stack because you can SSH into it. You own it when something that isn't on it notices it's broken before your users do — and knows exactly who to tell.

n8n in Managed Services: Automating the Grind

Sat, 12 Sep 2026 00:00:00 +0000

Managed services runs on the same two-hour blocks every day: check the ticket queue, chase the device that dropped off, answer the "is the internet down?" email, look at the same eleven warnings that have been warning for a month. None of it is hard. All of it is time.

So the actual skill in this job isn't the fixing — it's deciding what not to do by hand. I've been using n8n for that decision for a while now, and this is the story of what it actually looks like inside a real L2 queue.

Why n8n

I could have written Python scripts cron-tabbed to the same server. I did, for years. The difference n8n makes is that it keeps the visible process — trigger, condition, action — in one place the whole team can read, instead of a folder of scripts that only one person fully understands.

More importantly, n8n talks to everything out of the box. The ticket API, the monitoring webhook, Teams, email, and any REST endpoint I throw at it — with retries, error handling and credential vaulting built in rather than re-built in every script.

Job	What runs it
Password rotations	n8n schedule + device API
Disk-space triage	n8n + monitoring webhook
Ticket queue digests	n8n + ITSM API + Teams
Device status pings	n8n + SNMP / ICMP via REST

The password rotation nobody wanted to do

Start small. The first workflow I shipped was a rotating shared password that policy said had to change every 90 days. Three devices, one API, one boring task that was exactly the sort of thing to silently not happen for six months.

The workflow is almost insultingly simple:

{
  "trigger": "cron: 0 2 1 */3 *",
  "steps": [
    { "action": "generate_password", "length": 24 },
    { "action": "update_device_secret", "target": "device-a" },
    { "action": "notify_channel", "channel": "ops" }
  ]
}

Generate, push, announce. Around it I wrapped an n8n Error Workflow — if any step fails, a second workflow fires and posts the failure with the traceback instead of silently swallowing it. A bot that quietly fails is worse than no bot, because now you also trust the thing you can't trust.

Disk-space triage you can actually ignore

Every monitoring platform generates the same kind of noise: 8TB volumes at 88% "critical," checked twice a day, nobody actioning it because it's twice a day and it's always at 88%.

Instead of ignoring the alert (a bad habit), I made the workflow action it. When the monitoring webhook fires, n8n checks the trend: is this volume growing at a rate that matters in the next 30 days? If yes, it opens a real ticket with the trend attached. If no, it closes the alert with a note: "Reviewed — 65 days until projected full, review again on ."

That one workflow removed a surprisingly large chunk of the daily queue, because the queue was mostly things that were being looked at but never decided. n8n makes the decision; people only see the exceptions.

The ticket digest nobody asked for

The most-used workflow isn't even the clever one. Every morning at 06:30 it pulls the open-ticket list from the ITSM API, groups by age and priority, dedupes the "is the internet down" repeats, and posts a one-screen summary to the ops channel.

// Code node — collapse 40 tickets into 1 digest
return [{
  json: {
    summary: events.map(e => ({
      id: e.ticket_id,
      age: e.age_days,
      priority: e.priority,
      label: e.title.slice(0, 120)
    })).sort((a, b) => b.age - a.age).slice(0, 25)
  }
}];

The side effect I didn't expect: people argue about the summary format in the morning stand-up. That's not a bug. Paraphrasing the queue every day made everyone aware of what a 90-day-old "open but stale" ticket actually looks like — and once you see five of them in one screen, closing them becomes a team decision, not a chore for whoever draws the short straw.

What I learned

Automate the boring decisions first, not the impressive ones. A password rotation protects nothing glamorous, but it's the workflow that outlived every flashier project.
Every workflow ships with an Error Workflow. If it can silently rot, it will. Fail loudly into a channel people actually read.
Idempotency is the co-pilot. The disk workflow re-runs on the same alert must not create five tickets. Check-then-create, or dedupe by a stable key, before you ever call the create endpoint.
The queue is mostly decisions, not work. Half the tickets in managed services are already-resolved situations waiting for someone brave enough to close them. Automating the decision is worth more than automating the action.
Lock down the credentials. n8n's built-in credential storage means tokens and passwords live in the vault, never in the workflow JSON. Exported workflows without secrets are also a great way to share one with a colleague.

The cost

Money: $0. n8n runs self-hosted in Docker on the same box family I already run — though I keep it on a separate node from this site's own stack so the two don't fight over RAM.
Maintenance: it's one more stack to keep updated. Worth it.
Trust: the hardest line-item. It took a while before "the bot already handled it" was accepted as an answer in an incident call.

Automation in managed services isn't about replacing engineers. It's about making sure the engineers who stay are doing the work that needs judgment, and the machines handle the work that needs repetition. n8n just made that boundary visible — which was the first step to fixing it.

If you're in an L2 queue and your day still starts with manually typing the same four search terms into the ticket tool — that's your second-hour project. The first hour is telling n8n to do it for you.