“Today his body is an NPC in the zombie apocalypse of Project Zomboid. Tomorrow it’s an ESP32, Arduino, or Raspberry Pi controller with legs or wheels. The brain is already built — and it knows its job“
“Shoot!” — and a shotgun thunders behind my back
(The videos are in Ukrainian — but the gameplay speaks for itself.)
— Shoot! — I yell into the microphone, fighting off two zombies in front of me.
Behind my back, a double-barreled shotgun thunders. A calm voice in my headphones: “Holding the line!”
This is not a scripted NPC and not a multiplayer buddy. It’s the gpt-realtime-2.1 model, which hears my voice through the microphone, answers with a voice of its own — and controls a character in my game through a set of 34 tools. It follows me around, loots houses on request, takes the passenger seat in my car, and texts my in-game pager when I walk too far away to hear it.
Like any normal engineer, I occasionally need to let my brain rest — and games help with that. But this time the rest mutated into full-blown R&D: I built a mod for Project Zomboid that turns the game into a testing ground for a voice-driven AI agent with a body.
Here’s how it’s built, what rakes I stepped on — and what robotics has to do with it.
What the companion can do
Project Zomboid is a survival sandbox set in a zombie apocalypse: open world, loot, base building, hordes. Multiplayer with a friend is fun. But my friend no longer needs a second player:
- Hears and speaks in real time. I talk into the microphone — he answers with his voice and acts. No buttons, no chat, no “processing your request” pauses.
- Follows me with real pathfinding: runs when he falls behind, sneaks when I sneak.
- Covers me. Zombies close in — he draws the shotgun himself. Something starts chewing on my back — he shoots point-blank even without a line of sight.
- Loots on request. “Go into that house and bring me a first-aid kit” — route, search, return, hand-to-hand delivery. He understands Ukrainian item names: “grab the strawberry” just works.
- Rides with me. Takes a free passenger seat on his own, gets out right after me, and asks me to slow down at 80+ km/h. Exactly like real-life friends, hahah!
- Texts the pager. When I’m too far away (15+ tiles), his voice fades to silence like in a multiplayer voice chat, and his messages arrive as text on an in-game pager with a beep.
- Understands living speech, not commands. “Sic ’em”, “grab the boomstick”, “give me some space” — the prompt carries the persona of a Ukrainian survivor companion with a map of everyday phrases onto the tools. And if you say “your name is Max now” — the name above his head changes and persists in the save.
This fun experimental mod for Project Zomboid was built in a pair with Fable 5 — Anthropic’s new model — in just two evenings.
Why: a rehearsal for robotics
The honest motivation wasn’t the game. I wanted to get a feel for OpenAI’s realtime engine in the role it seems born for: the brain of an entity that has a body. An ear, a voice, a set of muscle-tools, state sensors — and no direct access to the world beyond them.
A game is the perfect proving ground for that. A “robot” in Project Zomboid costs zero hryvnias of hardware, you don’t mind feeding it to a dozen zeds, and the physics, the enemies, and the inventory are already written by the game’s developers. All the interaction mechanics are the same ones a real robot will have in the real simulation — that is, real life 🤭: sensors on the way in, tools on the way out, latency in between.
More on that at the end. First, the fun part: how it all works!
How it’s built: three processes and a mailbox
The most curious thing about the architecture: Project Zomboid’s Lua engine (Kahlua) is a solitary-confinement cell. Build 42 cut HTTP out of it, there’s no microphone access, and it can’t play a live audio stream. The only thing it is allowed to do is read and write files.
So the system lives in three processes:
- The mod inside the game (Lua) — the body: movement, combat, looting, the pager, and most importantly — the registry of 34 tools.
- The companion app (Node.js) — the bridge: reads the game’s file “mailbox” (
outbox.jsonl— statuses and events every second), writes commands back (inbox.lua), and runs an HTTP server for the voice page. - The voice page in the browser — the ear and the throat: a WebRTC channel directly to OpenAI, microphone and speakers. The companion only mints it an ephemeral key.
The page computes the spatial audio on its own: the game sends distance and coordinates every second, and the browser turns them into volume and stereo panning. Companion on the left — voice on the left. Walk fifteen tiles away — silence, and the pager takes over. A small thing, yet it does more for immersion than any graphics.
A separate detective story — Cyrillic. Kahlua’s lexer truncates every character to a single byte when loading code: “привіт” turned into ?@82VB. I had to build three workaround channels: visible texts go through the game’s translation files, voice lines travel in a separate UTF-8 file, and Ukrainian item names in commands are encoded as codepoint lists with a tiny decoder inside the sandbox. Now “grab the backpack” reaches the game intact.
34 tools — and not a single line of arbitrary code
The key architectural decision: the model has no access to the game’s code at all. All it can do is call tools from the registry: follow, guard_on, pickup, give, scout_ahead, enter_vehicle, retreat_to_me, rename — 34 in total. Each tool is a function described to the model with clear parameters and an honest execution result. A sandbox in its purest form.
Reflexes and decisions are split into layers — like in a living creature:
- Reflexes live in Lua — free and instant. A zombie at point-blank range or the first bite — the weapon comes out by itself, without a single call to the model. Following, kiting away from close zombies, the “aimed walk” past distant ones — all of that is game logic.
- Decisions belong to the model. Where to go, what to loot, when to retreat, what to say and how to react to “sic ’em” — that’s function calling in real time.
This isn’t just a pretty analogy — it’s the same architecture real robots are built on: low-level stabilization loops run on board, while the high-level decisions are made by the “brain” through a limited set of actuators.
And here came the most instructive rake of the experiment. The older gpt-realtime model simply couldn’t handle 34 tools — it got confused and dropped calls. Upgrading to gpt-realtime-2.1 with its 128K context solved everything without cutting functionality. The Realtime API has a catch too: tools must be baked in when the session is created — you can’t “top them up” later, the request is silently ignored, and the model starts acting out command execution with its voice while doing nothing at all. A one-man theater.
Rakes I stepped on while building the mod
The road from idea to a working companion is five architecture rewrites in two evenings. The highlights:
- The tamed zombie. The first version made the companion out of a real zombie: suppress the aggression, patch the looks to human. It worked — until I saved the game. After loading, the engine returned him to the world as a hostile zombie: all the runtime “taming” flags lived only in memory. A bro who wants to eat you after every reload makes for a so-so bro.
- The invisible man. The second attempt — an honest player character — ran into the Build 42 renderer: the engine draws a player only if it’s the local player. The name floats above his head, but there’s no body. Impossible to bypass from Lua — a Java-level bridge from a neighboring helper mod saved the day.
- Emergent pacifism. One day the model decided on its own to holster the weapon — “to keep the noise down”. That state got written into the save. On the next load, the unarmed companion was quietly eaten off-screen. After that, iron rules appeared in the prompt, a self-preservation instinct in the code, and death stopped being silent: “they got me… but I pulled through” — after which the AI friend simply respawned nearby.
- The football commentator. The first version of his “senses” commented on every single shot zombie with a separate line. Three layers of filters — combat episodes with hysteresis, per-event-type cooldowns, and the style rule “mundane things get one word” — turned the commentator into a companion of few words.
The experiment in numbers
- 34 tools — the model’s entire interface to the world;
- 2 evenings — about 6 hours — from idea to a voice companion, five architectures in a row;
- 3 processes — the game, the Node bridge, the browser with WebRTC;
- ~$0.3–0.7 per hour of active play with conversations (pricing: ~$0.02/min of my speech, ~$0.08/min of his replies);
- ~$10 — the entire billing: about six hours of playing with the companion plus all the tests during development;
- 0 lines of arbitrary code available to the model.
From zombies to ESP32
And now the main thought this whole thing was started for.
Look at this companion as a robot’s specification. The ear — a microphone. The voice — a speaker. The sensors — a status report every second: coordinates, distance, health, enemy count. The actuators — 34 tools: go, stay, take, give, get in the car. The brain — a realtime model that ties it all together with live conversation.
Now swap the Lua mod for firmware: instead of follow — “spin the wheels toward the target”, instead of pickup — “squeeze the claws”, instead of the game status — telemetry from sensors, and instead of “shoot” — well, let’s say, “water the tomatoes”. The brain, the protocol, the persona, the whole “reflexes on board — decisions in the model” logic — stay unchanged. The ESP32 and Arduino are already sitting on my desk. What’s missing is a 3D printer for the housings — it’s in the plans, and then these experiments will move from the zombie apocalypse onto real wheels or robo-legs.
Let me mark the boundary honestly: no robot is driving around my desk yet — that part is a vision. But now it’s a vision backed by a working prototype of the brain, not by fantasy. I’ve already written about how agents have changed development itself in the vibe coding article — it’s the same line: a model + a clear set of tools + a human who sets the task.
Honest about the limitations
- Immortality as a mechanic. The companion doesn’t die for good — he “pulls through”. A deliberate choice: he’s a friend, not a resource.
- The looting is “magical”: he takes items within a six-tile radius without an animation of walking up to the exact shelf. A compromise for reliability.
- Singleplayer only. Multiplayer is an entirely different architecture.
- I won’t publish the mod. It stands on two third-party alpha mods with Java patches, requires an OpenAI key in plaintext, and was conceived as a private experiment, not a product. This article is its release.
Summary
The cheapest way to learn to drive a robot is to give the model a body in a game. Six hours of building, six hours of playing, zero hardware, and ~$10 of OpenAI billing — and you’re holding a proven architecture: sensors on the way in, a tool sandbox on the way out, reflexes separated from decisions, voice as the interface. The game will end — the architecture will stay.
And if your business needs a companion like this — without the shotgun, but with bookkeeping and logistics: an agent that hears a task in plain human language and executes it through a clear set of tools inside your systems — have a look at the AI agents and chatbots service, or just drop me a line. My AI companion and I have already been through the proving grounds =)
