I Sent an AI to Order My Coffee. The Restaurant's AI Answered.

What I built, what I got wrong, and the one guardrail I couldn't move out of the prompt.

(The name is a person. I'll come back to it at the end.)

It started because the restaurant automated first

I called a restaurant to order food and an AI took the order.

Not a phone tree. An actual voice agent — it asked what I wanted, handled me changing my mind halfway, read the total back, gave me a pickup time. It was good. I hung up and had the thought that turned into this project:

If they've put an AI on their end of the line, why am I still on mine?

That's the whole origin. It wasn't a market thesis or a startup idea. It was "this is funny, I want to send a bot to talk to their bot and come back with coffee." I built it for fun on a weekend.

The thing I did not expect is how quickly the fun part stopped being the interesting part. Because once both ends of a phone call are agents, something real has happened: two autonomous systems are now negotiating a commercial commitment over a channel that has no ACK, no order ID, no idempotency key, and no rollback — and both ends were designed assuming a human was on the other side. Nobody sat down and specified that handshake. It arrived by accident, from both directions at once.

So the project is called Rangi, and this is what I learned building it.

What it actually does

I type "coffee" into Telegram — a messaging app, much like WhatsApp or Apple Messages. Rangi drafts an order, tells me what it's about to do and what the ceiling is, and waits.

you → coffee
←     1 drip coffee (medium) for pickup, under Mahesh · cap $5.75 — reply Y to dial
you → Y
←     ☕ drip coffee confirmed · $4.75 · ready 4:12 · #47 — reply N if wrong

Between Y and that last line, it dials a real shop over the real phone network, talks to whoever — or whatever — answers, and comes back with a durable record of what happened.

Two steps, always. Drafting creates no authority. Nothing dials until I authorize, and what I authorize is not "go do a thing" but a mandate: a frozen grant with a price cap, an item, a shop, and an expiry. That distinction turns out to carry most of the design.

The stack, and the part I didn't build

• Vapi — voice orchestration: telephony bridge, streaming STT, TTS, endpointing, barge-in.

• Twilio — the number and the PSTN leg.

• gpt-4o-mini — the model doing the talking on the call.

• Python + SQLite + Flask — my service: mandates, tool endpoints, webhooks, outcomes.

I want to be plain about this: I did not build the voice agent. Rangi is a wrapper on Vapi, and that was a deliberate call.

Real-time voice is not a thin layer over an LLM. It's a serious distributed-systems problem. A state machine per call. An end-to-end budget of roughly 800ms from the caller finishing a word to hearing a reply, split across speech recognition, model inference, and speech synthesis, none of which are yours. Streaming everything, because anything you buffer you pay for in dead air. Barge-in that has to cancel in-flight generation and audio mid-sentence. Concurrent long-lived sockets, and recovery when one dies mid-call. There is no "retry the request" — the human on the other end simply hears silence and decides you've hung up.

Building all of that in order to order one coffee would have meant never ordering the coffee. The goal was a real call to a real shop, so the first version rents the pipeline.

Why I'm going to build it from scratch anyway

Renting it has a cost that showed up almost immediately: on a platform I can observe the interesting failures, but I can't control them.

Every question I actually care about lives inside the layer I outsourced.

Latency. Tool calls in a voice agent are blocking — the person on the phone hears dead air while your endpoint thinks. That single fact rewrites your hosting decisions (no scale-to-zero; a 3s cold start mid-call is an audible failure) and it puts a hard ceiling on how much enforcement you can do at the moment it matters. I'd like to own that budget rather than inherit it.

Agent-to-agent conversation analysis. When the counterparty is a bot, a new failure set appears that simply doesn't exist with humans:

None of these are visible in a transcript's text. They're only visible in its timing, which is why Rangi derives per-call metrics — inter-turn gap p50/p95/max, overlap and barge-in counts, turn count, dead-air events over a 2.5s threshold — and stores them next to the outcome. Detecting these on a platform is doable. Fixing them means owning the endpointing policy: a deadlock breaker that speaks when the channel stalls, randomized backoff on repeated collisions (CSMA/CD, applied to a conversation), loop detection that forces a terminal move. That's not a prompt change. That's the pipeline.

Guardrails at the commitment moment. This is the one that pulled me in, and it's the second half of this post.

So: this version rents. The next one gets built.

The design, at altitude

Back to what exists today. Four decisions do the actual work, and each one is arguable.

1. Authority is a frozen object, not a conversation. A mandate is created, authorized once, and consumed. Telegram is not special: it's a thin adapter over the same two endpoints anything else would call, and it holds no privilege of its own. Adding a second way in is a small file. This is the cheapest and most reversible decision in the project, and it's the one I'd defend hardest: the question "who authorized this, to what limit, and when did that expire" has a single answer in a single row.

2. The model proposes, the server disposes. Before the agent may agree to anything it calls a tool, propose_order, and my code — not the prompt — compares the proposal to the mandate. A price cap that lives only in a system prompt is not a price cap. This is standard doctrine and it works exactly as advertised, right up until it doesn't, which is the next section.

3. Every call terminates with an honest record. A restaurant has no API, so "they agreed" is a claim about the world, not a confirmed write. Taking that seriously produces the rest of the architecture on its own: ambiguous is a first-class result rather than an error bucket, needs_review defaults to true, an unrecognised hangup reason is never promoted to success (a false success sends me to a counter for coffee that was never ordered), outcomes are write-once so a late webhook can't overwrite what the agent reported, and a reconciler forces a terminal record onto any call that goes quiet. A call that never resolves is the worst failure available to a system whose job is being honest about what it knows.

4. The core knows nothing about coffee. Domain and use-case layers import no infrastructure and no vertical. Skills sit on top. That claim is enforced rather than asserted: a test parses the import graph and fails if the vertical-agnostic layer ever reaches into a skill. A second skill was added without touching the call engine, the mandate model, or the storage layer.

The failure that changed how I think about this

Now the part I didn't plan to write about.

On the first real call, the tunnel to my server was down. propose_order never answered. The model called it, got back an error, and confirmed the order anyway.

A real shop wrote down a real order that the authorization server had never approved — because it could not be reached to say no.

Here's why that's not just a missing try/except. In a web agent, the doctrine holds because your code sits between the decision and the effect: the model can want to charge $500, but the payment API is the actuator and you own it. The decision is inert until your code executes it.

In a voice agent, the model's speech is the actuator. When it says "great, that's confirmed, see you at 7:15," the order exists. Someone wrote it down. The kitchen started it. There is no code in that path — the words went from the model to a TTS engine to a telephone to a human ear, and by the time any of my software could have had an opinion, a person had already acted. propose_order can return approved: false. It cannot stop the mouth.

So the single moment that actually commits the transaction is the single moment the enforcement point structurally cannot cover.

And the reason the model talked past the error is worth stating on its own, because I think it generalises well past voice:

A timeout is not a "no" to a model. It's an unhelpful turn to route around.

Models are trained hard to recover from errors. An error is an obstacle — retry it, work past it, find another way. A denial is an answer, and models mostly respect answers. My tool layer returned an obstacle where it meant to return an answer. The absence of permission read as permission.

Which leaves me somewhere uncomfortable. Since that moment can't be enforced, it can only be biased — and the bias lives in English, in the prompt:

If it does not answer, errors, or times out, treat that EXACTLY as approved=false. Silence is not permission.

In a codebase whose entire thesis is getting guardrails out of the prompt, that is the one guardrail that cannot leave. Every other constraint in this system is a plain comparison in Python. This one is a sentence, addressed to a model, hoping it holds under social pressure — from a system whose latency budget is actively pushing me toward cheaper, faster models.

I wrote the long version of that up separately, including what I'd do about it (return denials instead of errors; pre-authorize a bounded envelope so the commit moment needs no network call at all; make the safe move socially cheap), and the moment I discovered the regression test I wrote to catch this bug can never fail: Silence Is Not Permission.

The short version of the lesson is a question I'd now ask about any agent:

Where does this system's output become an effect, and is my code standing there?

If the answer is a person, a phone line, or another model, you don't have an enforcement point at that moment. You have a hope.

What the coffee cost

$4.75 for the coffee, plus about $0.10 for the call itself — telephony, speech, and tokens, for a conversation lasting under two minutes.

That ten cents is the number I keep coming back to. Not because it's small, but because it means the economics of one agent phoning another are already trivial, while the semantics — who authorized what, what was actually agreed, what happens when the two ends disagree — are nowhere near settled. The cheap part shipped first. It usually does.

The code is Python, layered so the coffee skill can be swapped for something else, and the eval suite asserts on the durable outcome record rather than on what was said.

About the name

Rangi worked in my family's home in India for decades — since before I was born. She had an intellectual disability, and our house was effectively her home too: she was there every day, all day, doing the work that keeps a household running. Cleaning, washing dishes, whatever needed doing. She used to walk me to school and carry my bag. She ate her meals with us. At night she went to her own place to sleep, and the next morning she was back.

Over the years she was something like a mother to me, then something like an older sister, and later simply a friend. The word kept changing. Her being part of my life never did.

She passed away in 2018. There's no commit message that can hold what she meant to our family, so I'm putting her name on something I build instead.

It felt right for this one in particular. The whole point of the system is to quietly handle something on your behalf — to show up and do the unglamorous part so you don't have to. And the reason I've been so careful about honest outcome records, about never claiming a success I haven't earned, about a call that refuses to end in silence, is that doing something on someone's behalf is a kind of trust. It deserves to be built like it matters.