Voice booking agent
A visitor talks to a page in Spanish and hangs up with a confirmed 30-minute appointment on the business calendar.
Built as a proof of concept for a client: ElevenLabs Agents handles voice, turn-taking and the model; Cal.com owns the calendar; and in between sits a Fastify backend whose entire job is to make sure the model never has to guess. The backend is the only component that talks to Cal.com, so the API key never reaches the agent and no booking payload is written by an LLM. The agent itself is versioned JSON applied with the ElevenLabs CLI — prompt, ASR, TTS, turn-taking and tool contracts live in the repository, not in a dashboard where the next deploy would overwrite them.
How it works
- BrowserElevenLabs SDK over WebRTC
- AgentSTT, turn-taking, LLM, TTS
- Webhook toolsHTTPS + bearer, into the backend
- BackendFastify: dates, options, idempotency
- CalendarCal.com API v2 → Google Calendar
The rules it is built around
- No date arithmeticToday’s date in the business timezone is injected per conversation and declared the single source of truth.
- No invented timesThe tool returns at most three options already phrased for speech; booking passes an option id, never a timestamp.
- Guarded twiceAn explicit read-back and a “sí” before the only irreversible action, plus an idempotency key per conversation.
- Read while spelledA client tool draws name and email into the page, so a misheard address is caught by eye, not by ear.
- Speakable failuresTool errors return 200 with a reason and a sentence to read aloud, so the model never improvises mid-call.
What is interesting inside
Every guessable decision moved out of the model
Timezone arithmetic lives in one file behind tests with a frozen clock. Cal.com returns dozens of raw slots; the backend picks three spread across the day and phrases them for speech, so the agent reads options rather than inventing them.
The page shows the booking, not the chat
The interface is built on the ElevenLabs SDK rather than the embedded widget: an orb that tracks the microphone and then the agent’s own output, a subtitle track, and a panel that fills in with the real times Cal.com returned, polled from the backend by conversation id.
Failure modes designed before the happy path
If the requested part of the day is full the rest of the day is offered; if the whole day is full the next seven are searched in parallel, because seven chained round-trips are audible silence. Tool errors come back as something the agent can say out loud.
Prompt iteration without burning voice minutes
A simulation harness runs six scripted conversations against the live agent in text and asserts that booking always followed an explicit confirmation, that no invented time was offered, and that confirming twice does not create two appointments. 153 tests, none of which touch the network.
Shipped like a product, not a demo
Bearer auth on both webhook tools, HMAC-verified post-call webhook with a replay window, constant-time comparisons, redacted logs, per-IP rate limiting, a multi-stage non-root Docker image, and boot-time validation that kills the process naming the missing variable instead of failing mid-call.
Built with
WHY IT EXISTSThe interesting problem in a voice agent is not the prompt, it is everything the prompt should not be doing. Dates, option phrasing, idempotency and error wording moved into a backend I can test; what is left for the model is conversation, which is the part it is actually good at.
When you need judgment,
not just code.
More than a decade shipping product across web, mobile and AI left me something more valuable than a stack: judgment. If your team is stuck on a technical decision, evaluating a stack, or wants a second opinion before sinking months into a direction, let's talk.
