Voice agent that books appointments: ElevenLabs, Cal.com and a backend that never lets the model guess

A voice agent that books appointments is a system where someone talks, a model understands what they are asking for, calls real tools, and hangs up with an appointment on a calendar. I built one in a lab repo: ElevenLabs Agents handles the voice—transcription, turn-taking, synthesis—Cal.com handles the calendar, and in between sits a Fastify backend whose only job is to make sure the model never has to guess. Not what day it is, not which time the chosen option corresponds to, not how that time is written in ISO with an offset. This post is the whole walkthrough: the architecture, the three tools, the prompt, and how I tested it without burning the 15 voice minutes the free plan gives you each month. The code is all on GitHub.

In this article:

• Fundamentals — What it solves · Why a backend and not a proxy
• Implementation — The agent lives in versioned JSON · The three tools · Pre-written options · The irreversible action · Speakable errors
• Operation — Testing without spending voice · The page shows the booking · What only a real call reveals

What a voice agent that books appointments actually solves

The task is concrete and narrow: someone opens a page, talks to an assistant in Spanish, and hangs up with a confirmed 30-minute appointment on the business calendar. It is not a chatbot that answers questions, and not a general assistant. It does one thing, and that is exactly what makes it buildable.

The system has four pieces and one direction:

Browser (ElevenLabs SDK)
        │ WebRTC
        ▼
ElevenLabs Agents        transcription · turn-taking · LLM · speech synthesis
        │ webhook tools (HTTPS + bearer)
        ▼
Backend (Fastify + TypeScript)
        │
        ▼
Cal.com API v2
        │
        ▼
Google Calendar

ElevenLabs never sees the Cal.com API key and never assembles a booking payload. The backend is the only component that talks to the calendar, and that boundary is what makes everything else reasonable.

The conversation follows a fixed order, written into the prompt as ten numbered steps: greet, pin down a concrete date, check availability, read the options, wait for a choice, ask for name and email separately, put the details on screen and spell the email back, wait for an explicit yes, book, say goodbye. Every step exists because the previous one can go wrong.

Why there is a backend and not a proxy

The first temptation is to point the agent’s tools straight at Cal.com. It works in the demo and breaks on the second call. The reason a backend sits in between is that every responsibility you take away from the model is an entire class of errors that stops existing.

Dates are the clearest case. The agent sends 2026-09-08 and afternoon. The backend resolves timezone, offset and format. No other part of the project converts dates, and that rule is written into the repo’s CLAUDE.md so it is still true six months from now.

One detail only shows up if you read the Cal.com API instead of assuming it: GET /v2/slots interprets start and end as UTC, but groups the response keys by the timeZone you requested. And each endpoint requires a different cal-api-version—2024-09-04 for slots, 2026-02-25 for bookings. Sending the wrong one does not produce a clear error.

The agent lives in versioned JSON, not in the dashboard

The agent is defined by JSON in the repo and applied with the ElevenLabs CLI. Nothing is configured by clicking in the dashboard, because anything changed there is overwritten by the next push. That is the difference between an agent you can review in a PR and one that only exists inside somebody’s account.

agent/
  agents.json                        CLI registry
  tools.json                         CLI registry
  agent_configs/
    appointment_scheduler.json       prompt, ASR, TTS, turn-taking, evaluation
  tool_configs/
    check_availability.json          webhook tool → POST /tools/availability
    book_appointment.json            webhook tool → POST /tools/book
    show_booking_summary.json        client tool  → runs in the browser

Applying a change is a sequence of four commands, because the CLI does not accept a single file: tools are standalone objects with their own ids and the agent references them by tool_ids.

# Store the shared secret and write the public URLs into the tool configs
pnpm agent:setup

# Create or update the tools and write their ids back into tools.json
cd agent && elevenlabs tools push

# Copy those ids into the agent's tool_ids
cd .. && pnpm agent:link

# Publish the agent
cd agent && elevenlabs agents push

The runtime configuration is where you decide whether the call feels natural. These are the values that matter and why they ended up this way:

The first_message is fixed rather than generated too, so every call starts identically and the first token is instant.

And there is one dynamic variable holding up everything else: {{current_datetime}} is injected when the conversation starts, computed on the server. The prompt declares it the single source of truth about today’s date. The visitor’s browser clock is irrelevant, because the agent has to reason about the business day, not the caller’s.

That name is a contract between two deploys: the page sends dynamicVariables: { current_datetime } and the prompt reads {{current_datetime}}. If you rename the variable, the agent push and the page deploy have to ship together, or it silently stops resolving mid-call.

Three tools: two webhooks and one that runs in the browser

The agent has exactly three tools. Two point at the backend and one runs inside the page.

Both webhook tools are authenticated with a bearer stored in the ElevenLabs Secrets Manager. One practical detail: the Secrets Manager substitutes the entire header value and documents no way to prepend a prefix, so the backend accepts both Bearer <token> and the bare token.

The third one is the interesting one. show_booking_summary has no URL: ElevenLabs forwards the call straight to the page, which draws the chosen time, the name and the email while the agent reads them back.

{
  "type": "client",
  "name": "show_booking_summary",
  "expects_response": false,
  "parameters": {
    "type": "object",
    "required": ["optionId", "name", "email"],
    "properties": {
      "optionId": { "type": "string", "description": "opt_1, opt_2 or opt_3" },
      "name": { "type": "string", "description": "Full name, exactly as dictated" },
      "email": { "type": "string", "description": "Email in lowercase, no spaces" }
    }
  }
}

It is a client tool for two reasons. The first is that it fixes the most frequent failure in this flow: names and emails dictated by voice are transcribed wrong often, and checking an address by ear alone is unreliable. Seeing it written while it is spelled to you does work. The second is that the personal data never leaves the browser it was dictated into: the public endpoint that feeds the page carries no name or email by design, so the page learns them from the agent, in that same browser, with no round trip to the server.

expects_response is false because the agent has nothing to wait for, and pausing mid-confirmation to hear back would only add latency.

If the general tool-calling mechanism behind this interests you, I covered it in the ReAct loop of a coding agent: the difference here is that the tools do not explore, they execute a closed contract.

Pre-written options: the model never writes a time

This is the central decision of the project. Cal.com returns dozens of raw slots for a day. The agent cannot read dozens of times out loud, and I do not want it choosing or formatting them either. So the backend does three things before answering:

Cal.com returns 12 slots
        │
        ▼
Filter: past, duplicate, outside the requested part of day
        │
        ▼
Spread 3 across the ordered list  ──►  indices 0, 5, 11
        │
        ▼
Phrase each one as spoken Spanish
        │
        ▼
{ id: "opt_2", spokenLabel: "el martes 8 de septiembre a las diez de la mañana" }

Spreading rather than taking the first three matters more than it looks: three consecutive slots at 9

// lib/slots.ts

// Spreads n picks as evenly as possible over an ordered list.
// With 12 slots and n=3 it returns indices 0, 5 and 11.
export function spreadIndices(length: number, n: number): number[] {
  if (length <= 0 || n <= 0) return [];
  if (length <= n) return Array.from({ length }, (_, i) => i);
  if (n === 1) return [0];

  const picked = new Set<number>();
  for (let i = 0; i < n; i += 1) {
    picked.add(Math.round((i * (length - 1)) / (n - 1)));
  }
  return [...picked].sort((a, b) => a - b);
}

// Each option comes out with an opaque id and a sentence ready to be spoken.
// The model receives "opt_2", never a timestamp.
export function selectOptions(slots: Date[], timeZone: string, max = 3): SlotOption[] {
  return spreadIndices(slots.length, max).map((index, position) => ({
    id: `opt_${position + 1}`,
    spokenLabel: spokenLabel(slots[index], timeZone),
    startsAt: toIsoWithOffset(slots[index], timeZone),
  }));
}

# lib/slots.py

# Spreads n picks as evenly as possible over an ordered list.
# With 12 slots and n=3 it returns indices 0, 5 and 11.
def spread_indices(length: int, n: int) -> list[int]:
    if length <= 0 or n <= 0:
        return []
    if length <= n:
        return list(range(length))
    if n == 1:
        return [0]

    picked = {round(i * (length - 1) / (n - 1)) for i in range(n)}
    return sorted(picked)

# Each option comes out with an opaque id and a sentence ready to be spoken.
# The model receives "opt_2", never a timestamp.
def select_options(slots: list[datetime], time_zone: str, max_options: int = 3) -> list[SlotOption]:
    return [
        SlotOption(
            id=f"opt_{position + 1}",
            spoken_label=spoken_label(slots[index], time_zone),
            starts_at=to_iso_with_offset(slots[index], time_zone),
        )
        for position, index in enumerate(spread_indices(len(slots), max_options))
    ]

<?php
// lib/slots.php

// Spreads n picks as evenly as possible over an ordered list.
// With 12 slots and n=3 it returns indices 0, 5 and 11.
function spread_indices(int $length, int $n): array {
    if ($length <= 0 || $n <= 0) return [];
    if ($length <= $n) return range(0, $length - 1);
    if ($n === 1) return [0];

    $picked = [];
    for ($i = 0; $i < $n; $i += 1) {
        $picked[(int) round($i * ($length - 1) / ($n - 1))] = true;
    }
    $indices = array_keys($picked);
    sort($indices);
    return $indices;
}

// Each option comes out with an opaque id and a sentence ready to be spoken.
// The model receives "opt_2", never a timestamp.
function select_options(array $slots, string $timeZone, int $max = 3): array {
    $options = [];
    foreach (spread_indices(count($slots), $max) as $position => $index) {
        $options[] = new SlotOption(
            id: "opt_" . ($position + 1),
            spokenLabel: spoken_label($slots[$index], $timeZone),
            startsAt: to_iso_with_offset($slots[$index], $timeZone),
        );
    }
    return $options;
}

What the agent receives is not a list to process, it is a sentence to say. Alongside the options comes a full spokenSummary—“Para mañana tengo a las nueve, a las diez y media o a las once y media de la mañana. ¿Cuál te sirve?”—and the prompt orders it to read that verbatim, no rephrasing, no added times, no reordering.

The backend also resolves the empty cases without handing the decision back to the model. If the requested part of day is full, it offers the rest of the day; changing the hour is less disruptive than changing the day. If the whole day is full, it searches the next seven in parallel: done sequentially that would be up to seven chained round trips to Cal.com, and that silence is audible on a call.

Booking is the only action that cannot undo itself

Everything else in this conversation can be repeated. Checking availability twice costs nothing, changing your mind is normal and the prompt explicitly says not to comment on it. Creating the appointment, on the other hand, writes to the business calendar and sends someone an email. That is why it is protected in three separate layers.

The first is the prompt. Step 7 calls show_booking_summary, spells the email character by character, reads the name and the chosen time, and asks literally “¿Está todo correcto?”. Step 8 only happens on a clear yes: silence, an “mmm” or a “creo que sí” are not confirmation and send it back to step 7.

The second is the tool configuration: interruption_mode: disable_during_tool. If the caller could interrupt mid-write, the agent would be left not knowing whether the appointment exists.

The third is in the code: one key per conversation makes two identical calls return the same booking instead of creating two.

// lib/scheduling.ts
export async function book(input: BookRequest): Promise<BookResponse> {
  // Idempotency: one conversation does not book twice even if the agent
  // calls the tool more than once.
  const existing = bookingStore.get(input.bookingKey);
  if (existing) {
    return { booked: true, duplicate: true, ...existing };
  }

  // The optionId is resolved against what check_availability stored for this
  // same conversation. The agent sends no date at all.
  const stored = optionStore.get(`${input.bookingKey}:${input.optionId}`);
  if (!stored) {
    return {
      booked: false,
      reason: "option_expired",
      // Spanish on purpose: the agent reads this to the caller.
      spokenConfirmation: "Ese horario ya no lo tengo a la mano. Déjame consultar la disponibilidad otra vez.",
    };
  }

  const booking = await cal.createBooking({
    start: new Date(stored.startsAtMs),
    attendeeName: input.name,
    attendeeEmail: input.email,
    timeZone,
  });

  const spokenConfirmation = `Listo, tu cita quedó agendada para ${stored.spokenLabel} a nombre de ${input.name}.`;
  bookingStore.set(input.bookingKey, { bookingUid: booking.uid, spokenConfirmation });

  return { booked: true, bookingUid: booking.uid, spokenConfirmation };
}

# lib/scheduling.py
def book(input: BookRequest) -> BookResponse:
    # Idempotency: one conversation does not book twice even if the agent
    # calls the tool more than once.
    existing = booking_store.get(input.booking_key)
    if existing:
        return BookResponse(booked=True, duplicate=True, **existing)

    # The option_id is resolved against what check_availability stored for this
    # same conversation. The agent sends no date at all.
    stored = option_store.get(f"{input.booking_key}:{input.option_id}")
    if not stored:
        return BookResponse(
            booked=False,
            reason="option_expired",
            # Spanish on purpose: the agent reads this to the caller.
            spoken_confirmation="Ese horario ya no lo tengo a la mano. Déjame consultar la disponibilidad otra vez.",
        )

    booking = cal.create_booking(
        start=datetime.fromtimestamp(stored.starts_at_ms / 1000, tz=timezone.utc),
        attendee_name=input.name,
        attendee_email=input.email,
        time_zone=time_zone,
    )

    spoken_confirmation = f"Listo, tu cita quedó agendada para {stored.spoken_label} a nombre de {input.name}."
    booking_store.set(input.booking_key, {"booking_uid": booking.uid, "spoken_confirmation": spoken_confirmation})

    return BookResponse(booked=True, booking_uid=booking.uid, spoken_confirmation=spoken_confirmation)

<?php
// lib/scheduling.php
function book(BookRequest $input): BookResponse {
    global $bookingStore, $optionStore, $cal, $timeZone;

    // Idempotency: one conversation does not book twice even if the agent
    // calls the tool more than once.
    $existing = $bookingStore->get($input->bookingKey);
    if ($existing !== null) {
        return new BookResponse(booked: true, duplicate: true, ...$existing);
    }

    // The optionId is resolved against what check_availability stored for this
    // same conversation. The agent sends no date at all.
    $stored = $optionStore->get("{$input->bookingKey}:{$input->optionId}");
    if ($stored === null) {
        return new BookResponse(
            booked: false,
            reason: "option_expired",
            // Spanish on purpose: the agent reads this to the caller.
            spokenConfirmation: "Ese horario ya no lo tengo a la mano. Déjame consultar la disponibilidad otra vez.",
        );
    }

    $booking = $cal->createBooking(
        start: (new DateTimeImmutable())->setTimestamp(intdiv($stored->startsAtMs, 1000)),
        attendeeName: $input->name,
        attendeeEmail: $input->email,
        timeZone: $timeZone,
    );

    $spokenConfirmation = "Listo, tu cita quedó agendada para {$stored->spokenLabel} a nombre de {$input->name}.";
    $bookingStore->set($input->bookingKey, ["bookingUid" => $booking->uid, "spokenConfirmation" => $spokenConfirmation]);

    return new BookResponse(booked: true, bookingUid: $booking->uid, spokenConfirmation: $spokenConfirmation);
}

The idempotency key did not have to be invented: in both tool schemas, bookingKey is wired to system__conversation_id, the variable ElevenLabs already provides. The model has nothing to remember.

There is one case Cal.com returns that is worth not flattening: when the event type requires the owner to confirm, the booking comes back as pending. Saying “quedó confirmada” there would be lying to the caller, so the sentence changes to “dejé solicitada tu cita… queda pendiente de confirmación”.

I wrote about this same theme before in the hard limits of an autonomous agent: the logic is identical, except the limit here is not an rm -rf, it is a write to somebody’s calendar.

Errors come back speakable, never as a 5xx

When a tool fails in a chat, the user sees a strange message and retries. When it fails on a voice call, there is a person waiting in silence and a model that is about to say something. That is why no error path in this backend returns a 5xx: they all return 200 with booked: false, a machine-readable reason, and a Spanish sentence the agent can read out loud.

// routes/tools.ts

// Errors go out as 200 with a sentence the agent can read. A 5xx would leave
// the model improvising in the middle of a call.
function handleToolError(error: unknown, tool: string, reply: FastifyReply) {
  if (error instanceof InvalidDateError) {
    // The date the agent sent is unusable: ask for it again.
    return reply.status(200).send(emptyAvailability("¿Me repites la fecha, por favor? No me quedó clara."));
  }

  const spoken = "Tuve un problema para consultar la agenda. ¿Intentamos de nuevo en un momento?";
  return reply.status(200).send(
    tool === "book_appointment"
      ? { booked: false, reason: "cal_error", spokenConfirmation: spoken }
      : emptyAvailability(spoken),
  );
}

# routes/tools.py

# Errors go out as 200 with a sentence the agent can read. A 5xx would leave
# the model improvising in the middle of a call.
def handle_tool_error(error: Exception, tool: str):
    if isinstance(error, InvalidDateError):
        # The date the agent sent is unusable: ask for it again.
        return jsonify(empty_availability("¿Me repites la fecha, por favor? No me quedó clara.")), 200

    spoken = "Tuve un problema para consultar la agenda. ¿Intentamos de nuevo en un momento?"
    if tool == "book_appointment":
        return jsonify({"booked": False, "reason": "cal_error", "spokenConfirmation": spoken}), 200
    return jsonify(empty_availability(spoken)), 200

<?php
// routes/tools.php

// Errors go out as 200 with a sentence the agent can read. A 5xx would leave
// the model improvising in the middle of a call.
function handle_tool_error(Throwable $error, string $tool): Response {
    if ($error instanceof InvalidDateError) {
        // The date the agent sent is unusable: ask for it again.
        return json_response(empty_availability("¿Me repites la fecha, por favor? No me quedó clara."), 200);
    }

    $spoken = "Tuve un problema para consultar la agenda. ¿Intentamos de nuevo en un momento?";
    if ($tool === "book_appointment") {
        return json_response(["booked" => false, "reason" => "cal_error", "spokenConfirmation" => $spoken], 200);
    }
    return json_response(empty_availability($spoken), 200);
}

The reason is for the log and the metrics; the spokenConfirmation is what the caller hears. And the prompt covers the other side: if booked is false, the appointment does not exist, and the agent still reads the sentence that already explains what happened instead of translating a technical error.

How to test the agent without spending voice minutes

The ElevenLabs free plan gives 15 voice minutes a month and you cannot buy more. A full conversation takes about two minutes, so the entire budget is six or seven calls. Iterating on the prompt by talking is not viable.

The way out is the simulation endpoint: scripted conversations that run against the real agent, in text, without consuming minutes.

pnpm simulate                 # all six scenarios
pnpm simulate happy-path      # just one

What matters is that the harness does not print transcripts for you to read. It asserts things: that book_appointment was always preceded by an explicit confirmation, that no time was mentioned which the tool had not returned, and that confirming twice does not produce two different bookingUid values. It exits non-zero if anything fails.

Underneath there are 154 unit tests that never touch the network, with a frozen clock for the date logic. And above it, every real conversation is scored automatically against three criteria defined in the agent configuration: that the appointment was created, that confirmation preceded booking, and that no invented time was mentioned.

Whether the appointment was created is decided by the result of book_appointment, not by what the agent said. An agent can say “quedó agendada” and be wrong, and that is exactly the class of error a transcript-based log never catches.

The page shows the booking, not the transcript

The page uses the ElevenLabs SDK, not the embedded widget. The widget is a closed component that brings its own chat bubble, its own feedback panel and its own floating button; the SDK is transport only—WebRTC, microphone, callbacks—so the interface is entirely the project’s.

That allows something the widget does not: showing the booking instead of the chat. An orb that tracks the microphone while the agent listens and its own output while it speaks, a state line, a discreet subtitle track, and a panel that fills in as it goes: the three times just offered, the one that was picked, the name and email as they were understood, and the confirmation.

Those facts reach the browser through two different channels, and which one carries what is deliberate:

The key that ties the two halves together did not have to be invented either: the page reads conversation.getId(), which is the same conversation id the agent sends as bookingKey, which is what the backend already files every offered option under.

Polling every two seconds, not server-sent events. The watched state changes about four times in a two-minute call, and a poll survives a laptop closing its lid or a phone dropping to 3G with no reconnection logic to get wrong.

What only a real call reveals

Simulation covers the logic; text says nothing about how it sounds. These are the symptoms that only appear when you talk, and where to fix them:

With six or seven calls of budget, spend them on exactly that and nothing else.

Frequently asked questions

Why not let the model call Cal.com directly?

Because then the model has to build the payload, and that means computing dates, formatting ISO with an offset, and choosing which times to offer. Each of those three is a source of errors that go undetected until someone gets an appointment at the wrong hour. With the backend in between, the agent sends a simple date and an optionId, and everything else is code with tests.

Can you do the same with Google Calendar directly, without Cal.com?

Yes, but Cal.com solves the tedious part for free: real availability according to business rules, event duration, buffers, confirmation and cancellation emails. Going straight to Google Calendar means implementing that availability logic by hand. For an MVP it is not worth it.

How do you stop the agent from inventing a time?

With three layers. The prompt forbids it explicitly and tells the agent that without calling the tool it does not know what is free. The tool returns a sentence already written for it to read verbatim, so it has nothing to compose. And an evaluation criterion reviews every finished conversation to verify that every time mentioned came from a check_availability response.

What happens if the caller interrupts while the appointment is being created?

They cannot: book_appointment is configured with interruption_mode: disable_during_tool. It is the only tool with that setting, precisely because it is the only irreversible action. Availability lookups do allow interruption, because abandoning one halfway leaves nothing half-written.

Does this work for phone calls or only in the browser?

The backend is the same. What changes is the transport: in the browser it comes in over WebRTC from the page, and for telephony you would connect the agent to a number. Everything I described—the tools, the prompt, the idempotency, the speakable errors—does not depend on the channel. The page does: with no screen there is no show_booking_summary, and a dictated email would have to be verified by ear alone.

How much does something like this cost to run?

Voice minutes are the dominant cost, not the LLM. With temperature: 0, a fixed prompt and short answers, the model’s share is marginal next to audio synthesis and transcription. If volume grows, the lever that moves the bill most is shortening the conversation, not switching models.

Conclusion

The pattern that makes this agent work has nothing to do with voice: take out of the model every decision a program can make better. The model does not compute dates, does not choose times, does not phrase how they sound and does not build the booking payload. It listens, calls the right tool in the right order, and reads what the tool already wrote.

The full repository is at github.com/devrchancay/elevenlabs-cal-demo, with the agent configuration, the tests and the simulation harness.

If you are going to build something similar, the order that worked for me was this: first the backend with its tests and its single date file, then the tools with closed contracts and speakable errors, then the prompt as a numbered procedure, and only at the end the voice. Spending voice minutes to discover a timezone bug is the most expensive way to find one.