How an AI phone agent actually works

Start with one ordinary call

Someone calls a contractor and says, “There is water coming through my ceiling. Can someone come out?” A useful receptionist needs to hear the request, ask an appropriate question, and follow the business’s actual process. A convincing voice is only one part of that job.

The diagram below describes a speech-to-text pipeline. Not every voice system works this way. It is the approach used by the current public Hank demo configuration we inspected on September 2, 2026.

One spoken turn in public Hank

1. 1. Connect the call: TwilioThe phone connection carries the caller's audio to the voice system.
2. 2. Hear the words: DeepgramSpeech recognition turns the audio into text the response model can use.
3. 3. Choose the response: ClaudeThe model uses the words, conversation history, and instructions to generate a reply or request an allowed action.
4. 4. Speak the reply: ElevenLabsThe voice service turns the response into audio. Eric is the voice selected for public Hank.
5. 5. Return the audio: TwilioThe caller hears the reply over the phone connection.

Vapi describes this separation as a transcriber, a response model, and a voice. The company names identify the providers selected for this configuration, not three interchangeable names for the same thing.

A voice is not the same as a brain

If an agent misunderstands an address, investigate the audio and speech recognition. If it hears the address correctly but gives an inappropriate answer, investigate the model’s instructions, context, and tools. If the answer is right but sounds awkward, investigate speech generation and timing.

Those are starting points for diagnosis, not proof of which vendor caused a failure. Noise, interruptions, configuration, and several services can contribute to the same symptom.

Changing Eric’s voice would not, by itself, teach Hank about consulting. That information belongs in the instructions and business context. Likewise, rewriting a sales pitch will not necessarily correct a word that was transcribed incorrectly.

What the conversation platform adds

A phone conversation is not a sequence of neatly completed messages. People pause halfway through an address, correct themselves, and say “right” while the other person is speaking.

The platform helps coordinate when to respond, when to stop, and how to recover. Vapi exposes timing and interruption settings. Waiting longer can protect a slow answer but make a quick exchange feel sluggish. Being too eager can cut a caller off. These settings need call testing, not just an attractive default.

ElevenLabs also sells an agent platform, separate from its voice service. ElevenLabs Agents combines speech recognition, a selected language model, speech generation, and turn-taking. Using ElevenLabs as a voice inside Vapi is different from using ElevenLabs to coordinate the whole conversation.

Saying it happened is not doing it

“I sent you the information” is a sentence. Sending the right information to the right person is a business action.

An agent may request a tool, such as checking a calendar or sending a text. The backend must validate that request, perform the action, and return a result. For a text, there can be several separate events:

1. Caller asks for informationIdentify what they want and the intended recipient.
2. Software validates and submits the messageA successful request does not yet establish delivery.
3. The carrier reports a statusRecord delivered or failed when that information becomes available.

See Twilio’s message-status documentation for the provider’s definitions.

What we observed, and what it does not prove

In a September 2 owner test of public Hank, the stored transcript showed a conversation about consulting followed by a request for information. A call-linked message record later showed provider-reported delivery.

That supports a narrow conclusion: this tested conversation connected to a real follow-up action. It does not prove perfect voice quality, uninterrupted service, sales conversion, or that every possible caller will get the same result. The review used the transcript and system records, not a separate listening assessment of the recording.

What to ask before buying

• Can I test the actual phone experience rather than just hear a selected clip?
• What does the agent do when it does not understand a name or address?
• Which actions can it take, and where can I verify their results?
• What happens when the calendar or messaging service is unavailable?
• Who investigates problems across the different providers?

You do not need to memorize every vendor. You need to know which part does which job, and who is responsible for making the whole experience work.

For a concrete service-business example, see how we scope phone answering for a garage door company. It separates a requested appointment, a confirmed booking, and a transfer attempt so the caller and owner know what actually happened.

Next: how to choose between an all-in-one platform and separate tools.