What "Sounds Human" Actually Means
When a caller hangs up and tells your customer "your agent sounds robotic", they're rarely complaining about voice quality. Modern TTS voices are excellent. The voice itself sounds fine. What sounds robotic is everything around the voice: the timing, the prosody, the lack of hesitation, the missing acknowledgements, the unnatural turn-taking.
Making a voice agent sound human is mostly NOT about picking a better voice. It's about a long list of small conversational details that, individually, you'd never notice, but cumulatively make the agent feel alive or feel dead.
This article covers the details that matter, ranked by impact.
The Robotic Stack You'll Build First
The default voice agent architecture, which sounds robotic, looks like this:
```
- Caller finishes speaking (silence detected)
- Wait fixed timeout
- Send full transcript to LLM
- Wait for full LLM response
- Send full text to TTS
- Play full audio response
- Listen for next caller turn
```
This produces a conversation that sounds like:
Caller: "Hi, I want to book an appointment for next Thursday."
[800ms silence]
Agent: "Sure, I can help with that. What time on Thursday works best for you?"
[user pauses to think]
Caller: "Maybe around..."
[agent doesn't react, just waits]
Caller: "...two o'clock?"
[800ms silence]
Agent: "Two o'clock on Thursday is available. Would you like me to book that?"
Every gap, every flat acknowledgement, every "perfect" response sentence with no hesitation, screams that this is not a person. The fix is not a better voice. The fix is rewriting how the agent participates in the conversation.
What Real Humans Do That Voice Agents Don't
Real receptionists do these things automatically. Voice agents have to be told to do them.
Backchanneling
Humans say "mm-hmm", "okay", "right" while the other person is talking. It signals attention. The absence of it makes the agent feel like it's not listening.
Implementation: while the user is still speaking and the LLM hasn't been called yet, intermittently emit a short backchannel sound ("mm-hmm" at low volume) over the audio stream. Use sparingly so it doesn't sound like a tic.
Hesitation Sounds
When a real person needs a moment to think, they say "uh", "let me see", "one sec". When an agent goes silent for 800ms before answering, it sounds dead. When it says "let me check" and THEN goes silent for 800ms, it sounds normal.
Implementation: when you fire any tool call you expect to take more than 300ms, immediately play a pre-recorded or pre-synthesized filler ("let me check that for you"). This is the single biggest unlock for sounding human during database lookups.
Acknowledgement Before Action
Humans repeat back the request before doing it. "Thursday at two, got it. Let me find that." Voice agents that go straight to "Thursday at two is available" feel cold. Adding the acknowledgement adds maybe 800ms of speech but makes the agent sound like it's listening.
Variable Sentence Length
Humans don't speak in 25-word complete sentences every turn. They say "Sure." They say "Got it." Then they say something longer. A voice agent that always emits 3-sentence responses sounds robotic regardless of voice quality.
Prompt your LLM explicitly: "Vary your sentence length. Sometimes one word. Sometimes a full sentence. Match the caller's energy."
Asymmetric Pauses
Humans pause longer before complex answers, shorter before easy ones. Voice agents have one fixed pre-speech delay. Vary it: if the response starts with "yes" or "sure", play it instantly. If the response is a multi-sentence explanation, give a brief "let me think" beat.
Picking the Right Voice
Voice selection matters less than people think, but there are still pitfalls.
Avoid the Default
Most TTS providers ship with a "default" voice that's been used in 10,000 demos. Callers have heard it. It pattern-matches as "AI" instantly. Pick a less popular voice from the provider's library.
Match the Voice to the Brand
A casual playful voice for a law firm. A clinical voice for a yoga studio. A warm friendly voice for a high-stakes medical consultation. These are mismatches. Test the voice with the brand and the use case.
Streaming Matters More Than Quality
ElevenLabs streaming and similar streaming TTS APIs let audio start playing while the LLM is still generating. A slightly lower-quality streaming voice will feel more human than a flagship quality voice that takes 600ms to render.
Test on a Real Phone
The voice that sounds beautiful in your headphones can sound thin and tinny on a 8kHz μ-law PSTN line. Always evaluate voices on the actual transport you'll deploy on, not your laptop speakers.
Prosody and SSML
Modern streaming TTS handles most prosody automatically. You don't need to hand-author SSML for every utterance. But there are specific places where small SSML hints make a big difference.
Use Pauses for Drama or Confirmation
```xml
Let me check.
```
The 500ms pause after "Let me check" makes the agent sound like it's actually checking. Without it, "Let me check yes that's available" sounds canned.
Emphasize Key Words in Confirmations
```xml
So that's
at
```
When confirming details back, emphasizing the actual variables (date, time, name, address) makes the agent sound like it's verifying carefully.
Slow Down for Numbers
Phone numbers, addresses, confirmation codes. Default speech rate makes them blur together. Slow down explicitly:
```xml
Your confirmation number is
```
Don't Over-SSML
Heavy SSML markup tends to make voices sound more synthetic, not less. Use it only for the specific cases where it helps. Trust the streaming TTS for everything else.
Interruption Handling Is Most of the Battle
If you do nothing else from this article, get interruption handling right. Talking over callers is the single most jarring "this is a bot" tell.
Detect Speech While Talking
Most teams only listen for speech during their "user turn". You have to listen continuously, including while the agent is speaking. As soon as the user speaks for more than ~150ms, treat it as a barge-in.
Stop Audio Immediately
When barge-in is detected:
- Stop sending audio bytes to the telephony provider
- Send a clear-buffer command if your provider supports it (Twilio's clear command, etc.)
- Cancel any in-flight LLM call
- Cancel any in-flight TTS request
The clear-buffer step matters. Without it, the agent stops generating new audio but the audio already in the telephony jitter buffer keeps playing for 200-400ms after the user started talking. That's the worst feeling: the agent "stopped" but is still talking over you.
Don't Restart Mid-Sentence
After interruption, don't pick up where you left off. Listen, respond fresh. The user interrupted because they had something to say. Honor it.
Handle Spurious Interruptions
If the user coughs, sneezes, or has background noise, you don't want to bail on a sentence mid-stream. Use a learned interruption classifier or at minimum require 200-300ms of confirmed speech (not just any audio energy) before treating it as an interruption.
Endpoint Detection That Doesn't Cut People Off
The flip side of interruption handling: when has the user actually finished their turn?
The default 500-800ms silence threshold from STT providers like Deepgram is too conservative for natural conversation. The agent waits forever before responding. But too aggressive and it cuts callers off mid-sentence.
The fix:
Look at Punctuation
If the transcript ends in a question mark or period, the user is probably done. Drop the silence threshold.
Check for Filler Endings
If the transcript ends in "um", "uh", "let me think", "and...", the user is NOT done. Wait longer.
Use a Learned End-of-Turn Model
Some STT providers ship learned models that beat the silence threshold. Enable them where available.
Tune Per-Conversation
If the user just started a complex question, give them more time. If they're answering yes/no, less time. The right threshold isn't fixed.
Things That Sound Tiny But Add Up
A scattered list of small fixes that compound into a noticeably more human agent:
- Don't be too perfect. Real receptionists say "okay so" and "alright let me see". Your prompt should explicitly allow this register.
- Vary energy. A friendly "Hi there!" for a greeting, a calm "Mm-hmm" for an acknowledgement, a serious tone for a complaint. Most agents speak in one flat register.
- Skip "I". A lot. Real humans say "Got it" not "I got it". "Let me check" not "I'll let me check". Compress.
- Avoid corporate phrasings. No "I would be happy to assist you with that". Just "sure, I can help with that". Or "yeah, no problem".
- Acknowledge before clarifying. When the user says something ambiguous, lead with "okay" or "got it" before asking the clarifying question. Going straight to the question feels interrogative.
- Don't repeat their phrasing exactly. If they said "I want to book an appointment", you say "let's get you booked", not "I'll book your appointment for you". Mirroring word-for-word feels mechanical.
- Match formality. If they say "yo, can I get a meeting", don't reply with "Certainly, I can arrange a meeting for you". Match the register.
How to Test "Sounds Human"
Don't trust your own ear. You wrote the prompts. You know what the agent is going to say. Real testing:
- Have someone who has never used the agent call it. Watch their face.
- Send the recording to 5 friends without telling them what it is. Ask "is this AI or a real person?"
- Listen back to your own calls a week later, when you've forgotten the exact responses. The robotic moments will jump out.
- A/B test two voices and two prompt styles with real callers. The differences are bigger than you'd guess.
This Is Exactly Why We Built the OnCallClerk SDK
The voice quality work is the hardest part of voice AI to get right and the hardest part to ship from scratch. Every detail in this article is a separate orchestration challenge: backchanneling needs a parallel audio pipeline, filler words need state-aware tool call wrappers, barge-in needs cross-stage cancellation, prosody needs careful TTS integration.
The OnCallClerk SDK ships with all of these baked in. Backchannels are automatic. Filler audio plays during tool calls. Barge-in is handled with proper buffer clearing. Endpointing adapts per-conversation. You bring your business logic and your tone-of-voice prompt; we handle the conversational craft.
The API reference shows how to configure agent personality and voice. The SDK drops in alongside whatever stack you're already running.
If your agent sounds robotic and you're tired of fighting the orchestration to make it sound real, this is the shortcut.
Keep Reading
- How to Build a Low Latency AI Phone Agent - Architecture overview
- How to Reduce Latency in Voice AI Agents - Performance debugging
- Why Voice AI Costs More Than Expected - Cost decomposition
- Fastest Way to Launch a Voice AI Product - Time-to-market guide