The operator's lens: what breaks in month three
Most voice agent comparisons are written from a demo. Demos are quiet, scripted, and never hit a queue. The problems that end voice programmes appear later: a per-minute bill that triples when average handle time slips, a turn detector that talks over customers with an accent, an escalation path that drops the caller instead of transferring them, and an audit request you cannot answer because transcripts were never the source of truth.
So the right question is not 'which platform is best' but 'which constraint do I want to own'. A model API hands you the lowest latency and the entire operational burden. A hosted platform hands you plumbing and a monthly ceiling on how weird your agent is allowed to be. An open framework hands you the truth about your costs and nothing else.
Score every candidate against the workflow, not the feature list. Map the call end to end — greeting, identification, the one thing the caller actually wants, the handoff — and count how many of those steps the platform makes cheap. The winner is usually the one that makes your worst step boring.
How we compare them
- End-to-end latency, not voice quality. The single number that decides whether a caller relaxes or talks over the agent is the gap between the end of their sentence and the first syllable of the reply. Under roughly 800ms a conversation feels human; past about 1.2s people start repeating themselves. Beautiful audio delivered late still sounds broken.
- Turn-taking and interruption. Real speech is full of overlaps, 'mm-hm's and false starts. Platforms differ enormously in whether they treat a half-second pause as your turn ending. Test with a noisy room and an indecisive speaker, not a clean script.
- All-in cost per minute. Hosted platforms quote a platform fee; model-agnostic ones quote a fee plus three vendor bills. Always model a realistic minute — including silence, hold time and the failed calls you still pay for — before comparing headline prices.
- Escape hatches. Ask what happens when the model is wrong: can a human take the call over, can you replay audio, can you export transcripts, and can you move to another vendor without rewriting your agent logic?
- Compliance and recording. Consent capture, retention windows, redaction of card and health data, and regional processing are not features you bolt on later. If your sector regulates calls, this criterion outranks latency.
The platforms at a glance
| Platform | Type | Architecture | Pricing shape | Best for |
|---|---|---|---|---|
| OpenAI Realtime API | Model API (build-your-own agent) | Speech-to-speech | Metered per audio minute in and out; no platform fee, but you pay for the orchestration you build. | Teams with engineers who want the lowest possible latency and full control of the stack. |
| Vapi | Voice agent platform | Either | Per-minute platform fee on top of the model, ASR and TTS vendors you select. | Engineering teams that want platform plumbing but refuse to be locked to one model vendor. |
| Retell AI | Voice agent platform | Cascade (ASR → LLM → TTS) | Bundled per-minute pricing that already includes model and voice costs. | Business teams shipping call-handling agents this quarter without a platform team. |
| ElevenLabs Agents | Voice-first platform | Cascade (ASR → LLM → TTS) | Character/minute credits, with voice quality effectively the premium you pay for. | Brands where how the agent sounds matters more than how the agent is wired. |
| Bland AI | Telephony-first platform | Cascade (ASR → LLM → TTS) | Flat per-minute, with enterprise tiers for dedicated infrastructure. | Contact-centre style workloads measured in tens of thousands of calls. |
| LiveKit Agents | Open-source framework + cloud | Either | Open source to self-host; usage-based if you take the managed cloud. | Product teams embedding voice inside their own application rather than on a phone line. |
| Pipecat | Open-source orchestration framework | Cascade (ASR → LLM → TTS) | Free framework; you pay only the underlying vendors. | Teams that want to measure the market honestly before committing to a vendor. |
Platform by platform
OpenAI Realtime API
Model API (build-your-own agent) · Speech-to-speech
Latency. Lowest class — a single model hears and speaks, so there is no transcribe-then-generate round trip.
Pricing. Metered per audio minute in and out; no platform fee, but you pay for the orchestration you build.
Strengths
- Genuinely interruptible conversation with natural barge-in and backchannel sounds.
- Tool calling in the same turn as speech, so the agent can look something up mid-sentence.
- No vendor layer between you and the model — nothing to migrate off later.
Trade-offs
- You own telephony, call recording, retries, analytics and compliance yourself.
- You cannot swap the language model for a cheaper one — the voice and the reasoning are the same model.
- Transcripts are a by-product rather than the source of truth, which complicates QA workflows.
Vapi
Voice agent platform · Either
Latency. Very good on a tuned cascade; competitive with speech-to-speech for short turns.
Pricing. Per-minute platform fee on top of the model, ASR and TTS vendors you select.
Strengths
- Model-agnostic: pick your own ASR, LLM and voice, then change any one of them without a rewrite.
- Strong developer surface — webhooks, server-side tool calls, call artifacts.
- Telephony, warm transfer and voicemail detection handled for you.
Trade-offs
- Costs stack: platform fee plus three vendor bills is easy to under-forecast.
- The flexibility is real configuration work; there is no good five-minute setup.
Retell AI
Voice agent platform · Cascade (ASR → LLM → TTS)
Latency. Consistently low, with turn-taking tuned for phone conversations rather than demos.
Pricing. Bundled per-minute pricing that already includes model and voice costs.
Strengths
- The most predictable bill of the hosted platforms — one number per minute.
- Turn detection and interruption handling work well out of the box on noisy phone lines.
- Practical guardrails: post-call analysis, structured extraction, compliance options.
Trade-offs
- Less freedom to hand-tune the pipeline than a model-agnostic platform.
- Voice library is good but not the widest available.
ElevenLabs Agents
Voice-first platform · Cascade (ASR → LLM → TTS)
Latency. Good; the voice layer is the fastest part of the chain.
Pricing. Character/minute credits, with voice quality effectively the premium you pay for.
Strengths
- The best-sounding synthetic voices in production use, including emotion and multilingual delivery.
- Voice cloning and consistent brand voice across an entire catalogue of content.
- Same voice engine powers narration, dubbing and live agents.
Trade-offs
- Agent orchestration is younger than the voice engine it sits on.
- Cost climbs quickly once you move from demos to sustained call volume.
Bland AI
Telephony-first platform · Cascade (ASR → LLM → TTS)
Latency. Tuned aggressively for outbound phone calls at volume.
Pricing. Flat per-minute, with enterprise tiers for dedicated infrastructure.
Strengths
- Runs its own stack end to end, so call quality is unusually consistent at scale.
- Purpose-built for high-volume outbound and inbound phone operations.
- Simple pricing that survives contact with a finance team.
Trade-offs
- Least flexible if you want a non-phone surface such as in-app or web voice.
- Fewer escape hatches when you need bespoke model behaviour.
LiveKit Agents
Open-source framework + cloud · Either
Latency. Excellent, with adaptive turn detection running close to the media layer.
Pricing. Open source to self-host; usage-based if you take the managed cloud.
Strengths
- You own the code path; the framework is inspectable and forkable.
- Real-time media is the core competency, so multi-party and video-plus-voice are natural.
- No per-agent platform tax if you run it yourself.
Trade-offs
- Self-hosting is a real operational commitment: scaling, media servers, observability.
- You assemble the agent behaviour; there is no non-technical builder.
Pipecat
Open-source orchestration framework · Cascade (ASR → LLM → TTS)
Latency. Depends entirely on the vendors you plug in; the framework itself adds very little.
Pricing. Free framework; you pay only the underlying vendors.
Strengths
- The clearest mental model of a voice pipeline: frames in, frames out.
- Swap any component — including to a speech-to-speech model — without changing your app.
- Ideal for research, evaluation and cost benchmarking across vendors.
Trade-offs
- No hosting, no dashboard, no compliance story out of the box.
- Every production concern is yours to solve.
What to pick, by situation
- Inbound support triage → Retell AI. Predictable bundled pricing, phone-tuned turn detection and post-call structured extraction mean you can staff and forecast the programme like a normal operations line.
- Outbound at volume → Bland AI. Owning the full stack keeps quality flat as concurrency climbs, and flat per-minute pricing survives a finance review.
- Voice inside your own product → LiveKit Agents. Real-time media is the core competency, there is no phone line to pay for, and you keep the code path.
- Lowest-latency concierge → OpenAI Realtime API. A single speech-to-speech model removes an entire round trip; accept that you are building the operations layer yourself.
- Vendor bake-off before committing → Pipecat. Swap components behind a stable interface and measure real cost and latency on your own traffic instead of someone else's benchmark.
The verdict
- If you need something in production this quarter and cannot staff a platform team: Retell AI.
- If you have engineers and refuse to be locked to one model vendor: Vapi.
- If latency is the product: OpenAI Realtime, with LiveKit Agents underneath if the surface is your app rather than a phone.
- If nobody can agree, run Pipecat for two weeks against real calls and let the numbers decide.
Voice agents fail operationally far more often than they fail technically. Choose the platform that makes your escalation path, your audit trail and your bill boring — then spend the saved attention on the script.