Voice AI agents in production 2026: LiveKit Agents, OpenAI Realtime, Pipecat, Vapi, and Retell compared

Voice was the last interface AI did not own. In 2024 most production voice agents were a glued-together demo: a Twilio number, a Whisper transcript, a GPT-4 call, and an ElevenLabs round trip that left callers waiting two seconds for a single “hold on.” In 2026 it is a category. LiveKit Agents reached 1.0 in April 2025 and ships an adaptive turn-detection model with native MCP tool support on the Python 1.5.x line. OpenAI made the Realtime API generally available on 28 August 2025 with gpt-realtime, SIP calling, remote MCP, and image inputs. Pipecat Cloud went GA on Daily’s enterprise infrastructure. Vapi and Retell hardened into the no-code stack for teams that need a phone number to ring on Monday and a dashboard by Tuesday. Gartner expects conversational AI to take $80B of contact-center labour out of the system in 2026; half of customer-service phone interactions in developed markets are forecast to be AI-handled by 2027. The technology is no longer the risk. The architectural calls are.

Why voice changed shape in 2025

Three forces collapsed the stack. First, the latency floor moved. Deepgram Aura-2 ships sub-200ms time-to-first-audio (tunable to ~90ms), Cartesia Sonic 4 publishes ~40ms, ElevenLabs Flash v2.5 sits at ~75ms. STT joined the party - Deepgram Nova-3 lands a partial in roughly 150ms. The 300ms response budget that defines natural conversation, once aspirational, became the new baseline. Second, the orchestration problem was solved in the open. LiveKit Agents, Pipecat, and the OpenAI Agents SDK all ship turn-taking, interruption, function calling, and telephony as first-class primitives. The DIY pipeline is no longer a competitive moat; getting it right is now a configuration concern. Third, the commercial pull caught up. Voice AI runs at roughly $0.40/call against $7–$12/call for a human agent, and Forrester pegs the 3-year ROI between 331% and 391%. The procurement objection collapsed.

The result is the most opinionated category we have shipped client work into in years. The platforms have converged on a small set of choices, and the architectural decisions a team gets to make have narrowed to four: cascaded or speech-to-speech, managed orchestration or open-source, telephony in-platform or BYO, and how seriously you take observability. Everything else is a knob.

Cascaded STT-LLM-TTS or speech-to-speech?

The single biggest decision a voice team makes in 2026 is whether to ship a cascaded pipeline (STT → LLM → TTS, three models in a chain) or a speech-to-speech model that takes raw audio in and emits raw audio out. The two architectures are not subtly different. They have different latency profiles, different debuggability stories, different price curves, and different failure modes.

Cascaded pipelines are the production default. They dominate enterprise deployments because every component is independently observable, every transcript is auditable for compliance, and the provider matrix is deep - five-plus credible STT vendors, seven-plus TTS, dozens of LLMs. The cost is predictable (~$0.0095–$0.17/min depending on the choice of provider and model), and you can hot-swap any single stage without rewriting the agent. The trade-off is the latency budget: STT runs 100–500ms, LLM runs 350–1000ms, TTS runs 75–200ms, plus 50–200ms of network between vendors. Streaming overlaps most of that, but a poorly configured stack still surfaces at 800ms–2s end-to-end.

Speech-to-speech (OpenAI gpt-realtime, Gemini Live, xAI Grok Voice, Moshi) skips the text layer. Audio in, audio out, one model. The benchmarks are striking - Moshi publishes ~160ms end-to-end on a 7B parameter model with full-duplex streaming, and OpenAI’s Realtime API gets close on the right network. Tone, prosody, and emotional carry-over are noticeably better because no information is lost through text. The price you pay is everywhere else: cost ranges 182x between providers (Gemini 2.0 Flash Live at ~$0.00165/min to GPT-4o Realtime at ~$0.30/min), audit logs are thinner because there is no canonical transcript, and the provider list collapses to OpenAI, Google, and a small set of open models. Hot-swapping is a rebuild, not a configuration change.

Our default on client work is a cascaded pipeline for any workload with a compliance or audit requirement, and a hybrid for latency-bound consumer experiences - speech-to-speech for the small-talk surface, cascaded for the tool-calling path where the transcript matters. The pure speech-to-speech case is real but narrow: real-time UX where prosody is the product (companions, coaching, conversational assistants) and the model never has to prove what it said.

The components a voice agent actually has

Strip away the SDK glue and a production voice agent is the same six-component shape across every platform.

Transport. WebRTC for browser and mobile, SIP/PSTN for the phone network, WebSocket for server-to-server. Audio quality, jitter resilience, and echo cancellation all live here. LiveKit, Daily, and Twilio are the production transports; rolling your own WebRTC at scale is a year of accidental complexity nobody on the engineering team signed up for.
Voice activity and turn detection. Two layers. Energy-based VAD (Silero is the default) decides when the user is speaking; a semantic turn-detection model decides when the user is done speaking. Rule-based VAD-only systems interrupt callers mid-thought; semantic detection (LiveKit’s adaptive turn model, Pipecat Smart Turn v2, VideoSDK Namo, Turnsense) cuts the worst failure mode by an order of magnitude. Plan on both.
STT. Streaming, with interim results, and a partial-final reconciliation strategy. Deepgram Nova-3, AssemblyAI Universal-Streaming, and OpenAI gpt-4o-transcribe are the production three. Word-error rate matters; tail latency matters more.
LLM. GPT-4.1, Claude Sonnet 4, Gemini 2.5 Flash, Llama 4 - pick the cheapest model that hits your tool-call accuracy bar. Voice agents are tool-bound far more often than chat agents, and a mid-tier model with a tight system prompt beats a frontier model with a sloppy one every time. Stream the response. Always.
TTS. Cartesia Sonic 4 at ~40ms time-to-first-audio is the latency leader; ElevenLabs Flash v2.5 (~75ms) wins on voice range and language coverage; Deepgram Aura-2 is the enterprise-on- prem play. Streaming, chunked, with mid-stream cancellation support is table stakes - without it, barge-in is broken.
Orchestration. The piece that holds it together - turn-taking, interruption, function calling, conversation state, retry logic, observability. This is what LiveKit Agents, Pipecat, the OpenAI Agents SDK, Vapi, and Retell all are. Most teams should not write this layer themselves in 2026.

The five frontier platforms

Five platforms cover most of the production voice deployments we see on engagements. Each has a clear shape.

LiveKit Agents. Open-source Python and TypeScript framework on the same WebRTC media servers that run millions of concurrent audio sessions. Reached 1.0 in April 2025; Python 1.5.x ships adaptive turn detection and native MCP tool support. LiveKit SIP went GA in 2025, and LiveKit Phone Numbers launched November 2025 with US local and toll-free in the Cloud dashboard. The default answer for teams that want full source control over the agent loop and the option to self-host.
OpenAI Realtime API. GA on 28 August 2025 withgpt-realtime, remote MCP servers, SIP-based phone calling, and image inputs. WebRTC for browser and mobile, WebSocket for server-to-server. The cleanest path when you want a single-vendor stack - the speech-to-speech model handles turn-taking, prosody, and tool calls in one round trip. The trade-off is the lock-in and the price.
Pipecat (+ Pipecat Cloud). Open-source Python framework from Daily with pipeline-based composition - Deepgram STT processor in, GPT-4 processor next, ElevenLabs TTS out. 40+ AI service plugins, SDKs for Python, JavaScript, React, iOS, Android, C++. Pipecat Cloud reached GA on Daily’s enterprise infrastructure with integrated PSTN and SIP, Krisp noise cancellation, HIPAA and GDPR compliance. The right answer for teams who want maximum composability without the operations bill.
Vapi. Developer-friendly managed voice platform. Deep customization, broad LLM provider matrix, transparent pricing that starts at $0.05/min and lands in the $0.20–0.33/min range in production. HIPAA adds $1,000/month. Good when you want a phone number live this afternoon and the flexibility to swap LLMs or TTS providers later.
Retell. The no-code-first competitor. Drag-and-drop builder, verified phone numbers, native SIP trunking, warm transfer, branded calling, knowledge-base retrieval in the base product. $0.07/min starter, $0.13–0.31/min in production. HIPAA, SOC 2 Type 1 & 2, GDPR included on every plan with 99.99% uptime. The default for ops teams who need compliance out of the box and a single dashboard for everything.

Two adjacent stacks are worth flagging. Twilio Voice Intelligence + ConversationRelay remains the right answer when the call already lives inside an existing Twilio Flex contact center and the migration risk of moving it is the binding constraint. Amazon Bedrock AgentCore + Pipecat is the AWS-native play for teams already deep in Bedrock and IAM - the Bedrock integration is published, documented, and supported.

LiveKit Agents: the open-source default

The pitch for LiveKit is “the agent loop is yours, the transport is solved.” The framework ships a streaming pipeline class that accepts STT, LLM, and TTS plug-ins, runs the adaptive turn-detection model, handles barge-in, and exposes tool calls as Python decorators. The same agent runs on a WebRTC room for browser users and on a SIP trunk for phone callers - the transport layer is uniform.

agent.py

bash

# pip install "livekit-agents[deepgram,openai,cartesia,silero,turn-detector]~=1.5"
from livekit.agents import (
    Agent, AgentSession, JobContext, WorkerOptions, cli, function_tool,
)
from livekit.plugins import deepgram, openai, cartesia, silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel

class SupportAgent(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions=(
                "You are Aria, a phone support agent for Acme. Be brief. "
                "If the user is mid-sentence, do not interrupt. Use tools "
                "to look up order status; never invent order numbers."
            ),
        )

    @function_tool
    async def lookup_order(self, order_id: str) -> dict:
        """Look up an order by ID. order_id is a 6-12 digit string."""
        # Internal API; egress-time auth on the gateway, not in this code.
        return await orders_client.fetch(order_id)

async def entrypoint(ctx: JobContext):
    session = AgentSession(
        # Streaming STT with interim results; partials reconciled on finals.
        stt=deepgram.STT(model="nova-3", interim_results=True),
        # Stream the LLM tokens straight into the TTS chunker.
        llm=openai.LLM(model="gpt-4.1-mini"),
        # ~40ms TTFA; chunked, cancellable, the only TTS budget that fits.
        tts=cartesia.TTS(model="sonic-4", voice="a0e99841-..."),
        # Energy VAD gates the audio; the semantic model decides EOT.
        vad=silero.VAD.load(),
        turn_detection=MultilingualModel(),
    )
    await session.start(agent=SupportAgent(), room=ctx.room)
    await session.generate_reply(
        instructions="Greet the caller and ask how you can help.",
    )

if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

Three things to notice. The TTS is Cartesia, not OpenAI - on a cascaded pipeline you almost always want the lowest-latency TTS in the category, because TTS is the last stage and any wasted ms is a perceived lag. The turn detector is a separate model on top of VAD; the energy-only path is the single largest source of production complaints. And tool calls are plain decorated Python functions - the framework handles streaming the tool call out of the LLM, executing it, and feeding the result back into the same turn. The same agent attaches to a SIP trunk by configuring a dispatch rule against a LiveKit Phone Number; no code changes in the agent itself.

OpenAI Realtime: speech-to-speech, single-vendor

OpenAI’s Realtime API took the opposite stance to LiveKit. The model - gpt-realtime at GA - accepts audio and emits audio in a single bidirectional stream. There is no separate STT or TTS to wire. Tool calls, MCP servers, image inputs, and SIP phone calls all live inside the same session. The browser path is WebRTC; the server-to-server path is WebSocket. Voice agents that live entirely inside the OpenAI ecosystem ship faster on this primitive than on anything else in the category.

src/agent/realtime.ts

typescript

import OpenAI from "openai";
import { RealtimeAgent, RealtimeSession, tool } from "@openai/agents-realtime";
import { z } from "zod";

const lookupOrder = tool({
  name: "lookup_order",
  description: "Fetch an order by ID from the internal orders service.",
  parameters: z.object({
    order_id: z.string().min(6).max(12),
  }),
  execute: async ({ order_id }) => {
    return await ordersClient.fetch(order_id);
  },
});

const agent = new RealtimeAgent({
  name: "Aria",
  instructions: [
    "You are Aria, a phone support agent for Acme.",
    "Speak in short sentences. Never interrupt the caller mid-thought.",
    "Use lookup_order for any order status question; do not guess.",
  ].join(" "),
  tools: [lookupOrder],
});

export async function startSession(audioElement: HTMLAudioElement) {
  const client = new OpenAI();
  const session = new RealtimeSession(agent, {
    model: "gpt-realtime",
    // Server-side turn detection model; client-side VAD is the fallback.
    turnDetection: { type: "server_vad", create_response: true },
    // Stream audio straight into a media element; cancellable on barge-in.
    audio: { output: { voice: "marin", audioElement } },
  });

  await session.connect({ apiKey: process.env.OPENAI_API_KEY! });
  return session;
}

The pattern that earns its keep on Realtime is end-of-turn delegation - let the server-side turn detector decide when to respond, then invalidate the audio buffer on every detected barge-in. The single biggest bug we see on Realtime deployments is leaving the audio element playing while the model speaks over a user who was already mid-sentence; the SDK does the right thing if you wire theserver_vad path and cancel on incoming audio. The cost story is the other side of the trade-off: GPT-4o Realtime lands at roughly $0.30/min on the public price card, which is competitive for a contact-center call (still a fraction of human agent cost) and uncompetitive for a chat-style assistant where a cascaded stack lands at a sixth of the price.

Pipecat: the composable pipeline

Pipecat’s defining choice is that everything is a processor in a pipeline. Audio comes in through a transport processor, runs through a VAD processor, a STT processor, a context aggregator, an LLM processor, a TTS processor, and out through the same transport. Each processor implements a uniform interface; swapping Deepgram for AssemblyAI or ElevenLabs for Cartesia is a one-line change. Pipecat Cloud - Daily’s managed deployment platform - runs the same pipeline code on enterprise infrastructure with integrated SIP, PSTN, Krisp noise cancellation, and HIPAA compliance.

bot.py

bash

# pip install "pipecat-ai[daily,deepgram,openai,cartesia,silero]"
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask, PipelineParams
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.services.cartesia import CartesiaTTSService
from pipecat.services.deepgram import DeepgramSTTService
from pipecat.services.openai import OpenAILLMService
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.transports.services.daily import DailyParams, DailyTransport

async def main(room_url: str, token: str):
    transport = DailyTransport(
        room_url, token, "Aria",
        DailyParams(
            audio_in_enabled=True,
            audio_out_enabled=True,
            vad_analyzer=SileroVADAnalyzer(),
            turn_analyzer=None,  # set to PipecatSmartTurn() in production
        ),
    )

    stt = DeepgramSTTService(api_key=os.environ["DEEPGRAM_API_KEY"])
    llm = OpenAILLMService(api_key=os.environ["OPENAI_API_KEY"],
                          model="gpt-4.1-mini")
    tts = CartesiaTTSService(api_key=os.environ["CARTESIA_API_KEY"],
                             voice_id="a0e99841-...",
                             model="sonic-4")

    context = OpenAILLMContext([
        {"role": "system", "content": "You are Aria, a phone support agent..."},
    ])

    pipeline = Pipeline([
        transport.input(),
        stt,
        llm.context_aggregator(context).user(),
        llm,
        tts,
        transport.output(),
        llm.context_aggregator(context).assistant(),
    ])

    task = PipelineTask(pipeline, params=PipelineParams(
        allow_interruptions=True,
        enable_metrics=True,
    ))
    await PipelineRunner().run(task)

The Pipecat pattern that pays off in production is the context aggregator pair - one before the LLM that captures the user’s finalised utterance, one after that captures the assistant’s spoken response. Without both, you cannot reliably reconstruct what the agent actually said versus what the LLM intended to say after a barge-in. With both, the conversation history is the same one the model sees on the next turn, which means tool-call accuracy stays stable over long calls. Pipecat Cloud takes the same pipeline file and runs it behind a managed SIP number with autoscaling and observability; on-prem, the same file runs on a single container.

The latency budget, by component

Numbers worth pinning the architecture to. Cascaded pipeline, mid-2026 production figures, streaming where applicable.

STT first partial: Deepgram Nova-3 ~150ms, AssemblyAI Universal-Streaming ~200ms, OpenAI gpt-4o-transcribe ~250ms.
End-of-turn detection: Silero VAD-only ~200ms fixed; LiveKit adaptive / Pipecat Smart Turn / Namo +20ms over VAD with semantically correct turn endings.
LLM first token: GPT-4.1-mini ~250ms, Claude Sonnet 4 ~300ms, Gemini 2.5 Flash ~200ms. Streaming the response is non-negotiable.
TTS time-to-first-audio: Cartesia Sonic 4 ~40ms, ElevenLabs Flash v2.5 ~75ms, Deepgram Aura-2 ~90ms tuned.
Network overhead: 50–200ms across vendor boundaries depending on region pinning.
Speech-to-speech end-to-end: 0.78s (xAI Grok Voice) to 2.98s (Gemini 3.1 Flash Live) on April 2026 figures. Moshi’s open 7B sits at ~160ms but is rare in production.

The number to design to is 300ms time-to-first-audible-response, and the path to it is overlapping streams - the LLM emits tokens while the STT is still finalising, the TTS starts speaking before the LLM is done, and the agent listens for barge-in throughout. None of the platforms above ship that pipeline correctly out of the box on the first run; getting it right is a configuration exercise, not a research project, but it is not optional.

Production patterns that survive Monday morning

Six patterns we apply on every voice engagement that ships.

Semantic turn detection on top of energy VAD. Silero (or equivalent) decides when audio is voice; a transformer-based turn model decides when the utterance is complete. Implementing turn detection cuts mid-thought interruptions by ~87% with ~20ms of added latency. The model is tiny and free. Run it.
Barge-in that cancels both the LLM and the TTS. When the user starts speaking while the agent is talking, you need to (a) immediately stop TTS playback, (b) cancel any unfinished LLM generation, (c) discard any audio buffered after the barge-in point, and (d) preserve the conversation history accurately. Skipping any one of those four is a separate production bug. LiveKit, Pipecat, and the OpenAI Agents SDK all ship the right shape; verify it survives your retry path.
Streaming everywhere. Streaming STT interim results, streaming LLM tokens, streaming TTS chunks. A single non-streaming hop turns a 600ms agent into a 1.6s agent. Audit your pipeline for buffer boundaries on every release.
Function calls with idempotency keys. Voice calls drop. The agent retries. If the retry triggers a duplicate payment, the postmortem is bad. Every tool call that has a side effect carries a stable idempotency key derived from the conversation ID and the tool-call hash; the downstream service dedupes on it. The cost of this discipline is one line; the cost of skipping it is a refund.
Three-layer guardrails. Prompt-level instructions for tone and refusal. A separate moderation pass on user input and agent output (cheap, run async). A hard policy layer that intercepts tool calls and rejects anything outside the allowlist (cancellations only for the calling number’s account, no PII reads outside the authenticated session). The voice surface is the one the regulator looks at first; design it like the regulator already showed up.
Observability with audio replay. Every call is a trace: STT partials and finals, LLM prompt and completion, tool calls and responses, TTS chunks, end-of-turn events, barge-in events, latency per stage. Audio replay is the artefact that turns a customer complaint into a 30-second debug. LangSmith, Langfuse, Helicone, Arize Phoenix, and Future AGI all ship voice- aware traces in 2026; pick one before the first production call, not after the first incident.

Workloads where voice agents earned their seat

Four production shapes where the voice surface changed the outcome on engagements over the last year.

Contact-center deflection. The headline use case. Voice agents handle the top of the call tree - order status, appointment scheduling, password resets, balance enquiries - and escalate to a human only on intent the agent cannot resolve. Healthcare contact centers report ~30% reduction in hold time and 85% reduction in agent training time after deployment; the financial-services figures are similar. Kore, Cognigy, and IrisAgent are the reference enterprise platforms; LiveKit and Pipecat are the underlying frameworks on most custom builds.
Outbound and reactivation. Insurance renewals, appointment reminders, debt collection, lead qualification. The economics are blunt - a voice agent at $0.40/call against $7–$12/call for a human, and the agent can be on the phone at 3am. Vapi and Retell are the dominant managed platforms in this shape; the open-source build path is Pipecat + Twilio SIP for teams that want the option to swap providers later.
Voice-first applications inside SaaS products. Voice notes that summarise themselves, in-product voice assistants that drive the UI, voice-mode tutors that pause when you do. The OpenAI Realtime API is the dominant build path here because the integration surface is one round-trip and the conversational quality is the highest in the category. LiveKit + GPT-4.1 + Cartesia is the open-source equivalent.
Voice in physical and embedded contexts. Drive-thru ordering, in-car assistants, smart-home interfaces, warehouse pick-and-place. Latency is the binding constraint - 200ms is felt, 500ms is broken. Pipecat is overrepresented here because the pipeline is small enough to run on edge hardware; LiveKit’s emphasis on physical-AI agents in 2026 product messaging is a sign of where the category is going.

Where voice agents still hurt

The category is production-grade; it is not friction-free. Five failure modes that show up on most non-trivial deployments.

Compounded tail latency. Every component is fast in isolation. The combined p99 is not. A 250ms STT plus a 400ms LLM plus a 75ms TTS plus 100ms of network is 825ms median and 1.6s p99 - and the p99 is what the caller remembers. Pin regions, co-locate providers, and benchmark the actual stack, not the vendor numbers.
Number-handling and disfluency. LLMs read “1234” as “one thousand two hundred and thirty-four” until you fight them; humans say “uh, hold on” and the STT drops the “uh.” A non-trivial slice of the prompt is teaching the model to read numbers as digits, dates as dates, and currency as currency in the agent’s voice. Plan on a dedicated formatting prompt and a verification pass for high-stakes utterances.
Prompt injection over the audio channel. A caller can read out a payload exactly the way a webpage can. The defence is the same: untrusted user content is treated as data, never as instructions, and the tool layer is the security boundary. Voice does not change the threat model; it just makes the attack vector audible.
Compliance recording is non-trivial. Two-party consent jurisdictions require a recorded notice played before the agent speaks; PCI and HIPAA require redaction at ingest (Krisp, Pindrop, AssemblyAI redaction are the production options). Retention windows are jurisdiction-specific. Treat this as a gating requirement at the procurement stage, not a feature to add later.
The handoff to a human is the hardest part. A warm transfer that preserves context, transcript, and caller intent is harder to ship than the agent itself. Retell, Vapi, and Cognigy ship this as a first-class primitive; LiveKit and Pipecat give you the building blocks. Test the handoff with as much rigour as the happy path - it is the moment the agent has already failed, and the caller is already frustrated.

Comparing the frameworks

LiveKit Agents. The right answer when you want source control over the agent loop and a clear path to self-host the transport. WebRTC is solved, SIP is GA, Phone Numbers ship out of the dashboard, and the adaptive turn model is the best open implementation in the category. Strongest for teams with engineering depth and a multi-region deployment.
OpenAI Realtime API. The right answer when the stack is OpenAI top-to-bottom and the conversational quality is the product. Speech-to-speech, single round trip, MCP and SIP support GA. The trade-off is the price and the vendor lock-in; the upside is the cleanest developer experience in the category.
Pipecat (+ Pipecat Cloud). The right answer when the pipeline is the product - when you need every processor swappable and a path from local dev to managed deployment on the same code. Pipecat Cloud is the polished operations layer; the open framework runs on AWS, GCP, on-prem, or embedded edge hardware with the same source. The default for teams that want optionality.
Vapi. The right answer when developer ergonomics and flexibility are the binding constraints - broad LLM provider matrix, transparent per-minute pricing, deep customisation surface. Plan around the compliance add-on cost and the absence of native warm transfer and branded calling out of the box.
Retell. The right answer when compliance, time-to-launch, and the no-code surface are the priorities. HIPAA, SOC 2, and GDPR on every plan; warm transfer, branded calling, and verified numbers in the base product. The default for ops-led teams replacing IVR before a quarter ends.

How to introduce a voice agent on a real project

Decide cascaded or speech-to-speech before the first commit. Cascaded is the default for anything with a compliance, audit, or provider-flexibility requirement; speech-to-speech is the answer when prosody and end-to-end latency are the product. Hybrid is real but adds operational surface.
Pin the latency budget to 300ms time-to-first-audible-response on a clean network. Benchmark the candidate stack against it before the implementation starts. The platforms ship the primitives; the configuration is where the budget is won or lost.
Pick semantic turn detection on day one. Energy-VAD-only systems interrupt callers mid-thought, and that is the single biggest driver of negative call survey scores. LiveKit, Pipecat, and OpenAI Realtime all ship a model; turn it on.
Wire idempotency keys through every side-effecting tool call. Calls drop. Retries happen. The downstream service should dedupe on a stable key, not on the assumption that the agent does not retry. The first refund is the lesson; design around it.
Define the human handoff before the agent. The warm-transfer path - what context is passed, how the transcript is summarised, which queue the call lands in - is the gating UX decision. Build the dashboard view of an in-progress agent call before you ship a single agent. The supervisor sees the conversation live; the agent is allowed to fail.
Run observability and audio replay from the first call. Voice incidents are unreproducible without the audio; the trace without the audio is half the story. LangSmith, Langfuse, Helicone, Arize, and Future AGI all ship voice-aware traces in 2026 - wire one in before the production cutover.
Treat the compliance recording, redaction, and retention rules as gating requirements. PCI, HIPAA, two-party consent, GDPR - all jurisdiction-specific, all non-negotiable. Procurement asks for this on the first call; engineering should not be the team that learns it on the third.
Pilot on inbound, scale on outbound. Inbound surfaces the conversational quality and the handoff path. Outbound surfaces the economics. Run both pilots before the production rollout; the lessons from one inform the other.

The 2026 trajectory

Semantic turn detection becomes the default. LiveKit’s adaptive model, Pipecat Smart Turn v2, VideoSDK Namo, and Turnsense are all in production. By year end the energy-VAD-only pipeline is a legacy choice. Expect every managed platform to ship a semantic turn model as a flip-the- switch primitive.
Speech-to-speech wins the consumer surface. Prosody, low end-to-end latency, and a single round trip beat cascaded for any workload where the transcript is not the product. Cascaded keeps the enterprise crown for another eighteen months on the strength of debuggability and provider choice, but the gap is closing.
Telephony stops being separate from the agent. LiveKit Phone Numbers, OpenAI Realtime SIP, Pipecat Cloud integrated PSTN, Retell verified numbers - the pattern of a single primitive for “voice agent on a phone number” is the new shape. Twilio is no longer the only door; the door is the agent framework.
Voice MCP eats voice plug-ins. LiveKit ships native MCP tool support in 1.5.x; OpenAI Realtime accepts remote MCP servers at GA. The same MCP server that exposes your tools to Claude Code and Cursor exposes them to your voice agent. The integration cost collapses; the security model has to keep up.
Edge and on-device voice agents arrive. Phi-4-mini, Llama 4 8B, and Moshi are deployable on a single consumer GPU. Latency-bound use cases - drive-thru, in-car, smart-home - move on-device by year end. Pipecat’s edge story and Deepgram’s on-prem Aura-2 are the early examples.
Voice agents become evaluatable like text ones. The 2024 problem was that you could not unit-test a voice agent. The 2026 answer is voice simulation: agents that act as the caller in scripted scenarios, measure response correctness, tool-call accuracy, and latency, and run as part of CI. Coval, Future AGI, Hamming, and LangSmith voice ship this; the discipline is catching up with the runtime.

Where to go from here

Voice is no longer the experimental surface of the AI stack. The latency budget is solved. The orchestration is solved. The telephony is GA on every serious framework. The remaining work is choosing the right level of abstraction for the team that has to operate the agent, getting the turn-detection and barge-in story right on day one, and treating the human handoff and the observability layer as gating requirements rather than later improvements. The platforms have stopped being the bottleneck.

If you want a second opinion on whether your call tree should go cascaded or speech-to-speech, or help wiring LiveKit, Pipecat, or the OpenAI Realtime API into a contact-center, outbound, or in-product voice surface, reach out.