Voice Agent Telephony: SIP, PSTN, and Narrowband Audio

A voice agent that works beautifully in a browser demo can degrade noticeably the first time it answers a real phone call. The models did not change. The audio did, along with the network path, and both are largely outside the control of the pipeline you spent weeks tuning.

Most commercial voice AI use cases are phone calls: inbound support lines, outbound campaigns, appointment reminders, callbacks to people who will never install your app. Those callers reach you over the public switched telephone network, and the PSTN imposes constraints that no model choice can overcome.

This guide covers what changes when telephony enters the picture and how to design for it. It builds on our voice AI agents guide and complements how voice AI latency works, which covers the pipeline budget this article deliberately leaves alone.

SIP and WebRTC are not competing choices

If your agent needs a dialable phone number, SIP is the only path. The PSTN was built on SIP signalling and does not support WebRTC [1]. That is not a preference; it is the network.

WebRTC is the right transport for browser and in-app callers, where it negotiates Opus and delivers far better audio. Most production deployments run both, and the important design consequence is that the agent runtime should be transport-agnostic [2]. The media edge differs; the prompts, knowledge base, integrations, escalation logic, and analytics should not [3].

The practical shape:

Behind all three sits one agent runtime. Teams that build separate stacks per transport end up maintaining two sets of prompts and two analytics views, which is the clearest dividing line between a pilot and a production build [3].

The two ways to connect a phone call

Managed WebSocket bridge. Providers offer a media-streaming product that terminates the PSTN call and opens a bidirectional WebSocket to your agent, so you never run a SIP stack yourself [4]. This is the fastest path to a working phone agent and the right starting point for most teams.

Your own SIP trunk and media gateway. A SIP trunk delivers the call to a gateway you operate, typically built on software such as FreeSWITCH, Asterisk, or Kamailio, which receives RTP over UDP, handles the audio conversion, and streams frames to your runtime [5]. More work, more control: over routing, over where media lands geographically, over transcoding, and over transfer behavior.

Either way, a trunk provider sits underneath handling carrier interconnects, number provisioning, caller ID attestation, and codec negotiation between the PSTN and whatever your backend expects [6].

The decision usually comes down to whether media placement matters to you. If your callers and your inference are in the same region as your provider’s media edge, the managed bridge is fine. If they are not, and the section below explains when that bites, running your own gateway may be the only way to fix it.

The narrowband problem

Telephony audio is standardized on ITU-T G.711: 8 kHz sampling, mono, 64 kbit/s, with mu-law companding in North America and Japan and A-law elsewhere [2]. Sampling at 8 kHz caps audio bandwidth below 4 kHz, roughly the 300 to 3,400 Hz band [6], which is why phone calls sound like phone calls.

The consequence for a voice agent is not cosmetic. Consonant energy above that ceiling is gone before any model hears the signal [2]. Speech recognition on narrowband audio is meaningfully less accurate than on 16 kHz or better [4], and recognition errors propagate straight through your agent: a misheard account number or medication name produces a confidently wrong answer that no amount of prompt engineering catches.

Two practical implications:

Where accuracy is critical, prefer the WebRTC path when you can. For healthcare, financial, or anything where a transcription error changes an outcome, an in-app call with Opus gives a material accuracy advantage over a phone call [4]. That is sometimes a product decision worth making explicitly: steer high-stakes interactions into the app, keep the phone line for the routine ones.

Do not waste synthesis on frequencies the network will discard. A TTS engine generating full-band audio for a call that ends at a PSTN gateway is producing content that gets stripped at the edge [6]. Systems serving both paths should configure output sampling per path: full band for WebRTC, narrowband for SIP.

Transcoding: where the sources disagree

Published guidance splits, and the split is worth understanding rather than resolving by default.

One camp argues for keeping G.711 end to end on SIP trunks to eliminate transcoding overhead in the transport layer [1]. The other argues for resampling to 16 kHz linear PCM at your own gateway using optimized DSP libraries, on the grounds that modern speech models expect 16 kHz and feeding them raw 8 kHz degrades accuracy, and that you should not make cloud APIs do the conversion [5].

Both are right about different costs. Transcoding adds processing time; not transcoding hands your models audio in a format they were not built for. The resolution depends on your stack:

• If your speech-to-text provider has telephony-optimized models, trained on real PSTN audio, passing G.711 through with minimal handling is often the better trade. Several leading providers do.
• If you are using speech-to-speech models, resampling matters more. Those models depend on acoustic detail, and feeding un-resampled 8 kHz audio into them can produce synthesis glitches and misread tone [5].
• If you transcode, do it once, at the edge, in memory, with efficient resampling, rather than repeatedly across service boundaries [5].

The general rule that survives both positions: align formats across the gateway and the speech engines, and transcode as little as possible [2].

Sampling rate of G.711 telephony audio

8 kHz

Which caps usable audio bandwidth below 4 kHz.

Approximate band that survives the network

300-3,400 Hz

Consonant energy above it is gone before any model hears the call.

Channel

Mono, 64 kbit/s

Mu-law companding in North America and Japan, A-law elsewhere.

Carrier routing is latency you cannot optimize away

The pipeline work covered in how voice AI latency works governs everything after audio reaches your stack. Before that, the carrier network charges you a toll on every call.

Each carrier hop adds roughly 20 to 50 milliseconds, and three to five hops is normal, so 150 to 300 milliseconds can be spent before your recognition model processes a single phoneme [3]. By call geography, published figures put a mobile call to a trunk in the same metropolitan area at roughly 50 to 100 milliseconds of carrier routing, cross-country at 100 to 200, and international at 200 to 400 [6].

Two things follow.

This cost is paid on every call regardless of how fast your stack is [6]. A team that gets its pipeline to 500 milliseconds and routes calls internationally has a caller experience closer to 900.

Where your provider’s media edge sits is a first-order design decision. If your callers are in one country and your provider’s nearest media edge is in another, every call takes an international detour before your agent hears anything. The fix is a trunk provider with domestic presence and inference co-located with it, which frequently means running your own gateway rather than accepting a managed bridge’s edge locations.

The related trap is number geography. A number provisioned in a different country than your callers causes international hairpinning on the PSTN before traffic ever reaches your provider. Local dialable numbers are not uniformly available, and in several countries DID provisioning is restricted or requires local registration and documentation. Check availability in your target market early, because it can invalidate an architecture that otherwise looked fine.

Caller ID attestation decides whether anyone answers

For outbound calling, the technical stack is only half the problem. Carriers increasingly flag calls without proper attestation as likely spam, which directly reduces answer rates and undermines the economics of an outbound programme [1].

In the US, that means STIR/SHAKEN attestation, with Level A being the standard to aim for on trunks used for outbound AI calling [1]. Practically: use numbers you legitimately control, register them properly with your provider, monitor your numbers’ reputation, and expect answer rates to be a function of caller ID health as much as of your agent’s quality.

Other jurisdictions have their own regimes and their own rules about automated calling, consent, and disclosure. Treat this as a regulatory requirement to check per market rather than a technical detail, alongside recording consent and the disclosure obligations covered in enterprise AI governance.

Design for transfer from the start

Every serious phone agent needs to hand off to a human, and transfer is where hastily built agents fall over.

Mechanically there are two approaches: send a SIP REFER to the carrier, which releases the call and is cheap, or anchor the media on your own session border controller and bridge the human in, which keeps you in the path and lets you continue recording or monitoring [5]. REFER is simpler; anchoring gives you continuity.

What matters more than the mechanism is what transfers with the call. A caller who has spent two minutes explaining a problem to an agent and then repeats all of it to a human has had a worse experience than if they had waited on hold. Pass a conversation summary, the caller’s verified identity, and what the agent already attempted. Also define what happens when no human is available, because that path gets exercised constantly and is usually unspecified.

Failure modes specific to telephony

• One-way audio. A classic SIP and NAT problem: signalling succeeds, media does not flow in one direction. Detect it by monitoring for calls with no inbound audio energy rather than waiting for complaints.
• Media gateway saturation. Transcoding is CPU-bound, and gateways degrade under concurrency in ways that show up as audio artifacts before they show up as errors.
• WebSocket disconnects mid-call. Media bridges drop under load and network events. Count reconnections per call as a metric, and make reconnection seamless rather than fatal.
• DTMF. Callers press keys, and tone handling has its own path that is easy to forget until an IVR handoff needs it.
• Silence and hold music. Inbound audio that is not speech will still hit your recognition model. Handle it explicitly rather than letting hold music become a transcript.
• Call recording obligations. Consent requirements vary by jurisdiction and often need to be announced at the start of the call, which is a product decision with a latency cost on every call.

The general reliability patterns in making LLM features reliable apply, with one telephony-specific tightening: there is no spinner on a phone call, so failure handling has to fit inside a conversational turn.

What to measure

Alongside pipeline metrics, telephony deployments need carrier-layer and call-outcome signals:

• Carrier-side latency, separated from pipeline latency, so you know which half to fix
• Call setup time and connection failure rate
• Audio quality signals: packet loss, jitter, one-way audio detection
• Transcoding load and gateway saturation
• Media bridge reconnections per call
• Answer rate and, for outbound, caller ID reputation
• Transfer rate, transfer success, and abandonment during transfer
• Call completion and where in the call people hang up

Segment all of it by number, region, trunk, and language. An aggregate that looks healthy routinely hides one trunk or one region performing badly. Instrumentation approach is the same as in LLM observability and tracing, with the alerting discipline from monitoring LLM quality in production.

Common mistakes

• Building separate agent stacks per transport instead of one transport-agnostic runtime.
• Testing only on browser WebRTC, then discovering accuracy loss on the first real call.
• Synthesizing full-band audio for calls that terminate at a narrowband gateway.
• Transcoding repeatedly across service boundaries instead of once at the edge.
• Ignoring where the provider’s media edge sits relative to callers and inference.
• Discovering local number availability restrictions after committing to an architecture.
• Launching outbound campaigns without attestation, then blaming the agent for low answer rates.
• Treating human transfer as an afterthought, so context is lost at handoff.
• Monitoring pipeline latency only, with no visibility into the carrier half.

Checklist

• One agent runtime serves every transport; only the media edge differs.
• Trunk provider and media edge are located near both callers and inference.
• Local number availability in each target market is confirmed.
• Audio format is aligned across gateway and speech engines, transcoded once at most.
• TTS output sampling is configured per path.
• Speech recognition is evaluated on real narrowband call audio, not clean recordings.
• Outbound numbers carry proper attestation and their reputation is monitored.
• Transfer passes context, and the no-human-available path is defined.
• Carrier-layer metrics are captured separately from pipeline metrics.
• Recording consent and disclosure requirements are handled per jurisdiction.

Keep reading

• Voice AI agents for the enterprise (the pillar for this series)
• How voice AI latency works
• Voice AI: build vs buy
• Voice AI for member support
• Making LLM features reliable

CMW AI Labs builds production voice agents on real telephony, including multilingual deployments where narrowband audio and regional routing decide whether the agent works at all. If your voice agent performs well in testing and worse on the phone, talk to our team.

Sources

1. ConnexCS, “WebRTC vs SIP for AI Voice Agents: Which Architecture Scales?” May 2026. https://connexcs.com/blog/webrtc-vs-sip-for-ai-voice-agents-which-architecture-scales/
2. Picovoice, “Voice Agent Telephony: PSTN, SIP, and WebRTC Transport.” https://picovoice.ai/guide/voice-agents/telephony-transport/
3. Relinns, “WebRTC vs SIP for AI Voice Agents: The 2026 Stack That Scales,” June 2026. https://relinns.com/blogs/webrtc-vs-sip-for-ai-voice-agents
4. Prodinit, “Voice AI Latency: Sub-250ms Architecture Guide,” June 2026. https://prodinit.com/blog/production-voice-ai-agents-latency-architecture
5. Hire VoIP Developer, “Speech-to-Speech vs STT-LLM-TTS for SIP Voice Agents,” August 2026. https://www.hirevoipdeveloper.com/blog/speech-to-speech-vs-stt-llm-tts/
6. Telecom Audit Guide, “AI Voice Infrastructure 2026: Streaming, WebSocket, and Telephony Integration Guide,” July 2026. https://www.telecomauditguide.com/ai-voice-infrastructure