Multilingual and Code-Switching Voice Agents

Supporting two languages is a solved problem. Every major speech vendor offers Hindi and English, or Spanish and English, as separate languages [1]. What is not solved is supporting the speakers who use both inside a single sentence.

That is code-switching, and it is not broken speech or poor fluency. It is rule-governed behavior by highly proficient multilingual speakers [2], and it is how enormous populations actually talk. Reported figures put more than 70% of Indian users as naturally code-switching in daily voice conversations [2], with estimates of over 250 million people in India engaging in code-switched communication [3]. It is not an India-specific phenomenon either: the same pattern appears as Spanglish, Franglais, Taglish, and many other pairs [4].

The cost of ignoring it is real. Code-switched utterances see substantially higher word error rates than their monolingual equivalents, even on modern commercial systems [2]. Recognition errors in a voice agent do not stay contained; they produce confidently wrong answers downstream.

This guide covers what breaks and what to do about it. It builds on our voice AI agents guide.

First distinction: between sentences or within them

These two are not the same engineering problem [4].

Inter-sentential switching happens at sentence boundaries. A caller asks one question in Tamil, the next in English. A system that detects language per utterance and routes accordingly handles this adequately.

Intra-sentential switching happens mid-sentence, and sometimes mid-word. This is the one that breaks things, and it is the more common register in casual speech.

Every architectural decision below follows from which of these you actually need to support. Most teams assume the first and discover the second in production.

Why monolingual pipelines fail

Deploy a monolingual recognition model and it treats each language as a closed system [5]. Traditional systems trained on monolingual data struggle on code-switched speech because phonetics, syntax, pronunciation, and vocabulary all differ across the languages being mixed [6].

The specific architectural failure is worth understanding, because it determines what you can fix by configuration and what you cannot. The critical variable is whether language identification happens at the utterance level or the frame level [1].

Utterance-level identification makes one language decision for an entire audio segment before transcription begins. It is cheaper and works well on monolingual speech. On intra-sentential code-switching it is wrong by design: it commits to a language before the switch has occurred, then processes the remaining frames through the wrong phoneme and vocabulary priors [1]. English phonemes get run through a Hindi model, or the reverse, and the output degrades accordingly.

No amount of prompt engineering or threshold tuning recovers from that. It is an architecture problem.

Two architectures, two sets of tradeoffs

The direction of the field favors end-to-end multilingual models for code-switched workloads [4], and the latency argument reinforces it: every routing hop in a cascade is time you do not have in a voice turn. See how voice AI latency works.

That said, cascades still win in specific cases: when one language dominates and switching is rare, when you need tight control over a single language’s vocabulary and accuracy, or when the best available model for your specific language pair is monolingual.

Morphology makes some pairs much harder

Code-switching difficulty is not uniform across language pairs, and the reason is grammatical structure rather than vocabulary size.

Tanglish, the mixing of Tamil and English, is a good illustration and is the dominant communication style in contact center operations in Tamil Nadu and among Tamil-speaking communities [1]. Tamil’s agglutinative morphology creates a specific problem: English words get inflected with Tamil suffixes, producing hybrid forms that appear in neither an English vocabulary nor a standard Tamil vocabulary [1]. A recognition system with perfect coverage of both languages independently still has no entry for the form that was actually spoken.

Kanglish, Kannada and English, presents the equivalent challenge in Karnataka [1]. Similar dynamics appear anywhere an agglutinative or heavily inflected language meets English.

Two practical consequences. First, out-of-vocabulary rates are structurally higher than a monolingual benchmark would suggest, which makes vocabulary boosting for domain terms more valuable than usual; several providers expose a keyterm or hotword parameter for exactly this [5]. Second, evaluating a provider on monolingual benchmarks for each language tells you very little about how it will perform on the hybrid forms your callers produce.

Transcription is only the first thing that needs per-language work

Teams solve recognition, declare the problem handled, and then discover the rest of the pipeline was tuned for English.

Turn detection. Silence thresholds and endpointing models tuned on English prosody misfire on other languages’ pause patterns and sentence structure. Backchannel vocabulary is language-specific, so a filter built on English backchannels lets other languages’ equivalents through as false interruptions. Full treatment in voice agent turn detection.

Generating on partial transcripts. Eager generation on partial hypotheses is a reasonable latency optimization for monolingual English. For code-switched speech it is risky, because partial hypotheses in mixed-language audio are revised mid-utterance more often, so the eager generation is discarded and repeated more than it saves.

Synthesis. Voice quality, speed, and availability vary enormously by language, and the fastest voice available in English frequently does not exist in your target language at all. Code-switched output is harder still: a synthesis voice that pronounces English loanwords with the wrong phonology sounds worse to a bilingual listener than either language alone would.

The language model layer. Instruction-following and tool-calling reliability are usually weaker in lower-resource languages, and a model that reliably produces structured output in English may not in Tamil or Kannada. Test tool calling per language, not just conversation.

Script and transliteration. Code-switched text is often written in Romanized form, with inconsistent transliteration of the non-English language and a high frequency of out-of-vocabulary terms [7]. If your transcript feeds anything downstream, retrieval, CRM matching, analytics, decide on a normalization policy early. Otherwise the same spoken word arrives in three spellings and none of them match your database.

Evaluate per language and per switch type

Aggregate accuracy numbers are actively misleading here. A system at 92% overall can be at 60% on the code-switched subset, and the aggregate will never show it.

Build evaluation sets that separate:

• Monolingual utterances in each supported language
• Inter-sentential switching
• Intra-sentential switching, including hybrid inflected forms
• Domain vocabulary in each language: product names, place names, and the terms your callers actually use

Use real call recordings rather than clean studio audio, and rather than synthetic code-switching written by someone asked to mix two languages, which tends to produce switching patterns real speakers do not use. Public code-switched corpora exist for some pairs and are useful reference points [3], but your callers’ register is what matters.

Then measure the downstream thing, not just word error rate. A transcription error on a filler word costs nothing; one on an account number, a medication, or a product name costs the whole interaction. Weight your evaluation by what an error actually breaks.

The broader method is the same as elsewhere in our evals and LLMOps series: sample real calls, categorize what went wrong, and let the counts drive what you fix. See error analysis for LLM failures.

Data residency deserves an early check

Speech data is personal data, and where it is processed matters commercially and legally.

The picture varies by market and is moving. In India, for example, the DPDP Act 2023 established a data protection framework whose cross-border transfer provisions were not yet in force as of 2026; it uses a blacklist model, with data flowing freely except to specifically restricted countries, and no restricted-country list had been published, meaning no active localization obligation prevented using offshore speech providers for Indian voice data [5]. That status is progressing, so verify it rather than relying on a snapshot.

The practical advice regardless of jurisdiction: know where each provider in your stack processes audio, whether it retains it, whether it trains on it, and what your customers’ contracts require. Enterprise buyers in multilingual markets ask these questions, and provider choice for lower-resource languages is often narrow enough that residency constraints eliminate options. Find that out during design, not during procurement. See enterprise AI governance.

What to measure in production

• Word error rate per language and per switch type, on sampled real calls
• Error rate on domain-critical entities specifically: numbers, names, identifiers
• Turn-taking metrics per language: interruption rate, dead air, caller repeat rate
• Tool-calling success per language
• Task completion and transfer-to-human rate per language
• Synthesis issues: mispronunciation reports, unnatural switching in output

Segment everything by language. A deployment where one language performs at half the quality of the others is the normal case, not an edge case, and an aggregate dashboard will hide it indefinitely. See monitoring LLM quality in production.

Common mistakes

• Treating multilingual support as language selection rather than mixed-language handling.
• Choosing a provider on monolingual benchmarks for each language separately.
• Utterance-level language identification for callers who switch mid-sentence.
• Adding languages to a cascade until routing hops eat the latency budget.
• Solving recognition and leaving turn detection, synthesis, and tool calling tuned for English.
• Eager generation on partial transcripts in code-switched audio.
• No transliteration or normalization policy, so downstream matching fails.
• Evaluating on synthetic code-switching rather than real calls.
• Reporting aggregate accuracy across languages.
• Discovering residency constraints after selecting providers.

Checklist

• The required switch types, inter-sentential and intra-sentential, are explicitly decided.
• Language identification granularity matches those requirements.
• Providers were evaluated on code-switched audio, not monolingual benchmarks.
• Domain vocabulary boosting is configured per language.
• Turn detection thresholds and backchannel filters are tuned per language.
• Synthesis voices exist and sound acceptable in every supported language.
• Tool calling and structured output are tested per language.
• A transliteration and normalization policy protects downstream matching.
• Evaluation sets separate monolingual, inter-sentential, and intra-sentential cases from real calls.
• Data residency is confirmed for every provider handling audio.
• All production metrics are segmented by language.

Keep reading

• Voice AI agents for the enterprise (the pillar for this series)
• Turn detection and interruption handling in voice agents
• Telephony for voice agents: SIP, PSTN, and the narrowband problem
• How voice AI latency works
• LLM evals and LLMOps

CMW AI Labs builds multilingual and code-switching voice agents in production, including Tamil and English deployments where English-tuned pipelines do not transfer. If your voice agent works in English and falls apart in your users’ actual register, talk to our team.

Sources

1. Gnani.ai, “Why Speech Recognition Fails on Hinglish: The Code-Switching Problem in Indian ASR.” https://www.gnani.ai/resources/blogs/blog-code-switching-speech-recognition-hinglish-asr
2. Mihup, “The Challenge of Mixed-Language Commands: Hinglish, Tamilish & Code-Switching in Voice AI,” June 2026. https://mihup.ai/blog/mixed-language-commands-hinglish-tamilish-code-switching-voice-ai
3. Singh, Singh, and Kadyan, “HiACC: Hinglish adult & children code-switched corpus,” Data in Brief, July 2025. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12329218/
4. Gladia, “Code Switching in Speech Recognition: ASR Guide 2026,” July 2026. https://www.gladia.io/blog/what-is-code-switching-in-speech-recognition
5. Deepgram, “Hinglish Voice AI: Why ASR Fails and How to Fix It,” May 2026. https://deepgram.com/learn/hinglish-voice-ai-speech-recognition
6. Tsang, “Code-Switching Classification on Hinglish,” University at Buffalo CSE Technical Report 2026-41, May 2026. https://cse.buffalo.edu/tech-reports/2026-41.pdf
7. “Code-Mix Sentiment Analysis on Hinglish Tweets,” arXiv:2601.05091. https://arxiv.org/abs/2601.05091