How well does AI handle crosstalk? The honest answer, with the data.

Most people assume the hard part of multi-speaker transcription is telling two similar voices apart – deciding whether that sentence belongs to Speaker A or Speaker B. That is not where the errors come from. Modern diarization is quite good at clustering distinct voices. What it struggles with is far more basic: hearing two people at the same time.

The task of labeling who spoke when is called speaker diarization, and if the term is new, our primer on what speaker diarization is covers the mechanics. Its accuracy is measured by Diarization Error Rate (DER), which sums three kinds of mistake: missed speech (the system detected no one when someone was talking), false alarm (it detected speech in silence), and speaker confusion (it heard someone, but tagged the wrong person).

Here is the finding that reframes the whole problem. Across systems, the dominant error is missed speech, not confusion. And missed speech is exactly what crosstalk produces: when two talkers overlap, the model typically registers only one of them. So the reason AI struggles with people talking over each other is not that it mixes up their voices – it is that it fails to notice the second voice at all.

Last verified: 2026-07-12. Diarization benchmarks move as models are updated; the numbers below are current as of this date.

The cleanest way to see that crosstalk – not the model – is the bottleneck is to run one system across datasets of increasing overlap. The open-source pyannote 3.1 pipeline, benchmarked in its own repository, does exactly this, and the error climbs in lockstep with how much people talk over each other.

On VoxConverse (relatively clean video) it scores 11.2% DER. On AMI meeting audio it lands at 18.8% (individual headset mics) and 22.7% (a single distant mic). On DIHARD III, a deliberately hard collection, 21.4%. And on CALLHOME telephone conversations – two people, one channel, constant interruption – 28.5%. Same model, no retraining. The error more than doubles from the cleanest condition to the most overlap-heavy one.

The pattern holds for newer commercial systems too, and in a way that confirms the diagnosis: the hardest, most overlapping datasets are precisely where better overlap handling buys the most. pyannoteAI's Precision-2 model cuts CALLHOME from 28.5% to 16.6%, DIHARD III from 21.4% to 14.7%, and AMI-IHM from 18.8% to 12.9% versus the open-source pipeline. The biggest absolute gains are on the noisiest, most talked-over audio.

One caution before you take any of these numbers to the bank: DER is not comparable across papers. Different benchmarks score it differently – some forgive small timing gaps with a collar, some ignore overlap entirely, others score every millisecond. We flag the scoring convention wherever it matters, and you should never rank two tools by DER figures pulled from different studies.

Enough to matter, and more than most people guess. In the AMI meeting corpus, roughly 81% of voiced speech is a single speaker, about 15% is two people at once, and around 4% is three or more. So close to one second in five of actual talking is overlap. And most of it is the everyday kind – two-speaker overlaps make up about three-quarters of all overlap regions. This is not exotic chaos; it is a normal meeting where people finish each other's sentences.

Telephone calls carry meaningful crosstalk too. CALLHOME, built from two-party conversations, is commonly reported at roughly 12-13% overlapped speech on average (a secondary figure – treat it as approximate). The verified anchor is the outcome: that same overlap is why one model scores 28.5% DER on CALLHOME versus 11.2% on clean video.

The cruel twist is how far the damage exceeds the airtime. DIHARD III's evaluation partitions contain only about 8.75% to 9.35% overlapped speech, yet the best systems still scored 13-19% DER. Overlap occupies a small slice of the timeline and does an outsized share of the harm – a few percent of a recording can dominate the error.

If crosstalk really is the bottleneck, then teaching a model to handle overlap should move the numbers a lot on its own. It does. Adding overlap-aware neural resegmentation – a step that specifically looks for regions where more than one person is speaking – dropped DER on AMI headset audio from 29.7% to 23.8%. That is a 20% relative improvement, 5.9% absolute, from doing nothing but handling overlap better.

The same overlap-aware segmentation delivers consistent gains across corpora over a strong baseline: 17% relative on AMI, 13% on DIHARD III, 13% on VoxConverse. A single change, aimed squarely at crosstalk, buys double-digit improvements everywhere it is tried. That is what a bottleneck looks like when you widen it.

And the error decomposition tells the same story from the inside. In that overlap-aware AMI result, the errors broke down as 13.0% missed detection, 7.2% speaker confusion, and 3.6% false alarm. Missed speech is nearly double the confusion – and even after adding overlap handling, missed detection is still the largest slice. A 2025 benchmark across 196.6 hours of multilingual audio found the same thing across every model tested: missed speech is the leading error mode, ahead of speaker confusion, and the gap widens as more speakers pile on. The machine is not confused about who is talking. It is deaf to the second voice.

Not all reported DER figures are earned the same way, and the gap between them is where a lot of marketing lives. Many CALLHOME-style results forgive a quarter-second of slop around every speaker boundary (a 0.25s collar) and some ignore overlapping regions entirely – both of which flatter the score.

DIHARD III did neither. It was scored with no forgiveness collar and with overlapping speech fully counted, which is the toughest reasonable way to grade diarization. Under those rules, the winning system reached 13.45% DER when handed perfect speech-activity boundaries (Track 1), and 19.37% DER when it had to detect speech from scratch (Track 2). Those are the best results anyone posted, on a leaderboard designed to punish exactly the overlap failures we have been describing.

The takeaway is not that 13% is bad. It is that the honest number, on hard audio, scored without mercy, is meaningfully worse than the 5-8% figures vendors cite on clean benchmarks. pyannoteAI itself frames state-of-the-art as 5-8% DER on clean audio but 15-25% on challenging real-world recordings. When you read a diarization accuracy claim, the first question is always: on what audio, scored how?

DER is a time-based metric. You do not read time. You read a transcript, and what makes you wince is a sentence attributed to the wrong name – the interviewer's question stapled onto the subject's answer. That mistake has its own metric: Word Diarization Error Rate (WDER), the fraction of correctly-transcribed words handed to the wrong speaker.

WDER is not the same as Word Error Rate, which measures whether the words themselves are right – our word error rate explainer covers that one. WDER assumes the words are correct and asks only whether the speaker label is. It is the metric closest to how a labeled transcript actually reads, and it is highest exactly where speakers overlap.

The good news is that word-level speaker errors are large but correctable. Running an LLM over the raw output as a cleanup pass – Google's DiarizationLM – cut WDER by 55.5% on the Fisher telephone set (5.32% down to 2.37%) and by 44.9% on CALLHOME-English (7.72% to 4.25%). Note that these gains depend on the underlying ASR-plus-diarization stack and telephone-domain data; they show that word-level speaker errors are large and fixable, not that any given recording will halve. But the direction is clear: post-processing can rescue a meaningful share of mis-attributed lines.

The uncomfortable truth is that no shipping consumer tool solves heavy crosstalk. Segment-level diarization plus overlap-aware detection is now standard, and an LLM cleanup pass can nearly halve word-level speaker errors – but if three people are genuinely talking at once, the audio simply does not contain two of them cleanly, and no model conjures back what the microphone never captured.

So the biggest wins are upstream, before any AI runs. Record separate tracks or give each speaker their own mic – on separate channels, there is no overlap to resolve, because each voice lives in its own file. Place mics close to reduce bleed. Denoise first. If you control the recording, this single decision beats any diarization model on the market. Our guide to transcribing multiple speakers walks through the setup, and the broader improving transcription accuracy guide covers the rest of the upstream fixes.

When you cannot re-record – you have one messy file and one shot – the realistic workflow is a strong model to get labels close, then a human correction pass on the overlap regions, which is where the errors cluster. Knowing that missed speech dominates tells you where to look: scan for lines where one speaker's turn swallowed the start of the next.

We are not going to claim Pepys solves crosstalk, because the published data says no consumer tool does. What we can do is make the correction pass fast. Pepys gives you per-word speaker labels you can hand-edit, so when the model misses the second voice in an overlap – the failure this whole post is about – reassigning that stretch is a click, not a re-listen from zero.

If your audio is clean and separated, diarization will mostly do the right thing and you will barely touch it. If it is a messy three-way conversation, no tool on the market gets it perfect, and the honest advice is to record separate tracks next time. In between – the ordinary two-person interview with normal interruptions – you want good labels plus easy correction, which is the speaker diarization workflow we built. Read the data first; it will tell you what to expect before you upload a single file.