pyannote.audio — The Open-Source Speaker Diarization Toolkit, Honestly

What is pyannote.audio?

pyannote.audio is an open-source PyTorch toolkit for speaker diarization — the task of figuring out “who spoke when” in an audio file. Given a multi-speaker recording, pyannote produces a list of speaker turns with start and end timestamps and a speaker label for each turn. It does not transcribe (no text output) and does not identify named speakers (output is “Speaker A, Speaker B” not “Alice, Bob”). Both of those layers come from downstream processing — usually Whisper for transcription and a manual or AI rename step for named labels.

Project basics

• ● Maintainer: Hervé Bredin (originally LIMSI/CNRS, now at École Polytechnique) and a community of contributors
• ● License: MIT for the code; models on Hugging Face are gated behind a user agreement (free to accept for research and commercial use as of 2026)
• ● Hosting: github.com/pyannote/pyannote-audio for the code; huggingface.co/pyannote for the models
• ● Framework: PyTorch
• ● Underlying tasks: voice activity detection, speaker segmentation, speaker embedding, clustering — all packaged as a single Pipeline abstraction

Version timeline

If you're starting a new project in 2026, use 3.1. The 2.x models still work for legacy projects but are no longer actively maintained, and the Powerset architecture in 3.x meaningfully improves overlap handling.

The current model — pyannote/speaker-diarization-3.1

The 3.x line introduced the Powerset multi-class formulation described in Bredin (ICASSP 2023), “Powerset multi-class cross entropy loss for neural speaker diarization.” The core insight: instead of treating diarization as a sequence of binary speaker-vs-not-speaker decisions plus a clustering step, the model directly predicts which subset of speakers is active in each frame. This handles overlapping speech natively (a frame can be labeled “Speaker A + Speaker B both active” rather than forced into one or the other) and removes the heuristics that previous segmentation + clustering pipelines required.

What changed from 3.0 to 3.1

• ● Improved handling of short speaker segments (under 1 second)
• ● Better behavior on noisy audio without retuning
• ● Same Powerset architecture; incremental rather than fundamental changes

Internal pipeline (high-level)

1. Audio is segmented into short windows (~5 seconds)
2. Each window is encoded into a speaker representation
3. The Powerset head predicts which subset of speakers is active per frame
4. Speaker embeddings are clustered to assign consistent labels across windows
5. Output: list of (start, end, speaker_label) tuples covering the full audio

The 3.1 model depends on pyannote/segmentation-3.0 for the windowing step — both models are gated separately on Hugging Face and both agreements must be accepted.

Install and run (5 minutes)

Minimum working example. Assumes Python 3.9+, a Hugging Face account, and either CPU or GPU.

Step 1: Install

pip install pyannote.audio

Step 2: Accept Hugging Face agreements

Visit both model pages in the browser, scroll to the agreement section, and accept:

• ● huggingface.co/pyannote/speaker-diarization-3.1
• ● huggingface.co/pyannote/segmentation-3.0

Then generate an access token at huggingface.co/settings/tokens (read scope is sufficient).

Step 3: Run a 5-line example

from pyannote.audio import Pipeline

pipeline = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-3.1",
    use_auth_token="hf_YOUR_TOKEN_HERE",
)

# Optional: move to GPU for production speeds
# import torch
# pipeline.to(torch.device("cuda"))

diarization = pipeline("audio.wav")

for turn, _, speaker in diarization.itertracks(yield_label=True):
    print(f"{turn.start:.1f}s - {turn.end:.1f}s: {speaker}")

Expected output:

0.0s - 4.2s: SPEAKER_00
4.5s - 9.1s: SPEAKER_01
9.3s - 12.8s: SPEAKER_00
...

Common gotchas

• ● 401 Unauthorized: Almost always means you accepted the speaker-diarization-3.1 agreement but not segmentation-3.0. Accept both.
• ● Slow on CPU: ~real-time processing speed. For batch workloads, move to GPU with pipeline.to(torch.device("cuda")).
• ● Long model load time: 5-15 seconds on first call. Cache the pipeline object; don't reload per request.
• ● Audio format issues: pyannote expects 16kHz mono WAV ideally; other formats work but converting upfront avoids edge cases. Use torchaudio or ffmpeg to normalize.

Whisper + pyannote — the standard self-hosted pipeline

pyannote on its own gives you speaker turns but no text. Whisper on its own gives you text but no speaker labels. Together they produce the standard self-hosted transcription + diarization output: text with speaker attribution per segment. This combination is the default for self-hosted production pipelines in 2026.

The pattern

1. Run Whisper (or faster-whisper) with word-level timestamps
2. Run pyannote separately to get speaker turns
3. Align: for each transcribed segment, look up which speaker was active during that time range
4. Output: a list of (start, end, text, speaker) tuples

WhisperX — the easiest integration

WhisperX wraps Whisper + pyannote + word-level alignment in a single library. Minimum working example:

# pip install whisperx
import whisperx

device = "cuda"
audio_file = "meeting.wav"
batch_size = 16

# 1. Transcribe with Whisper
model = whisperx.load_model("large-v3", device, compute_type="float16")
audio = whisperx.load_audio(audio_file)
result = model.transcribe(audio, batch_size=batch_size)

# 2. Align for word-level timestamps
align_model, metadata = whisperx.load_align_model(
    language_code=result["language"], device=device
)
result = whisperx.align(
    result["segments"], align_model, metadata, audio, device,
)

# 3. Diarize with pyannote
diarize_model = whisperx.DiarizationPipeline(
    use_auth_token="hf_YOUR_TOKEN", device=device,
)
diarize_segments = diarize_model(audio)
result = whisperx.assign_word_speakers(diarize_segments, result)

# Output: result["segments"] with speaker labels attached
for segment in result["segments"]:
    print(f"[{segment['speaker']}] {segment['text']}")

For production (not WhisperX): most production pipelines use faster-whisper (CTranslate2 backend, 2-4x faster than reference Whisper) plus pyannote called directly, with custom alignment logic tailored to the use case. WhisperX is the right starting point but most teams diverge as they scale.

Related: Whisper transcription, speaker labeling concepts, word-level timestamps reference.

Accuracy in 2026 (honest)

Diarization accuracy is measured by DER (Diarization Error Rate): the percentage of time that diarization output disagrees with ground truth, summing missed speech, false alarms, and speaker confusion. Lower is better. The numbers below are from the published pyannote/speaker-diarization-3.1 model card; verify against the current card before relying on them in production planning.

Real-world vs benchmark. Benchmarks are useful for comparing models but don't map directly to what you'll see in production. Practical accuracy on real audio:

• ● Clean two-speaker studio audio (podcast interview, recorded sales call): 95%+ correct speaker attribution. Boundaries off by 100-300ms at speaker transitions but the labels are right.
• ● Meeting-room audio, 3-4 speakers, shared microphone: 75-85% correct attribution. Drops further if heavy overlap or speakers stand at different distances from the mic.
• ● Telephony audio (8kHz, narrow band): 70-80% — Whisper and pyannote both prefer 16kHz; resampled 8kHz audio loses information that diarization relies on.
• ● Heavy overlap (debates, family arguments, talkative meetings): Drops significantly. Powerset architecture helps but doesn't solve the problem entirely.

Comparison context. Commercial services (Deepgram, AssemblyAI) report 1-3 DER points better on their internal benchmarks, typically through specialized data augmentation and proprietary post-processing. Whether that's worth $0.40-$0.60/hour vs free pyannote depends on your scale and operational tolerance. See our best speaker diarization tools comparison for broader context.

When to use pyannote vs a hosted service

The decision usually comes down to operational tolerance, not raw accuracy. Both pyannote and hosted services are good enough for production; the question is what you want to spend engineering time on.

A realistic decision rule: if processing volume is <500 hours/month, hosted almost always wins on total cost (engineering time + ops + accuracy). At 500-5000 hours/month, the comparison gets close. Above 5000 hours/month, self-hosted typically wins on cost if you have engineering capacity to maintain it.

Operational reality check

The hidden cost of self-hosting that devs underestimate before they've done it. None of these are blockers but they're real:

Comparison vs alternatives

Honest snapshot of the diarization landscape as of June 2026. Verify benchmark numbers against each tool's current docs before committing.

Open-source vs commercial split. The open-source side is dominated by pyannote with NeMo and SpeechBrain as alternatives. The commercial side is dominated by Deepgram and AssemblyAI for API-first usage. SaaS pipelines (VexaScribe and similar) abstract the diarization detail entirely — VexaScribe specifically runs Whisper + pyannote internally, so the “hosted” option for end-users is functionally the same stack described on this page, just operated by us instead of by you.

Where VexaScribe fits — we run Whisper + pyannote

Honest framing: VexaScribe's production pipeline is Whisper Large-v3 + pyannote 3.1, the same stack described on this page. We wrote this guide because we operate it daily — the gotchas around HF gated access, GPU provisioning, model load times, and version churn are things we've actually hit and solved, not theoretical. If you decide to operate the stack yourself, this page is what we wish we'd had when we started. If you decide you'd rather use someone else's operated version, we're an option — the user-facing output is what you'd build with Whisper + pyannote (transcription with speaker labels, TXT/DOCX/SRT/VTT/JSON exports), at $2-$20/month subscriptions.

When self-hosted pyannote is still the right answer: data sovereignty requirements (audio cannot leave your infra), research that needs reproducibility or custom training, very high volumes where a dedicated GPU fleet amortizes ops cost, or anything regulated where the chain of custody matters. We run the same stack and recommend self-hosting in those cases, not us.

When a hosted Whisper + pyannote pipeline (ours or similar) makes sense: non-technical colleagues need transcripts and you don't want to build a custom UI; small projects where building the pipeline isn't worth the engineering time; bridge solution while you scope a self-hosted build; you've evaluated pyannote and decided the ops overhead isn't worth it for your volume. For these, our hosted pipeline exists — same models, less maintenance.

More relevant: our honest diarization tools comparison, our transcription API comparison covering Deepgram, AssemblyAI, Speechmatics, and similar dev-focused services.

Methodology & disclosure

Sources: pyannote.audio repository at github.com/pyannote/pyannote-audio (MIT license; maintained by Hervé Bredin et al.). Current model card at huggingface.co/pyannote/speaker-diarization-3.1. Powerset architecture from Bredin, “Powerset multi-class cross entropy loss for neural speaker diarization” (arXiv:2210.13513). Whisper paper at arXiv:2212.04356. WhisperX at github.com/m-bain/whisperX. faster-whisper at github.com/SYSTRAN/faster-whisper. Deepgram and AssemblyAI pricing from their respective public pricing pages as of June 2026.

Benchmark caveats: DER numbers reported here are from the published speaker-diarization-3.1 model card and may shift slightly as the model is updated. For production planning, verify against the current model card and consider running your own benchmark on representative audio from your workload. Commercial vendor benchmark claims (Deepgram, AssemblyAI “1-3 DER better”) are based on each vendor's own published benchmarks; we have not independently reproduced these and recommend treating cross-vendor benchmark comparisons with appropriate skepticism.

Disclosure: This page is published by VexaScribe, a hosted transcription SaaS. Our production pipeline is Whisper Large-v3 + pyannote 3.1 — we operate the exact stack described on this page. That gives the guide credibility (the gotchas are ones we've actually hit) but it also gives us a commercial interest worth surfacing: if you decide running pyannote yourself isn't worth it, we'd like you to consider our hosted version. We've tried to keep the page honest on when self-hosting is genuinely the right answer (data sovereignty, research, very high volume) — those readers should stay with pyannote and not us. The recommendation rules in the “when to use pyannote vs hosted” section are written from that honest tradeoff, not from a goal of converting every reader.

Editorial standards: See our editorial standards.