Verified July 2026

WhisperX — Whisper with Word-Level Timestamps and Diarization

WhisperX is an open-source Python library that combines faster-whisper (fast Whisper transcription via CTranslate2), a wav2vec2 model (forced word-level alignment), and pyannote 3.x (speaker diarization) in one pipeline. Published by Bain et al. at Interspeech 2023, it's the most complete self-hostable solution for producing a Whisper transcript with per-word timestamps AND speaker labels in one call.

Architecture — three-stage pipeline

WhisperX isn't a new model — it's an orchestrator that chains three existing models. That's the first thing to understand.

  1. 1

    Transcription — faster-whisper

    WhisperX first runs faster-whisper (CTranslate2 implementation of Whisper) to transcribe the audio into segments with segment-level timestamps. Same output as vanilla Whisper Large-v3, but faster thanks to the optimized CTranslate2 backend.

  2. 2

    Forced alignment — wav2vec2

    A language-specific pre-trained wav2vec2 model (facebook/wav2vec2-large-xlsr-53-english for English, -french for French, and so on) takes the audio and the transcript, then computes precisely where each word starts and ends. Result: per-word timestamps instead of per-segment.

  3. 3

    Diarization — pyannote 3.x

    pyannote.audio segments the audio by speaker (who spoke when) and assigns a speaker label to each word (SPEAKER_00, SPEAKER_01, …). Optional stage enabled with --diarize; requires a Hugging Face token to access the gated pyannote models.

Reference publication: Bain, M., Huh, J., Han, T., & Zisserman, A. (2023). WhisperX: Time-Accurate Speech Transcription of Long-Form Audio. Interspeech 2023 (arXiv:2303.00747). Repo: github.com/m-bain/whisperX.

WhisperX vs faster-whisper vs whisper.cpp

The three tools live in the same ecosystem (all Whisper-based) but each has a different job. Here's the honest comparison on 8 attributes.

AttributeWhisperXfaster-whisperwhisper.cpp
Per-word timestampsYes (wav2vec2 forced alignment)No (segment-level only)Partial (word_timestamps flag, less precise)
Speaker diarizationYes (pyannote 3.x integrated)No — add separatelyNo — add separately
Speed (GPU CUDA)Fast (uses faster-whisper)Fastest (no overhead)N/A (CPU-first)
VRAM (Large-v3)~4-6 GB (transcription + alignment + diarization)~2-4 GB (transcription only)0 (CPU) — RAM ~4 GB
CPU supportTechnically yes, slowYes with int8 quantizationOptimized for CPU (SIMD, Metal, CoreML)
Mobile / edgeNo (Python + CUDA)No (Python)Yes (iOS, Android, Raspberry Pi)
Hugging Face token requiredYes (pyannote gated) if --diarizeNoNo
Best-fit use caseVerbatim interviews, precise subtitles, multi-speaker podcastsBatch server transcription, API pipelinesMobile apps, on-device, edge, offline

Summary: WhisperX adds clear value when you need per-word timestamps or diarization. Otherwise, faster-whisper suffices and is lighter. For anything mobile / edge / CPU, whisper.cpp wins.

Installation

Prerequisites: Python 3.9+, PyTorch with CUDA (recommended for serious use), a Hugging Face account if you want diarization.

1. pip install

pip install whisperx

Installs faster-whisper, torch, transformers, pyannote.audio, and wav2vec2. Total ~2 GB.

2. Hugging Face token (for diarization)

Create an account at huggingface.co, then accept the terms on these two pages:

  • pyannote/segmentation-3.0
  • pyannote/speaker-diarization-3.1

Then create a user token at huggingface.co/settings/tokens and export it:

export HF_TOKEN=hf_xxxxxxxxxxxxxxx

3. Command-line usage

whisperx audio.mp3 \
  --model large-v3 \
  --language en \
  --diarize \
  --hf_token $HF_TOKEN \
  --output_format srt

4. Python API — output with per-word timestamps

import whisperx

device = "cuda"
audio_file = "interview.mp3"

# 1. Transcription
model = whisperx.load_model("large-v3", device, language="en")
audio = whisperx.load_audio(audio_file)
result = model.transcribe(audio, batch_size=16)

# 2. Word-level alignment
model_a, metadata = whisperx.load_align_model(
    language_code="en", device=device
)
result = whisperx.align(
    result["segments"], model_a, metadata, audio, device
)

# 3. Diarization
diarize_model = whisperx.DiarizationPipeline(
    use_auth_token="hf_xxxxx", device=device
)
diarize_segments = diarize_model(audio)
result = whisperx.assign_word_speakers(diarize_segments, result)

# result["segments"] now contains:
# [{ "start": 0.32, "end": 2.15, "text": "Welcome to the show today...",
#    "words": [{"word": "Welcome", "start": 0.32, "end": 0.71, "speaker": "SPEAKER_00"}, ...],
#    "speaker": "SPEAKER_00" }, ...]

Per-word timestamps — why it matters

Vanilla Whisper gives you segment-level timestamps (typically 5-30 seconds each). WhisperX drops to per-word precision via wav2vec2 alignment. Side-by-side comparison:

Vanilla Whisper (segment-level)

{
  "start": 4.20,
  "end": 9.85,
  "text": "Today we're going to talk about transcription"
}

WhisperX (per-word)

"words": [
  {"word": "Today", "start": 4.20, "end": 4.62},
  {"word": "we're", "start": 4.68, "end": 4.85},
  {"word": "going", "start": 4.89, "end": 5.21},
  {"word": "to", "start": 5.28, "end": 5.36},
  {"word": "talk", "start": 5.42, "end": 5.79},
  {"word": "about", "start": 5.85, "end": 6.10},
  {"word": "transcription", "start": 6.15, "end": 6.72}
]

Practical uses: SRT subtitles at 2-3 words per cue instead of blocks of 15 words; verbatim qualitative interviews with per-word timestamps for NVivo/ATLAS.ti; karaoke or word-by-word video editing effects; precise clip extraction for short-form video.

Accuracy inheritance

Important point: WhisperX does not improve transcription accuracy. It inherits from faster-whisper (so from Whisper Large-v3). On clean English audio, that means 93-95% word accuracy per the FLEURS benchmark. What WhisperX improves: temporal precision (per-word timestamps) and adds diarization.

For diarization, pyannote 3.x achieves a DER (Diarization Error Rate) around 11-19% on standard benchmarks — good but not perfect. Expect higher DER on audio with heavy speaker overlap, 4+ speakers, or unusual acoustic conditions.

See how accurate is Whisper for the detailed transcription benchmarks, and pyannote.audio for diarization detail.

Honest limits

Hugging Face token required for diarization

The pyannote models (segmentation-3.0, speaker-diarization-3.1) are gated. You must create an HF account, accept the terms on each model page, then get a user token. Friction step but legitimate — pyannote requires this to track usage.

Significant VRAM for Large-v3

~4-6 GB needed on GPU for Large-v3 with diarization enabled. RTX 3060 (12 GB) is a comfortable minimum; RTX 3060 8 GB works with reduced batch_size. On 4 GB GPUs, use the medium or small model.

Heavy Python dependencies

The full install (torch + faster-whisper + pyannote + transformers + wav2vec2) is around 2 GB. Not ideal for lightweight containers. Docker Compose recommended for isolation.

No mobile / edge version

WhisperX is Python + CUDA. For iOS, Android, Raspberry Pi, or edge, use whisper.cpp — WhisperX is incompatible.

Alignment fails on very noisy audio

wav2vec2 forced alignment assumes the transcribed text matches the audio. On heavily degraded audio (loud noise, overlapping voices), alignment can be wrong even if the transcript looks correct.

Diarization DER varies by language and conditions

pyannote has more training data in English than other languages. Expect higher DER on non-English audio, especially with speaker overlap. See /pyannote-audio for the detail.

Which tool for which job?

Honest decision tree between WhisperX, faster-whisper, whisper.cpp, and managed services.

JobRecommended toolWhy
Qualitative interview verbatim (NVivo, ATLAS.ti) with speaker identificationWhisperXPer-word timestamps + integrated diarization — output ready for qualitative analysis
Precise SRT subtitles for video editing (Premiere, DaVinci, CapCut)WhisperXPer-word timestamps prevent overly long subtitles
Multi-speaker podcast with speaker attributionWhisperXIntegrated pyannote diarization without extra code
Batch server transcription, single speaker, segment-level timestamps OKfaster-whisperLighter, no HF token, faster startup — WhisperX would be overkill
Mobile app (iOS/Android) with on-device transcriptionwhisper.cppWhisperX requires Python + CUDA — impossible on mobile. whisper.cpp compiles to a native binary
Raspberry Pi, edge, or CPU-only serverwhisper.cppCPU optimization (SIMD, GGUF quantization, Metal on Mac) — WhisperX unusable without GPU
Production API with SLA (uptime, guaranteed latency, monitoring)Managed serviceWhisperX / faster-whisper / whisper.cpp are DIY tools. For SLA, use a hosted service — see /transcribe-audio-to-text

Frequently asked questions

What is WhisperX?

WhisperX is an open-source Python library that combines three components in one pipeline: (1) faster-whisper for fast Whisper transcription via CTranslate2, (2) a wav2vec2 model for forced word-level alignment (replaces Whisper's segment-level timestamps with per-word precision), and (3) pyannote.audio 3.x for speaker diarization. Published by Max Bain et al. (Interspeech 2023, arXiv:2303.00747), maintained at github.com/m-bain/whisperX. It's the most complete self-hostable solution for producing a Whisper transcript with per-word timestamps AND speaker labels in one call.

WhisperX vs faster-whisper — which should I use?

faster-whisper if you only need fast transcription without diarization or word-level timestamps. It's lighter (no pyannote dependency, no Hugging Face token), faster to start, and covers most batch transcription workflows. WhisperX if you need precise per-word timestamps (subtitle editing, karaoke, forced lyric alignment) OR integrated speaker diarization (multi-speaker podcasts, interviews, meetings). WhisperX uses faster-whisper internally for the transcription step, so you're not choosing between them — WhisperX adds two extra stages on top.

WhisperX vs whisper.cpp — which should I use?

whisper.cpp if you target CPU, mobile (iOS, Android), edge, or Raspberry Pi. No Python dependencies, single C++ binary, supports Metal (macOS) and CoreML. WhisperX if you have a GPU (Nvidia CUDA), use Python, and need per-word timestamps + diarization. WhisperX requires a GPU for production use (Large-v3 on CPU is impractically slow), while whisper.cpp is optimized for CPU. Both have their place — a mobile app uses whisper.cpp, a Python server pipeline uses WhisperX.

How do I install WhisperX?

Three commands. (1) `pip install whisperx` — installs faster-whisper, torch, transformers, pyannote, and wav2vec2 dependencies (~2 GB total). (2) Create a Hugging Face account and accept the gated-model terms for pyannote/segmentation-3.0 and pyannote/speaker-diarization-3.1 (both are HF-gated) — get a user access token. (3) Export HF_TOKEN in your environment or pass it via the Python API. Then: `whisperx audio.mp3 --model large-v3 --diarize --hf_token YOUR_TOKEN`. Prerequisites: Python 3.9+, PyTorch with CUDA if using GPU (recommended). VRAM: Large-v3 needs ~4-6 GB.

What is forced alignment in WhisperX?

Forced alignment takes a known text (the Whisper transcript) and an audio file, then precisely computes where each word starts and ends in the audio. WhisperX uses a pre-trained wav2vec2 model per language (facebook/wav2vec2-large-xlsr-53-english for English, -french for French, etc.) for this step. It's different from transcription: the wav2vec2 model doesn't guess what's said — it aligns already-transcribed text with millisecond precision. Result: per-word timestamps suitable for subtitle editing, karaoke, and qualitative research verbatim.

How accurate is WhisperX?

WhisperX does not improve transcription accuracy — it inherits from faster-whisper (so from Whisper Large-v3: 93-95% word accuracy on clean English audio per the FLEURS benchmark, comparable on other Tier 1 languages). What WhisperX improves: temporal precision (per-word instead of per-segment) and adds diarization. For diarization, pyannote 3.x achieves a Diarization Error Rate (DER) around 11-19% on standard benchmarks — good but not perfect. Expect higher DER on audio with heavy speaker overlap, 4+ speakers, or unusual acoustic conditions.

Does WhisperX work on CPU?

Technically yes, but not in practice for production. Large-v3 models on CPU take 5-10× the audio duration to process (30 min of audio = 3-5 hours of processing), which is unusable for serious work. Smaller models (base, small) are usable on CPU with reduced accuracy. For pure CPU, whisper.cpp is the correct choice — it's optimized (SIMD, GGUF quantization, better latency). For WhisperX in production, an Nvidia GPU with at least 6 GB VRAM (RTX 3060 or better) is the recommended baseline.

Is the Hugging Face token mandatory?

Only if you want diarization. The pyannote models (segmentation-3.0 and speaker-diarization-3.1) are gated on Hugging Face — you must create an account, accept the terms on each model's page, and get a user token. If you don't use --diarize, the token isn't needed — WhisperX works with just faster-whisper + wav2vec2 alignment. The token step is friction but legitimate: pyannote requires this to track academic vs commercial usage. It's free.