pyannote developer on retainer: speaker diarization Pipeline.from_pretrained('pyannote/speaker-diarization-3.0') without num_speakers on 60-minute podcast episodes — ECAPA embedding drift across 30-minute monologue exceeds agglomerative clustering threshold — 2-speaker podcast inflated to 4–6 speaker IDs per episode — DER 0.31 vs expected 0.10 — 3 weeks — pyannote developer on monthly retainer

pyannote.audio 3.0.x • Python 3.11 • PyTorch 2.1.x • speaker diarization on 50-hour podcast corpus (2-3 speakers per episode, 30-60 min episodes) • Pipeline.from_pretrained('pyannote/speaker-diarization-3.0') without num_speakers • host delivers 30-minute uninterrupted monologue during interview segment • ECAPA-TDNN embeddings drift in speaker space over long monologue • agglomerative clustering threshold calibrated on meeting-room audio (frequent speaker interleaving) not podcast-format audio (long monologues) • same physical speaker split into 2-3 cluster IDs • speaker count inflated 2→4-6 per episode • DER 0.31 (expected <0.10) • developer checks min_duration_on, min_duration_off, ECAPA embedding model for 3 weeks • fix: pipeline(audio_file, num_speakers=2) for known 2-speaker interview format • DER 0.31→0.08 • 1.5h retainer engagement

Westfield pyannote Speaker Ltd. was building an automated transcription and speaker attribution system for a 50-hour podcast corpus spanning 50 episodes of an interview-format show (host + single guest per episode, with rare three-party call-ins). Each episode ran 30 to 60 minutes. The developer used Pipeline.from_pretrained('pyannote/speaker-diarization-3.0') and ran inference on each episode’s audio file. After processing 50 episodes, evaluation against hand-labeled ground truth returned a DER (diarization error rate) of 0.31 — more than three times the expected sub-0.10 for a 2-speaker setup. Manual inspection of the diarization output showed that each episode was consistently attributed to 4-6 distinct speaker IDs when only 2 were present. After three weeks of investigating min_duration_on, min_duration_off, ECAPA embedding model selection, and audio preprocessing, the team engaged a pyannote retainer developer. The pipeline’s agglomerative clustering was not constrained with num_speakers=2, and its default threshold — calibrated on meeting-room corpora where speakers interleave every 15-30 seconds — was splitting same-speaker segments that were separated by 30-minute silences into distinct speaker clusters due to embedding drift.

The diarization pipeline at the time of the engagement:

# Westfield pyannote Speaker Ltd. — podcast speaker diarization
# pyannote.audio 3.0.x, Python 3.11, PyTorch 2.1.x

from pyannote.audio import Pipeline
from pyannote.metrics.diarization import DiarizationErrorRate
import torch

# Load the pre-trained speaker diarization pipeline
pipeline = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-3.0",
    use_auth_token="hf_...",   # HuggingFace token
)
pipeline.to(torch.device("cuda"))

# ← Problem: pipeline called without num_speakers.
#
# pyannote/speaker-diarization-3.0 uses a two-stage approach:
#
# Stage 1 — Segmentation + Embedding:
#   - Voice Activity Detection (VAD) identifies speech segments
#   - ECAPA-TDNN extracts 192-dim speaker embeddings over 10-second sliding
#     windows with 1-second stride across each detected speech segment
#   - Embeddings represent the "average voice" of whoever spoke in that window
#
# Stage 2 — Agglomerative Hierarchical Clustering (AHC):
#   - Computes cosine similarity between all pairs of 10-second embeddings
#   - Merges clusters greedily until stopping criterion is met
#   - Stopping criterion: threshold = 0.715 (default for speaker-diarization-3.0)
#     This threshold was optimized on VoxConverse + AMI + AISHELL4 datasets
#   - VoxConverse: news/debate recordings, avg turn duration ~8-20 seconds
#   - AMI: 4-person meeting rooms, avg turn duration ~12-25 seconds
#   - Neither dataset has single-speaker monologues exceeding 10 minutes
#
# For a 60-minute podcast episode:
#   - Speaker A (host): delivers 25 minutes of opening monologue (minutes 0-25)
#   - Speaker B (guest): answers questions for 20 minutes (minutes 25-45)
#   - Speaker A (host): closing segment, 15 minutes (minutes 45-60)
#
# The 25-minute opening monologue produces ~1,500 ECAPA embeddings (10s windows,
# 1s stride over 1,500 seconds). Over 25 minutes of speech, the host's voice
# characteristics gradually shift:
#   - Minutes 0-5: warm-up, slightly higher pitch, faster rate (2.3 syllables/sec)
#   - Minutes 10-15: peak engagement, lower pitch, authoritative tone
#   - Minutes 20-25: topic wrap-up, lower energy, slightly slower rate (1.9 syllables/sec)
# ECAPA-TDNN captures prosodic variation as well as speaker identity.
# The cosine distance between embedding at t=0 and embedding at t=25min can
# exceed 0.20-0.30 for a single speaker who has shifted vocal register.
#
# When AHC compares the opening monologue embeddings (t=0 to t=5min) against
# the closing segment embeddings (t=45 to t=60min), cosine distance > 0.715
# threshold causes them to remain in separate clusters — same physical speaker,
# two cluster IDs. This repeats as the speaker traverses multiple sub-segments.
#
# Result: 60-minute 2-speaker episode → 5-6 speaker IDs in diarization output

def diarize_episode(audio_path):
    # ← Missing: num_speakers=2 for known 2-speaker interview format
    diarization = pipeline(audio_path)   # unconstrained clustering
    return diarization

# Batch process 50 episodes
results = []
for episode_path in episode_paths:
    diarization = diarize_episode(episode_path)
    num_speakers_found = len(set(diarization.labels()))
    results.append({
        'path': episode_path,
        'speakers_found': num_speakers_found,
    })
    print(f"  {episode_path}: {num_speakers_found} speakers found (expected 2)")

# Output sample:
#   ep001.wav: 5 speakers found (expected 2)
#   ep002.wav: 4 speakers found (expected 2)
#   ep003.wav: 6 speakers found (expected 2)
#   ep004.wav: 4 speakers found (expected 2)
#   ep005.wav: 5 speakers found (expected 2)
# Average speakers found: 4.7 (expected: 2.0)

# DER evaluation against ground truth:
metric = DiarizationErrorRate()
for hyp, ref in zip(hypotheses, references):
    metric(ref, hyp, uem=uem)
print(f"DER: {abs(metric):.3f}")  # DER: 0.313
# Expected: DER < 0.100 for 2-speaker podcast with clean audio

Why ECAPA speaker embeddings drift over long monologues and what agglomerative clustering does with the drift

ECAPA-TDNN (Emphasized Channel Attention, Propagation and Aggregation) encodes a speaker’s voice characteristics into a 192-dimensional embedding by processing an audio segment and producing a single vector that captures the speaker’s spectral, prosodic, and temporal characteristics. For a 10-second sliding window, the embedding reflects the speaker’s voice during those specific 10 seconds — including momentary pitch level, speaking rate, vocal energy, and spectral envelope. These characteristics are not perfectly stable for a single speaker across a 25-minute monologue: a speaker warming up at the start of an interview, shifting to a lower authoritative register while making a key argument, and then slowing down for a closing wrap-up will produce ECAPA embeddings that drift in the 192-dimensional space by cosine distances of 0.15 to 0.35, even though a human listener would immediately recognize these as the same voice.

The agglomerative hierarchical clustering (AHC) algorithm used by pyannote/speaker-diarization-3.0 computes cosine similarities between all pairs of 10-second window embeddings and merges the most similar pairs first, stopping when the inter-cluster distance exceeds the threshold of 0.715. This threshold was calibrated on corpora where speaker segments are rarely longer than 2-3 minutes — meeting participants take short turns, and the same speaker’s embeddings from two different 2-minute turns are close in embedding space because the speaker hasn’t drifted much in the intervening few minutes. When the same threshold is applied to 25-minute monologues, the drift between the speaker’s early and late embeddings exceeds 0.715, causing the clustering to treat the same physical speaker as multiple distinct speakers.

# Diagnostic: measure ECAPA embedding drift across a long monologue
# pyannote.audio 3.0.x, Python 3.11

from pyannote.audio import Pipeline, Inference
from pyannote.audio.pipelines.utils import get_model
import torch
import numpy as np

# Load the ECAPA-TDNN embedding model
embedding_model = Inference(
    "pyannote/embedding",
    window="whole",
    use_auth_token="hf_...",
)

# Step 1: Extract embeddings at 2-minute intervals throughout host monologue
# (minutes 0 to 25 = 1500 seconds)
episode_path = "ep001.wav"
timestamps = list(range(0, 1500, 120))   # every 2 minutes

embeddings = []
for t_start in timestamps:
    # Extract 10-second window at each timestamp
    from pyannote.core import Segment
    segment = Segment(t_start, t_start + 10)
    emb = embedding_model.crop(episode_path, segment)
    embeddings.append(emb.flatten())

embeddings = np.array(embeddings)   # shape: (13, 192)

# Step 2: Compute pairwise cosine distances
from scipy.spatial.distance import cdist
distances = cdist(embeddings, embeddings, metric='cosine')

print("Embedding drift in 25-minute host monologue:")
print(f"  t=0min vs t=5min:  cosine_dist = {distances[0,2]:.3f}")
print(f"  t=0min vs t=10min: cosine_dist = {distances[0,5]:.3f}")
print(f"  t=0min vs t=15min: cosine_dist = {distances[0,7]:.3f}")
print(f"  t=0min vs t=20min: cosine_dist = {distances[0,10]:.3f}")
print(f"  t=0min vs t=25min: cosine_dist = {distances[0,12]:.3f}")
# Output (typical for a 60-min podcast host):
#   t=0min vs t=5min:  cosine_dist = 0.063
#   t=0min vs t=10min: cosine_dist = 0.142
#   t=0min vs t=15min: cosine_dist = 0.218
#   t=0min vs t=20min: cosine_dist = 0.271
#   t=0min vs t=25min: cosine_dist = 0.314
#
# AHC stopping threshold: 0.715
# But AHC operates bottom-up: adjacent 10-sec windows are very close (0.03-0.08)
# and merge first. The accumulated drift between early and late clusters exceeds
# the AHC single-linkage or average-linkage threshold partway through the monologue.
# Depending on linkage criterion and episode characteristics, this split point
# occurs when the drift between the two emerging sub-clusters exceeds the threshold.

# Step 3: check how many speaker IDs the pipeline assigns to this episode
diarization = pipeline(episode_path)
print(f"\nSpeaker IDs for ep001.wav: {sorted(diarization.labels())}")
# Output: ['SPEAKER_00', 'SPEAKER_01', 'SPEAKER_02', 'SPEAKER_03', 'SPEAKER_04']
# 5 speaker IDs for a 2-speaker episode.
# SPEAKER_00, SPEAKER_02, SPEAKER_04 are all segments of the same host.
# SPEAKER_01, SPEAKER_03 are segments of the guest.

Fix: pipeline(audio_file, num_speakers=2) constrains AHC to exactly 2 clusters

The fix is to pass num_speakers=2 (or the known speaker count) to the pipeline call when processing interview-format audio where the number of participants is known in advance. When num_speakers is set, pyannote/speaker-diarization-3.0 bypasses the automatic threshold-based stopping criterion and instead forces AHC to merge all clusters until exactly num_speakers remain. This overrides the drift-induced premature stopping: even if the host’s embeddings at t=0 and t=25min have cosine distance 0.31, the AHC algorithm must still merge them (along with all other clusters of this speaker) into a single final cluster, since the target is 2 total. DER recovers from 0.31 to 0.08 for the podcast corpus.

# Fix: pass num_speakers=2 for known 2-speaker interview format
# pyannote.audio 3.0.x, Python 3.11, PyTorch 2.1.x

from pyannote.audio import Pipeline
from pyannote.metrics.diarization import DiarizationErrorRate
import torch

pipeline = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-3.0",
    use_auth_token="hf_...",
)
pipeline.to(torch.device("cuda"))

def diarize_episode_fixed(audio_path, num_speakers=None):
    # ← FIX: pass num_speakers when speaker count is known
    # For interview-format podcasts: num_speakers=2
    # For panel discussions: num_speakers=3 or 4 if known
    # For unconstrained recordings: omit num_speakers (auto-detect)
    if num_speakers is not None:
        diarization = pipeline(audio_path, num_speakers=num_speakers)
    else:
        diarization = pipeline(audio_path)
    return diarization

# For interview podcast corpus: use num_speakers=2
for episode_path in episode_paths:
    diarization = diarize_episode_fixed(episode_path, num_speakers=2)
    num_speakers_found = len(set(diarization.labels()))
    print(f"  {episode_path}: {num_speakers_found} speakers found (expected 2)")

# Output after fix:
#   ep001.wav: 2 speakers found (expected 2)
#   ep002.wav: 2 speakers found (expected 2)
#   ep003.wav: 2 speakers found (expected 2)
#   ep004.wav: 2 speakers found (expected 2)
#   ep005.wav: 2 speakers found (expected 2)

# DER evaluation after fix:
metric_fixed = DiarizationErrorRate()
for hyp, ref in zip(hypotheses_fixed, references):
    metric_fixed(ref, hyp, uem=uem)
print(f"DER after fix: {abs(metric_fixed):.3f}")
# DER after fix: 0.078   ← down from 0.313

# Alternative: if speaker count varies by episode (some have call-ins):
# Option A: query episode metadata for expected speaker count
# Option B: use min_speakers / max_speakers for a soft constraint:
diarization_soft = pipeline(
    audio_path,
    min_speakers=2,
    max_speakers=3,   # allow call-ins but don't exceed 3
)

# Option C: for a mix of 2-speaker and 3-speaker episodes, classify first:
# A simple heuristic: if num_speakers_unconstrained > 4 → almost certainly a
# 2-speaker episode with drift. If num_speakers_unconstrained = 3-4 → might be
# a genuine 3-speaker call-in. Use metadata (episode notes, description) to
# determine the correct num_speakers per episode rather than a fixed global value.

# ---- Hour breakdown ----
# Triage:     0.25h — DER 0.31; computed per-episode speaker counts;
#             all episodes showing 4-6 IDs for 2-speaker format;
#             no audio quality issues (SNR, clipping, encoding artifacts)
# Root cause: 0.5h — Pipeline without num_speakers; AHC threshold 0.715
#             calibrated on meeting-room corpora (short turns); 25-min monologue
#             causes embedding drift > threshold; same speaker split into 2-3 IDs
# Fix:        0.5h — pipeline(audio, num_speakers=2); forces AHC to merge all
#             same-speaker clusters regardless of drift
# Validation: 0.25h — DER 0.31→0.08; per-episode speaker count 4.7→2.0; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →