Who is talking,
and when?

Pick a stage to see what it computes, the shape it hands on, and where it runs. Stages 1 to 9 turn sound into vectors and carry enough parallel arithmetic to reward a GPU. Stages 10 to 16 turn vectors into people, using small matrices and sequential decisions, and they stay on the CPU on purpose.

A mono 16 kHz file is a list of air-pressure numbers, 16,000 samples per second. Amplitude over time is all it holds. Nothing in those numbers separates one voice from another, and nothing marks where a turn begins.

The segmentation model takes a fixed 10-second window. speakrs slides that window across the file with a step that depends on the execution mode. Smaller step, more overlap, more model calls. A neural window is least reliable at its edges, so the overlap buys accuracy back there. It pays a second time in stage 5, where speakrs averages what several windows say about how many people are talking.

For one window the model returns [589, 7]: 589 output frames, each with seven class scores. Three convolution blocks reduce 160,000 samples to those 589 frames, four LSTM layers walk them in order, and three linear layers produce the classes. The recurrence is why this stage does not saturate a GPU as cleanly as the embedding network does — h[t] depends on h[t-1], so time cannot be parallelized the way channels can.

The exported graph ends in LogSoftmax, so those seven numbers are log-probabilities rather than raw logits. The next stage takes an argmax and LogSoftmax is monotonic, so the decode is identical either way, and the Rust signature calls them logits.

The seven follows from the slot count. speakrs builds the class list at run time from every subset of three local speakers with at most two active at once: one empty set, three singles, three pairs.

Because windows overlap, each output frame is covered by several of them. speakrs sums the decoded activity across windows and rounds, giving one integer per frame: how many people are talking right now. That number is used at the very end, to decide how many speakers a frame is allowed to have.

This averaging is also why quantizing the segmentation model is safe in practice. A small error in one window is outvoted by its neighbors. Measured on this pipeline, W8A16 segmentation ran 41 % faster with no measured DER change, while the same treatment applied to the embedding model regressed DER.

The asymmetry is the interesting half. Segmentation output gets voted on. Embedding output does not: a speaker vector is consumed once, by a distance, and nothing downstream can outvote a bad one. That is the working explanation for why the same quantization is free in one model and expensive in the other. It is a hypothesis the project is still testing, not a result.

To decide that two anonymous slots are the same person, speakrs needs a representation of the voice, not the words. That means leaving the waveform behind.

The result is [998, 80] per chunk: 998 feature frames, 80 mel bands. Read it as a small grayscale image, time across, frequency up, energy as brightness.

CUDA has a sharp edge here. The exported embedding graph carries one DFT node, and in ONNX Runtime 1.24.2 that node is the only operator in the graph with no CUDA registration, so it lands on the CPU. The filterbank is cheap; the break it puts in the GPU graph is not. The CoreML export already sidesteps this by using the matrix form.

The window destroys the edges of each frame — w[0] is exactly zero, so sample 0 contributes nothing. Frames therefore overlap: at a 25 ms frame and a 16.875 ms hop, what gets attenuated at one frame's edge sits near the centre of the next.

Each of the three local slots gets its own activity mask over time. The mask multiplies the filterbank row-wise: if slot A is silent at frame t, that whole 80-band row contributes nothing to A's embedding. One filterbank, three masks, three different inputs to the same network.

Not every mask is worth embedding. If Alice and Bob overlap, their spectra are mixed and the resulting vector is a blend of two people. speakrs therefore keeps an embedding for clustering only when at least 20 % of the chunk's frames have exactly one active speaker. Slots whose total activity falls under MIN_SPEAKER_ACTIVITY = 10.0 are skipped before inference runs at all.

The embedding network is a WeSpeaker ResNet34. Small kernels slide across the time–frequency image; early layers find frequency edges and harmonics, later layers combine them into something speaker-specific. Each block computes y = x + F(x), the same residual shape a transformer uses around attention and MLP, and for the same reason: each block learns a correction, not a replacement.

After the convolutions, masked statistics pooling turns a variable amount of speech into a fixed vector: a mask-weighted mean and standard deviation per channel, concatenated, then one matrix multiply down to 256 numbers.

Finally the vector is L2-normalized. Direction carries the identity; length does not. After normalization, cosine similarity is a plain dot product.

Now there is a matrix E of shape [N, 256]: N reliable local voice observations, each from one slot in one window. Nothing in it says who is who. Clustering decides that.

Every embedding starts as its own cluster. speakrs L2-normalizes, computes all pairwise Euclidean distances, then merges the closest pair repeatedly using centroid linkage, cutting the dendrogram at a distance of 0.6. On unit vectors that Euclidean threshold has an exact cosine reading, shown live below.

The pairwise distance matrix is E · Eᵀ and would love a GPU. The merge loop would not: each step depends on the last, and with a few thousand observations the whole stage is a rounding error in the total runtime.

Before refining, speakrs projects the 256-dimensional embeddings down to 128 PLDA dimensions: a learned rotation and scaling that stretches the directions which historically separate speakers and shrinks the ones that mostly encode channel and noise.

Loading those parameters involves matrix products, an inversion, and a generalized eigenvalue problem. Those amplify precision error, so speakrs does the whole thing in f64. This is the opposite of the convolution case, where FP16 error averages out across thousands of accumulations.

AHC says "observation 2 is Alice". VBx says "observation 2 is 75 % Alice, 25 % Bob", and iterates: build a soft speaker model from the current responsibilities, re-score every observation against every model, softmax the scores, repeat. speakrs runs a GMM variant with no HMM transitions, fa 0.07, fb 0.8, and a maximum of 20 iterations, stopping early when the ELBO improves by less than 1e-4.

The last three stages are bookkeeping, and all of it is CPU work. Weighted averages of the original 256-dimensional embeddings give one centroid per surviving speaker. Every local slot is then scored by cosine against every centroid, with a constraint: two slots that are active at the same time may not be assigned to the same person. Then the local activity is added back onto a global timeline, and each frame keeps the top k speakers, where k is the count that overlap-averaging produced back in stage 5.

What is left is list processing: turn runs of active frames into time ranges, drop flickers shorter than the minimum duration, bridge short gaps, merge adjacent ranges for the same speaker, write RTTM. In the default configuration the hysteresis thresholds are onset = offset = 0.5 with no minimum-duration filtering; the Fast modes raise both minimums to 3 frames because a 2 s step produces more single-frame flicker.

Strip the prose and the pipeline is a ladder of tensors. Each rung is a change of representation, and the interesting rungs are where the dimensionality collapses.

Almost every stage here is matrix math, so asking whether a stage is matrix math settles nothing. The question that settles it is whether a stage's matrices are large enough for the arithmetic to outweigh the kernel launch and the transfer.

Diarization error rate is missed speech plus false alarm plus speaker confusion, as a fraction of reference speech. Lower is better. These are collar-0 results against the reference pyannote pipeline running the same models, measured on an Apple M4 Pro.

Accuracy is a tie, and the component columns say what kind of tie. Across all eight datasets speakrs and pyannote agree on missed speech and on false alarm to within 0.2 points. Every DER difference between them, in either direction, is speaker confusion. The two agree on how much speech there is and where it sits. They disagree only about who it belongs to.

Which makes the CUDA numbers awkward. On CUDA, speakrs reproduces pyannote exactly on AMI IHM and Earnings-21 — same DER, same missed, same false alarm, same confusion, to the tenth — and comes out 0.2 ahead on VoxConverse dev. The confusion that the CoreML build pays is not in the algorithm, because the same algorithm does not pay it on CUDA. It is somewhere in the backend, and nobody has found it yet.

AVA-AVD is worth staring at for a different reason. Every implementation in the table scores between 45 and 51. On in-the-wild film audio this whole family of pipelines fails, and arguing about a point and a half between them is not a useful thing to do.

That leaves throughput, which is not a tie at all.

Same models, same decode, same clustering algorithm, and 21 to 36 times the throughput of the Python pipeline on the same machine — 50 times on ICSI in Fast mode. The gap comes from the choices above: batch the windows, keep the filterbank next to the embedding model, run the backbone once per chunk instead of once per window, and leave the small sequential math on the CPU.

Throughput is finished work. The confusion column is not.

All numbers on this page were read from the speakrs source tree and its benchmark tables on 2026-09-01. Benchmarks used collar 0 ms and pyannote batch size 32.

Written for someone who knows how to program and how a matrix works, and who has read enough about language models to recognise a softmax. No machine-learning background assumed. Dotted underlines anywhere on the page open the matching entry.

Every constant, threshold, and measurement above has an anchor in the repository. The shape ladder is the one place with example values, and it marks N and K as data dependent where they appear.