smdesai/Nemotron-3-Diarization-CoreML

Nemotron 3 Diarization — Core ML

Core ML conversion of NVIDIA's Nemotron 3 Diarization streaming speaker-diarization model (up to eight speakers, 10 ms output frames) for iOS 18+ / macOS 15+. Converted from checkpoint revision a435e9867d79e789e90053f9b6d6834053af564a (Nemotron-3-Diarization.nemo, SHA-256 867c53f552998f772e5b5e5c082962ae85ee7ca5669c2bc17d7f615133d4e96d). No weights were retrained, quantized or pruned.

The model is split into two graphs that a host streaming loop calls once per chunk: a preencoder (Mel features → 80 ms frame embeddings) and an encoder/head (packed speaker cache + FIFO + chunk embeddings → speaker probabilities). The Arrival-Order Speaker Cache, FIFO and chunking logic of Streaming Sortformer run on the host and are not part of these graphs; see "Streaming loop" below.

Contents

Path	Description
low-latency/pipeline.json	Live: FP16 preencoder + FP16 BC1S encoder/head
low-latency/models/preencoder_fp16.mlpackage	Live preencoder
low-latency/models/encoder_head_fp16_bc1s.mlpackage	Encoder/head, Neural Engine layout (channel-first BC1S, pointwise Conv2d projections, per-head attention)
low-latency/frontend/	Mel frontend parameters, Hann window and Mel filterbank
low-latency/learned_silence_embedding.f32le	Learned silence embedding (512 × float32) used to pad the speaker cache
offline/pipeline.json	Published offline profile: FP32 preencoder + FP16 BC1S encoder/head
offline/models/	Offline preencoder_fp32.mlpackage and encoder_head_fp16_bc1s.mlpackage
offline/frontend/, offline/learned_silence_embedding.f32le	Offline frontend and cache-padding assets
SHA256SUMS	SHA-256 of every file

.f32le files are raw little-endian float32 in C order. The .mlpackage directories load with MLModel.compileModel(at:) / Xcode. Ship the compiled .mlmodelc: the BC1S encoder needs a ~35 s Neural Engine compile on first load, which the system caches for a compiled model at a stable path.

Streaming profile

Graphs under low-latency/ are fixed-shape for the model card's "Low latency" (1.04 s) configuration, applied synchronously (cache update inside each step):

Parameter	Value
Chunk length	9 encoder frames (0.72 s)
Left / right context	1 / 4 encoder frames (0.08 / 0.32 s)
Speaker cache / FIFO	264 / 264 frames
Speaker-cache update period	222 frames
Encoder frame	8 Mel frames (80 ms)
Output resolution	10 ms (8× upsampled), 8 speakers

For file analysis, load offline/pipeline.json. Its separate graphs use the published offline profile: central 340, left 1, right 40, cache 264, FIFO 40, update period 300, and packed capacity 685. Preencoder input is [1,3048,128], output [1,381,512]; encoder input is [1,685,512], native output [1,5480,8], and cache output [1,685,8]. Names and input/output types match the live interface below. Resolve paths relative to the selected manifest; never use the live shapes with offline graphs.

Only these two production pipelines are included. Alternate precision manifests, unused model variants and conversion reports are omitted. Keep each selected model package intact; SHA256SUMS covers both profiles.

Frontend

16 kHz mono float32 audio → pre-emphasis 0.97 → centered STFT (FFT 512, Hann window 400, hop 160, constant zero padding) → power spectrum → 128 Slaney Mel bins → log(x + 2^-24). No dither, no feature normalization. floor(samples / 160) valid frames. The window and filterbank are the checkpoint's own buffers, which NVIDIA stores in bfloat16; they differ slightly from freshly computed float32 values. Project each frame's power spectrum onto the filterbank separately (a batched matrix product changes the float rounding and makes long streams diverge).

Graph interfaces

Preencoder

Name	Shape	Type
in mel_features	[1, 112, 128] time-major (left 8 + central 72 + right 32 frames, zero-padded)	float32
in mel_length	[1] valid frames	int32
out chunk_embeddings	[1, 14, 512]	float32
out chunk_embedding_length	[1] = ceil(mel_length / 8)	int32

Live encoder/head

Name	Shape	Type
in packed_embeddings	[1, 542, 512]: speaker cache, then FIFO, then chunk embeddings, zero-padded	float32
in packed_length	[1] valid packed rows	int32
out native_probabilities	[1, 4336, 8] speaker activity at 10 ms (542 × 8 rows)	float32
out cache_probabilities	[1, 542, 8] per packed row, for cache/FIFO updates	float32

Probabilities are sigmoid outputs; the model card's default decision threshold is 0.5.

Streaming loop

Per chunk: slice left/central/right Mel frames, run the preencoder, pack [speaker cache | FIFO | chunk embeddings] up to 542 rows, run the encoder/head, emit the central chunk's rows of native_probabilities, then update the FIFO and speaker cache from cache_probabilities following NVIDIA NeMo's Streaming Sortformer (Arrival-Order Speaker Cache compression every 222 frames, learned silence embedding for padding). On the final call, flush the remaining chunks with zero right context and emit only rows up to the true end of the audio.

Validation (live profile; includes reference variants not distributed here)

Against the original NeMo FP32 model with the same streaming profile:

• FP32: strict agreement (absolute 2e-6 + relative 2e-5) on every output of 305 teacher-forced checks, a 30 s closed-loop stream (Mac CPU and iPhone 17 Pro) and a 120 s stress stream crossing five cache compressions (Mac CPU). Identical segments; 30 s sample DER 4.52% (zero collar, overlap included).
• FP16 (both encoder layouts): FP16 rounding exceeds the strict FP32 tolerance but the 30 s DER is identical to FP32 (4.52%), with 4 of 3,000 frames changing a speaker decision on iPhone.
• BC1S encoder: outputs byte-identical to the FP16 standard-layout encoder on the iPhone Neural Engine for 30 s, 376 s and 36 min recordings.

Speed, iPhone 17 Pro (iOS 27.2), CPU + Neural Engine, 36 min recording: BC1S encoder 102.9 s (20.8× real time, 34 ms per encoder call) vs standard FP16 133.6 s (16.0×, 44 ms); BC1S peak memory is about 40–60 MB higher. An Instruments trace showed every BC1S encoder call on the Neural Engine.

These are checks against the reference model on a few recordings, not a new accuracy benchmark; see NVIDIA's model card for accuracy.

Offline validation

On iPhone 17 Pro, the offline pipeline completed 30 s and 98.64 s recordings with 3,000 and 9,864 output frames. Speaker activity differed from the released FP32 offline reference on one 10 ms frame in each recording. The 30 s sample DER was 2.3819% (zero collar, overlap included). Mixed precision does not pass the strict FP32 probability tolerance. These sample checks do not establish corpus-wide accuracy. First model loading took about 43 s; subsequent loading took less than a second.

License

Use is governed by the OpenMDW License Agreement, version 1.1 (see LICENSE), the license of the original model. See NOTICE for origin.