A one-hour meeting, eight speakers: NVIDIA’s open-source model can tell who is talking

NVIDIA today released an open-weight speech model: Nemotron 3.5 Transcribe. The 30B-A4B in the model name corresponds to about 34B total parameters, with about 4.3B activated per token. It can be self-hosted on a single H100 80GB, and its job is to turn a recording into useful text.

Here’s what it does in one go:

• 40-language transcription
• Translation between English and 31 languages
• Up to 8 speakers, with speaker labels supported in 13 languages
• Single recordings up to 1 hour, 65,536-token context
• You can feed the model names and technical terms as context guidance
• You can ask questions about a recording, but audio QA was trained only in English

One model replaces several steps in what used to be a pipeline. Transcription, speaker separation, translation, and QA used to require separate models or services; now one inference pass outputs speaker labels and timestamps.

Place on the leaderboards

NVIDIA says: on the Hugging Face English Open ASR leaderboard, Nemotron 3.5 Transcribe ranks first among open-weight models, with an average WER of 4.71% across nine datasets. It scores 4.62% on the multilingual leaderboard and 10.98% on the long-audio leaderboard.

The speaker diarization line is more worth watching. VoiceArena tested 14 systems with cpWER, where lower is better. Nemotron 3.5 Transcribe scores 32.4%, ranking first, slightly lower than NVIDIA’s own Multitalker Parakeet plus Nemotron 3 Diarization combination.

On David AI’s 21-language speaker diarization leaderboard, it also has the lowest DER, 18.26%, followed closely by the dedicated Nemotron 3 Diarization model.

In Artificial Analysis’s independent evaluation, it ties for first among open-weight models with another open-source model, with AA-WER 2.8%, while using only about one-sixth of the other model’s activated parameters.

The demo uses a 1940 film

NVIDIA chose a clip from His Girl Friday for the demo. This 1940 screwball comedy is known for extremely fast speech and lots of overlapping dialogue. Using it to show speaker labels and overlapping-speech handling makes the intent clear: don’t just look at single-speaker read-aloud in quiet conditions.

Architecture and speed

The model uses a hybrid Mamba-Transformer MoE architecture. On the audio side, it uses a Transformer ASR encoder plus a Sortformer speaker encoder; the language backbone is Nemotron 3.5 Lightning 30B-A3B. Speaker information enters the language model at the encoding stage, and labels are generated directly, without needing a later alignment round.

On the serving side, it uses vLLM 0.32.0 plus NeMo Speech, with MTP speculative decoding. NVIDIA’s numbers: on RTX PRO 6000 Blackwell, with NVFP4 plus DFlash2 speculative decoding, it processes 2,238 seconds of audio per second, returning a transcript for a 30-second recording in 0.13 seconds. However, these optimizations are not yet in the open-source version; the open-source vLLM integration is roughly twice as slow, and it is positioned for model development use.

The local hardware bar is not low. BF16 recommends one H100 80GB. RTX 5090 can only use the NVFP4 quantized version. Some commenters replied that this thing is “fucking huge,” and others asked when it could be quantized down to 6 to 8GB of memory. NVIDIA says the quantized version and the NeMo-Speech.cpp local deployment tool will be released later.

Accompanying datasets and fine-tuning

NVIDIA also released the Granary v2 dataset list: 463,000 hours, 48 languages, 16 source corpora. Four reusable data-processing stages: audio preparation, transcription, task-label generation, and high-fidelity filtering. Developers can run their own recordings through the same NeMo Curator pipeline.

The medical-domain fine-tuning case is more concrete. On Eka Care’s benchmark of medical speech with Indian doctor accents, out-of-the-box WER is 30.8%. With a prompt specifying English and Latin script, it drops to 23.8%. Then, fine-tuning on about 153 hours of synthetic speech, 29 hours of accented English, and 63 hours of medical recordings reduced held-out speaker WER from 22.44% to 12.17%, a relative reduction of 45.8%. The whole adaptation process takes about 14 GPU hours on one GB300.

This is a complete path: open weights, an open data pipeline, open fine-tuning tutorials, and even a skill that lets a coding agent automatically run the fine-tuning workflow. It is not just dumping a checkpoint.

A few things worth checking first

A commenter, Max Petrusenko, raised a specific question: for an 8-speaker, 1-hour recording, diarization tends to have problems at audio chunk boundaries. The model processes audio internally in 30-second chunks; someone labeled speaker 4 earlier may come back as a new label in the next chunk. Label stability across boundaries is worth testing in practice.

Another selling point repeatedly mentioned is local deployment. In places like Germany, where there are compliance requirements for meeting recordings, keeping audio off your own machine is a procurement reason. This is also the anchor of NVIDIA’s open-source strategy: give you the weights, data, and toolchain, and let the model run on its own GPUs.

Audio QA was trained only in English; don’t expect it for other languages. It is also an offline model, so live-streaming scenarios need another model. The model card also reminds that speaker labels are only for distinguishing different people within the same recording and do not represent real identities.

Model, demo, and blog links:

• Model: nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B
• Demo: nemotron-transcribe
• Blog: Nemotron 3.5 Transcribe and Granary v2

Transcription accuracy has already been pushed to WER around 4%. Next, the comparison is who can stably label eight people in the same recording, work across languages, and also answer “what decisions were made in the meeting.” The difficulty of that is on a different order from pure transcription. NVIDIA puts four capabilities into one model, at the cost of a 34B body. For individual developers, when the quantized version arrives may matter more than leaderboard rank.