One-Hour Recordings, 40 Languages: NVIDIA Open-Sources a Transcription Model, but Hold Off on Real-Time Captions

NVIDIA has released Nemotron 3.5 Transcribe with open weights, available in BF16, FP8, and NVFP4. It puts multilingual transcription, speaker attribution, translation, and audio Q&A into the same checkpoint, with day-0 serving through vLLM and NeMo Speech. Both the model card and the launch post point to the same positioning: offline recording transcription, with priority on long audio.

Let's start with the specs. Total parameters are 34B, with a MoE architecture, about 4.3B active parameters, including the audio encoder. Context length is 65,536 tokens, supporting recordings up to 1 hour long. Transcription covers 40 languages, 31 languages can be translated to and from English, and 13 languages support speaker attribution, labeling up to 8 speakers. NVIDIA recommends deploying on 1 H100 80GB.

The architecture diagram is worth a look too. On the audio side, there is a Transformer ASR encoder plus a Sortformer speaker encoder, followed by a CTC timestamp alignment head, then a hybrid Mamba-Transformer MoE language backbone from Nemotron 3.5 Lightning. In plain terms, speaker information and acoustic features go into the model together, so the generated transcript can indicate who is speaking instead of relying entirely on post-processing to bolt on labels.

On the leaderboard side, the vLLM launch post says English Open ASR ranks first among open-weight models with 5.33% WER; multilingual Open ASR averages 4.62% WER across 7 languages. The model card later lists a 9-public-set average of 4.71%, and a long-audio average of 10.98%. These two WER figures use different methodologies, so don't compare them interchangeably when citing them.

The efficiency part is more interesting. On Artificial Analysis, Nemotron 3.5 Transcribe achieves about 2.80% AA-WER with about 4B active parameters; Voxtral Small is 2.77% with 24B active parameters. Accuracy is close, while active parameters are about one-sixth. That comparison is more useful than simply shouting that it is number one.

Speaker diarization is the headline feature this time. On David AI's 21-language diarization leaderboard, it gets 18.26% overall DER and ranks first; in VoiceArena's 14-system comparison, it has 32.4% cpWER and also ranks first. NVIDIA shared a meeting recording example, having the model add speaker labels, timestamps, translation, and then find PII.

One judgment: a pretty leaderboard does not mean it can directly replace existing transcription pipelines. The model card states clearly that this is an offline model, not real-time streaming ASR. Processing internally in 30-second chunks does not equal a 30-second latency promise. Under the vLLM post, users asked about 8kHz telephone audio, crosstalk, diarization under overlapping speech, and the WER of NVFP4 versus BF16 on Indian English accents. The official model card does not give complete answers for these combinations. Production environments don't pose questions based on leaderboards.

Deployment also depends on version. The day-0 vLLM integration is in NeMo Speech, requires vLLM 0.32.0, OpenAI-compatible chat completions, and supports MTP speculative decoding. NVIDIA also notes that this open-source integration is aimed at model development and is expected to be about 2x slower than the optimized NIM implementation. 100 hours of audio takes under 3 minutes on a single RTX PRO 6000 Blackwell using the upcoming NVFP4 quantization, 30-second segments, and high concurrency; that optimization has not yet made it into the open-source version.

The fine-tuning pipeline is also released. The public Granary v2 manifest covers 463,062 hours, 48 languages, and 16 source corpora, with NeMo Speech fine-tuning tools. In the medical ASR case, full test set WER drops from 30.8% out of the box to 23.8% after changing the prompt, and 11.8% after fine-tuning. On held-out speakers, the best prompt baseline is 22.44% WER, and after fine-tuning it is 12.17%, a relative reduction of 45.8%. The entire adaptation takes about 14 GPU-hours on a single GB300.

Limitations are clear too. Mandarin Chinese is listed as Supported in the support matrix, but may still need fine-tuning; audio understanding, reasoning, and voice instruction following are only evaluated in English, so don't assume other languages are equally robust. Speaker labels only distinguish different speakers within a recording and do not establish real identities. Timestamps require enabling CTC alignment; the DFlash2 speculative decoding path does not support timestamps.

If you want to try it, Hugging Face has an online demo, and model weights, Granary v2, the vLLM deployment guide, and the fine-tuning tutorial are all provided. Baseten, DeepInfra, and Simplismart also provide hosted endpoints. SGLang-Omni can transcribe entire files, but timestamps, speaker output, translation, and speculative decoding are not in that integration.

Links here:

• Model weights: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B
• Online demo: https://huggingface.co/spaces/nvidia/nemotron-transcribe
• Release blog: https://huggingface.co/blog/nvidia/nemotron-transcribe-granary
• vLLM deployment guide: https://github.com/NVIDIA-NeMo/Speech/blob/main/docs/source/speechlm2/vllm_serving.rst
• Granary v2: https://huggingface.co/datasets/nvidia/Granary-v2

One-sentence close: Long recordings, multiple languages, and who is speaking—NVIDIA has delivered plenty on these three fronts this time. For real-time captions and messy 8kHz data like in phone customer service, wait for more specific evaluations first.