Mimi VAP Is Open Source: Predicting Every 80 Milliseconds, Voice Agents Need to Learn to Take Turns

Voice agents often reveal their weaknesses in two places: interrupting, or waiting too long. Gabriel Skantze released an open-source model called Mimi VAP on X, with the goal of enabling machines to judge in advance who will speak next in a conversation. The project code and weights are on GitHub under the MIT license.

Mimi VAP is a turn-taking model. It is based on Voice Activity Projection (VAP), proposed by Gabriel Skantze and Erik Ekstedt in 2022, and replaces the original CPC encoder with Kyutai's frozen Mimi audio encoder. Mimi is also used in Moshi. The model estimates the current voice activity of both conversation participants every 80 milliseconds, as well as the relative voice activity for up to 2.4 seconds into the future. In other words, it does not just answer 'is there sound now?' but also guesses 'who will keep speaking in the next few seconds, and who might take the turn?'

This prediction is very practical for voice agents. VAD silence thresholds can only determine whether there is sound at the moment, but human turn-taking points often occur before the other person has completely finished speaking. Mimi VAP aims to distinguish three situations: brief pauses within the other person's turn, gaps suitable for a response, and moments when a backchannel is appropriate. A backchannel is a short response such as 'mm', 'right', or 'I'm listening' that does not take over the turn but keeps the conversation going.

The model details are not complicated: on top of the frozen Mimi, there are 18.3 million trainable parameters. The training data consists of about 6,630 hours of two-person conversations, and it does not rely on transcribed text or explicit turn annotations; it learns only from conversational voice activity. It supports streaming inference, and the code and weights are downloadable.

Its boundaries should also be made clear. It predicts voice activity, not semantics. Voice agents can use it for turn-taking, interrupting, and backchanneling strategies, but topic understanding still has to be handled by ASR and dialogue models. Many voice assistants are still waiting for silence thresholds; Mimi VAP offers a different approach: start estimating the next step before the other person has finished speaking.

Sources: