Should a Video Assistant Speak First? StreamVibe Makes 'When to Shut Up' a Benchmark

The paper ranked second on Hugging Face Daily Papers on October 27 is called StreamVibe. Its authors include Chengzhi Liu, Yue Cao, and others, and its paper page carries a Google affiliation. The problem it wants to solve is very specific: video models can already understand visuals, but most wait for explicit user instructions. In real interactions, users often do not say everything, and their intentions change.

The paper breaks this kind of interaction into three types of intent: explicit, implicit, and dynamic. Explicit means the user asks directly; implicit means the user does not ask, but their actions already reveal a need—for example, while browsing a webpage, they want to know where to click next; dynamic means the need changes over the course of the interaction. Existing methods mainly rely on explicit requests and have limited handling of implicit intent and continuous state updates.

StreamVibe provides a three-part suite. The first is StreamVibe-Bench, with 4.1K evaluation samples and 85K training samples, covering explicit, implicit, and dynamic intent. The second is StreamVibe-VL, a streaming proactive model that infers implicit intent from continuous observations and adjusts its interactions as intent changes. The third is StreamVibe-Harness, which uses hierarchical memory to store user intent as persistent state with a lifecycle, tracking the latest intent and environment state.

The experiments yield three conclusions: StreamVibe-VL retains general video understanding ability and improves response timing and answer quality by 10.7% over baselines; existing proactive models are still limited in understanding user intent and judging when to respond, especially under implicit and dynamic intent; StreamVibe-Harness improves response consistency and adaptation in sustained interactions, raising proactive interaction performance by another 12.3%.

There is a more practical question in the comments. One user said that this kind of proactive video agent—one that can watch you reach for the third coffee capsule—is the sort of project you'd want to prototype over the weekend. Another user followed up: a proactive video agent needs to know exactly when to stay quiet. A cooking assistant that speaks up every time you pick up a knife will quickly get turned off. Did the paper test for false interruptions?

This question is closer to the make-or-break line for a product than the 10.7% and 12.3% figures. The value of proactive interaction lies in reducing operating costs; the risk lies in adding disturbance. The model has to judge whether intent is stable, whether the current response is worth interrupting for, and whether the user has already solved the problem themselves. The paper puts the problem on the table with benchmarks and memory mechanisms, but false interruption rates, user patience, and exit costs in real-world scenarios still depend on follow-up evaluations.

Paper links: Hugging Face Papers, arXiv. Video from AK's tweet.