Google Just Released Interactive Generation—But Did the Open-Source Community Already Do It Seven Months Ago?

Google recently released Gemini4, which comes with an instant interactive generation feature. A netizen dug up an open-source project from July called Incantation, arguing that it does almost the same thing, even in more granular detail: using natural language to control character actions in video content in real time.

This video was posted on July 2 by Bilibili creator 星空-dark-法师 and now has 42,000 views. The cover has one line: 'In other words, you can interact with video content in real time.' The description specifically notes: all demos are generated by video generation models; they are not built on a game engine and do not involve code agents or explicit environment programming. The model was trained on Elden Ring data, with Wan2.2-5B as the base.

The project is called Incantation. Its paper is on arXiv 2605.18601, and the authors come from institutions including Shanghai Jiao Tong University and NVIDIA Research. The core idea is to use natural language as the action interface for a video world model.

In traditional interactive video generation models, control signals often come from animation IDs, device input, or scene captions. These signals are bound to specific entities or engines at the design stage. Incantation takes a different approach: every 0.25-second latent frame receives a structured natural-language prompt, such as 'the player performs [Action A], and the Boss performs [Action B].' To control multiple entities, you add multiple prompt slots without changing the model itself.

The most striking data in the paper are three comparisons:

Natural language more than doubles performance on cross-entity transfer, and it can also control actions beyond the training vocabulary. The baseline action ID has no input interface at all when it encounters new words. In addition, after training on two-entity scenes, the model can generalize to controlling three entities at the same time.

In terms of implementation, Incantation makes two key design choices. Text cross-attention acts only on the current noisy frame, while historical frames retain bidirectional self-attention, preventing the current action prompt from contaminating already generated content; a sliding KV-cache handles long videos by caching the original keys and dynamically applying rotary position encoding after the sliding window, avoiding stale positional information. The student model is distilled via ODE-initialized Self-Forcing, reaching 163 ms per frame with 2-step inference, or about 19.7 FPS. Long-horizon stability has also been validated: FVD remains stable over a 2-hour rollout.

The same architecture was also transferred to the King of Fighters world, with only the action slots changed, and the results still held. This shows that language as an action interface is not effective only in a single game. The project has released a real-time demo (reactor.inc/incantation), code (GitHub), and a dataset (Hugging Face). The dataset contains 45 hours of Elden Ring boss fights and 83 hours of King of Fighters clips, with each entity's actions annotated at 0.25-second granularity.

As for whether Google really brought that open-source author on board, no one knows. But at least, for the 'new feature' at the big tech launch event, the open-source community had already given a sufficiently clear answer seven months earlier.