A Few Photos, a Room Where You Can Push Doors, Drop Apples, and Run Robots

Take a few photos of a room, then make a drawer slide open, an apple fly toward the monitor, and a robot walk in through the doorway. Yihang Luo and others at MMLab, Nanyang Technological University, put this demo into Mundo: input multi-view photos of a place, and get a world you can explore, edit, and simulate. The project page is at mundo-world-model.github.io, and the technical details are in the Hugging Face blog.

Video from Yihang Luo's X post.

The term world model has been used very broadly lately. Mundo's division of labor is specific: the scene is represented by 3D proxies, state is recorded in code, MuJoCo computes geometry, collisions, and contacts, and the video model is only responsible for rendering the simulation results into images. Being able to accurately make a drawer slide open is more informative than empty talk about 'understanding the world'.

The plain-language version: Mundo does not ask the video model to guess the entire process out of thin air. It first puts the positions of objects and their physical relationships in a room into a simplified 3D scene, then lets the simulator compute what happens next, and finally uses a video diffusion model to fill in realistic appearance.

First, compute the space accurately

Reconstruction starts from multi-view images or video. A 3D foundation model recovers camera poses and dense geometry, and a 3D detector gives initial bounding boxes for the main objects. A VLM agent then enters an analysis-by-synthesis loop: edit the scene program, render from the input cameras, and compare with the original images. Another reviewer agent checks all input views each round. The boxes do not aim for fine geometry; they only need to represent position, size, orientation, and rigid-body simulation. Objects that need fine contact can have rough models generated separately or taken from an asset library.

The authors compared Boxer, SpatialLM, and Mundo on the same hotel room. Boxer and SpatialLM miss some objects, and some boxes deviate from their targets; Mundo's boxes fit the objects more closely. The difference can be seen by overlaying the 3D reconstruction in the figure with the input views.

The world is maintained in code

The scene ultimately is code. The program records objects, poses, mass, friction, and applied forces, and also defines how state updates over time. Changing physical properties, moving the camera, and mounting the camera on a robot are all done through this scene program.

In the rendering stage, the simulator first outputs proxy frames. Proxy frames are flat-colored by object category and sent into the video diffusion model together with the input context images. Camera parameters are converted into Plücker rays and added to the corresponding latents. Long videos are inferred in chunks, 124 frames per chunk, and each chunk uses the last 34 frames of the previous chunk as a fixed prefix to keep motion continuous. Training used about 40,000 video pairs, fine-tuning MiniMax-H3's Ref2VA with clips of 124 frames at 640×480.

What it can do, and where it gets stuck

Spatial exploration is a basic capability. Given 1 to 12 context images and a specified camera path, it generates new viewpoints; areas not captured by the input images are inferred by the model. The more context images, the more evidence there is, and the less the model has to guess.

In physical interaction, apples, drawers, collisions, contacts, and robot manipulation are all computed by the simulator. The robot application is more practical: 3D proxies record what each object is and where it is, and the robot can plan routes by semantic start and end points. The authors show navigation from a hotel lobby to a room, and also an action of opening a door in a laundry room.

The limitations are also laid out in the blog. The current version is slow: reconstruction for this kind of scene usually takes 10 to 20 minutes, and proxy rendering runs at about 1.5 frames per second on 8 A800 GPUs with 20 inference steps. Small objects tend to float, and the video model hallucinates when rendering interactions with small objects. The system has no long-term memory; after long dynamic changes, when previously occluded areas reappear, details may not be preserved.

What makes Mundo worth watching is not just generating another video that looks like a room. Editable 3D proxies, physics simulation, and video rendering are connected together. Many world model demos stop at image generation; Mundo at least puts drawers, apples, and robots into the same computable scene. The approach looks more cumbersome, but what it buys is explorability, operability, and reproducible experiments. To see more, go directly to the project page and blog.