A Basketball Video, One Instruction, and the LLM Ran the Entire Animation Pipeline

Robert Scoble reposted a tweet demonstrating a complete pipeline. Upload a video of someone dribbling a basketball indoors, enter one sentence: "Create a simple 3D animation from this video." The rest is run by the LLM agent itself: setting up the scene, using FlowHMR for motion capture, retargeting, fixing motion, setting the camera, and rendering. Finally, MiniMax-H3 turns the 3D render into a realistic video.

The video is split into left and right parts. The left is the reference video, and the right is the input frame: a person in a blue sports outfit and red shoes dribbling a basketball, with a wooden floor, brick wall, and metal-framed ceiling all present. At the bottom there is a progress bar and a Coding agent prompt indicating which step is currently in progress.

FlowHMR is the core tool in this pipeline, from the work by Zhanke Wang et al., with the paper on arXiv (2610.03691). It recovers physically plausible 3D human motion from monocular video. It has 0.46B parameters, and its inference pipeline chains four modules: YOLOX for person detection, VGGT-Omega for camera estimation, SAM 3D Body for extracting body features, and finally FlowHMR outputs motion data in .npz format; if FBX export is installed, it can also output an additional .fbx.

Physical plausibility is where this tool focuses its efforts. FlowHMR training has two stages: first pretrain a flow-matching foundation model, then use GRPO for post-training, with reward signals from PHC+ and a physics simulator. Every generated motion is thrown into the simulator for the controller to track once; the better the tracking, the higher the score. The official Tracking Success metric rises from 78.79% for the base model to 82.47%, with the evaluation set being Wild-4K, composed of 4,182 real-world videos.

The code is open source under Apache 2.0, with weights on Hugging Face. The repository has complete documentation for inference, training, web demo, and data generation, and also provides prompts that can be pasted directly into a coding agent. Developers can follow along to turn a video into 3D motion data.

What is worth noting is how this is organized. Progress on individual models is of course important, but the clearer signal in this demo is that the agent is orchestrating the toolchain. Previously, doing mocap required running detection, camera estimation, solving, retargeting, and rendering step by step, with parameter tuning and result checking at every step. Now this sequence of operations is packaged as steps the agent can call, and one natural-language instruction triggers the entire pipeline.

Behind "one sentence" is a large amount of interface design work, making the output format, error handling, and default parameters of each step reliably consumable by the agent. This engineering work is not light.

Video to 3D and then back to video—this pipeline already runs end to end. How good the results are, you can pull the code and try for yourself.

Project address: https://github.com/flowhmr/flowhmr