Robots Learn Pen Spinning from Video, but Get Stuck on the Force Humans Never Captured

The Harvard Computational Robotics group released a video on Halloween: a robotic hand performs dexterous movements around a jack-o'-lantern, and a humanoid robot carries boxes. The project is called GRASP, and it performs physics-based motion retargeting for dexterous hands and humanoid robots. In the video, the robotic hand even writes GRASP.

General readers can think of motion retargeting as mapping human motions onto a robot. The trouble is that the mapped motion may look similar, but the contacts and forces may not hold up. Heng Yang wrote in a tweet that kinematically retargeted motions may look right, but the contacts and dynamics they require are not achievable by the robot. The subsequent reinforcement learning can only chase a physically inconsistent reference, and the more it learns, the more awkward it becomes.

GRASP's approach is to first fix the reference motions in simulation. It does not require a differentiable simulator. Instead, it stores rollouts that have already been run, including rejected proposals, fits a local quadratic model using these control-cost pairs, extracts gradients from the model, and uses them to guide the next sampling step. The acceptance test still uses the simulator's original cost, with a Metropolis-Hastings correction, ensuring that the sampling target is not biased by the surrogate model. For long motions, it uses MPC rolling optimization. Technically, it builds on the TSMC work; Heng Yang says that even a simple quadratic model can provide useful directions for MCMC rejuvenation.

The results are fairly solid. On OakInk2's 1,352 bimanual tasks, the Sharpa hand achieves a 94.2% success rate, with an average object position error of 2.87 cm. With the same optimizer configuration switched to Wuji and XHand, the success rates are 92.4% and 95.0%. On real hardware, the Sharpa hand performed open-loop trajectory replay, and a tracking policy was learned on the Unitree G1. In the PPO curves, the GRASP-corrected references are easier to track than the original kinematic references.

A commenter asked a straight question: if object mass or friction is changed during evaluation, how much of the PPO tracking benefit remains? Another asked whether GRASP adjusts contact timing or only pose. The material does not provide answers. These two questions are closer to real-world deployment than whether a robot can spin a pen. Pen spinning and box carrying can demonstrate contact-rich motions, but for cutting a tomato in a kitchen—how deep the knife presses and when the fingers close—cannot be learned from the few pixels in a video. A commenter used tomato cutting as an analogy and included an image comparing a human cutting a tomato with a robotic arm cutting a tomato.

First make the motion physically valid in simulation, then let the policy learn it—the order is right. The remaining robustness still has to be tested item by item. The paper, code, and data are all on the project page; the code and dataset have also been released.