Real-world interaction that RL can't handle: LLM + robot harness offers a different solution
Harrison Kinsley's robot Chonk is entirely LLM-driven, without RL/VLA. It successfully grasped toys it had never seen before on the first try. When the Wi-Fi disconnects, the harness automatically recovers. Kinsley believes LLM + harness may be better suited than RL for complex real-world interaction.
Harrison Kinsley (Sentdex) has updated his robot project Chonk again. The most notable change this time: Chonk's control is entirely LLM-driven, with no RL and no VLA.
The task instruction was brief: find the children's toys on the ground, pick them up, and put them in the black box. This time the toys were ones Chonk had never seen before, and it succeeded on the first try. In the video, Chonk picks up multiple toys, recovers from a failed grasp, and adjusts its grip based on the shape of the toy. This clip is at 5x speed.
Kinsley says he spent several years using RL/VLA and various simulators to solve this type of problem. There are simply too many edge cases. Even if the task itself can be handled, sim2real using only camera data is already enough of a headache.
He started to change his approach: like the harness for a coding agent, let the LLM directly control the robot. Not all tasks require RL. But things like gait that require learning physical laws still need a highly scalable sim.
In the second video, Chonk faces a toy it has never seen before. The LLM's reasoning process can be seen in the debug panel in the upper right corner. It succeeds on the first attempt. At the end of the video, the Wi-Fi signal drops and the control loop crashes. The harness corrects itself and then completes the task. This clip is at real-time speed, with no acceleration.
A user asked whether, after the Wi-Fi disconnects, the harness would recheck the gripper and object state to avoid repeating operations that had already been completed. Kinsley said the gripper camera is always watching. Initial grasp failures and toys slipping out are not uncommon, so the system continuously confirms whether there is actually anything in the gripper.
As for the grasping strategy, Kinsley admits it is still fairly clumsy: try one approach first, and if that doesn't work, rotate 90 degrees and try again. This part still needs improvement.
A commenter noticed another detail: when Chonk sees only the cabinet and the floor, it scans 360 degrees to find the target, instead of, like some agents, making up a toy out of thin air and then ending the task.
Someone in the comments joked, "Yann LeCun in shambles." LeCun has always been skeptical of the LLM route; seeing this kind of purely LLM-driven robot, he probably won't be able to stay calm.
发布时间: 2026-11-23 05:33