Robot Practices Fall Recovery, and the Reward Function Needs One Rule First: Keep Its Head Off the Ground

Sentdex's Harrison Kinsley has updated @asimovinc's fall recovery feature. The video shows a robot model with an orange body, black joints, green and blue connecting rods in the middle of its legs, standing on a gray grid floor. This test round captures how it recovers after falling; standing up is only one of the outcomes.

The tweet mentions two to-dos. The UI currently lacks a visible way to apply force; the force is invisible, so there is no way to push the robot over, which means some kind of object-throwing feature needs to be added. The other is reward design: Kinsley is also considering adding a reward for behavior that does not end in a full face-plant, and thinks this could help with the real robot. This is very specific, indicating that a face-hitting-the-ground path has already appeared in training.

reward is the signal in reinforcement learning that scores actions. In the replies, Viber · fireply.ai offered a short suggestion: reward the robot for keeping its head off the ground, and adding one line to the reward might fix the face-plant. The suggestion is not flashy, but it is very engineering-minded. Fall recovery is often viewed as a control problem, but during actual training the reward function determines which shortcut the robot takes. If you only reward standing up, the policy may find a path where it uses its face for support and then gets up; writing keeping the head off the ground into the reward amounts to tightening the acceptable actions.

Another commenter only replied progress. Jonathan Garcia's reply fits well in a work log. Updates like this feel more like a work log in developer circles: change the UI, tune the reward, watch the video, change it again. Next, if the UI can throw objects, the training environment can actively create disturbances; add a head constraint to the reward, and the tests will get closer to the shoving and falling that real robots encounter.