AI builds an iron pickaxe from scratch in Minecraft with keyboard and mouse, without touching APIs
Researchers from Tencent and other institutions have released HiSTEVE 1.0, a hierarchical VLA agent that enables AI to complete long-horizon tasks in Minecraft using only vision and native keyboard-and-mouse controls. It achieves 85.0% on MineStudio Bench atomic tasks and a 37.4% average success rate on OpenHA.
AI builds an iron pickaxe from scratch in Minecraft with keyboard and mouse, without using APIs at any point. This is HiSTEVE 1.0, released by researchers from Tencent and other institutions: a hierarchical vision-language-action (VLA) agent aimed at enabling AI to complete complex tasks in open worlds like a human player.
Minecraft is a procedurally generated open world where players gather resources, craft, and build. Long-horizon tasks test both planning and motor control. Existing agents often excel at one but not the other: some can only perform low-level actions and lack long-term planning; others can plan but rely on environment APIs or predefined skills. HiSTEVE tries to combine both, without using any privileged information—it only looks at the screen and operates with keyboard and mouse.
The architecture has two layers. The upper layer is a capability-aware agent framework responsible for orchestration: it uses structured memory to track progress, a VLM planner to select subtasks within the VLA's capabilities, and a policy-coupled task-completion state detector to determine whether a subtask is complete and decide whether to continue, recover, or replan. The control layer is a single VLA that executes all subtasks using only visual perception and native keyboard-and-mouse actions.
In the game, players need to click in the inventory interface and also move and act in a 3D scene, and these two types of interaction differ greatly. HiSTEVE uses mode-specific RMSNorm branches and separate mouse-movement token embeddings to reduce mutual interference. Training has three stages: world knowledge injection, imitation learning, and reinforcement learning.
Results: 85.0% on MineStudio Bench atomic tasks, a 37.4% average success rate on OpenHA, and it can even craft an iron pickaxe from scratch like a player. In the paper's comment section, one commenter asked whether there was a demo video; another said watching the AI figure things out in an open world was quite interesting; and someone else summed it up: one misclick in the inventory interface can waste the previous ten minutes of planning.
发布时间: 2026-10-27 04:01