AI Operating Desktop Apps: From Guessing Buttons to Reading Buttons

Vercel Labs has open-sourced agent-desktop. Vercel is the company known for frontend deployment, and Labs is its experimental project pool. The project in one sentence: let AI operate desktop apps without screenshots, without pixel matching, and without taking over your mouse.

Over the past two years, when people talk about computer use, the mainstream approach has been to feed screen captures to a model, have it guess where the button is, click once, and take another screenshot. This approach works, but it is expensive, slow, and error-prone, and your cursor gets dragged around.

agent-desktop takes a different approach. It reads the operating system's accessibility tree, converts the current app's UI structure into JSON, and returns it, with each actionable element carrying a ref. The agent clicks or types against the ref, reads the UI again to confirm the result, and repeats.

The returned tree looks like this:

{
  "version": "2.6",
  "ok": true,
  "command": "snapshot",
  "data": {
    "app": "Finder",
    "snapshot_id": "s8f3k2p9",
    "ref_count": 59,
    "complete": true,
    "tree": {
      "role": "window",
      "name": "Downloads",
      "children": [
        { "role": "treeitem", "name": "Documents", "ref_id": "@s8f3k2p9:e8", "available_actions": ["Click"] }
      ]
    }
  }
}

A ref belongs to a particular snapshot. If the UI changes, an old ref will return STALE_REF or AMBIGUOUS_TARGET, and you have to take a new snapshot. This design forces the agent to look at the result before acting, rather than clicking blindly over and over.

One key point: agent-desktop has no model inside. It is a CLI and C library—the hands, not the brain. Only when connected to Claude, GPT, or another model does it become complete automation.

A few practical capabilities:

• Background by default. Ref operations go through accessibility APIs, without moving the pointer or stealing focus. Only with --headed is physical input allowed.
• Three platforms: macOS 13+ via AX API, Windows 10 1809+ via UI Automation, and Linux Ubuntu 24.04+ via AT-SPI.
• 60 command names, 56 actionable commands, covering observation, clicking, input, windows, notifications, clipboard, waiting, and sessions.
• Native MCP: agent-desktop mcp exposes each command as an MCP tool and returns JSON.
• Apache 2.0, installable via npm and Homebrew.

Usage is straightforward:

npm install -g agent-desktop
agent-desktop status
agent-desktop snapshot --app Finder -i
agent-desktop click @s8f3k2p9:e8
agent-desktop snapshot --app Finder -i

Those three lines form a complete round: look, act, look again.

The accessibility tree approach has a clear boundary: not every app exposes it well. For custom-drawn UIs, Canvas, WebView embedded in Electron, and games, the tree may be empty or contain only a shell. With such apps, the screenshot camp still has to come back. agent-desktop solves the part of the desktop where software is willing to be read, not the entire desktop.

Another detail from the README: snapshots, clipboard contents, screenshots, and command output may contain private information; don't put them into public logs or uploads.

What's interesting is where this infrastructure came from. The accessibility tree was originally built for visually impaired users and assistive technologies; now it has become the desktop interface for AI. Something built for accessibility incidentally solves part of the automation problem.

One commenter online judged that the next generation of power users won't be people who know how to use 100 software apps, but people who can get AI to use those 100 apps for them. That claim is a bit absolute, but it points to the direction of change: software used to be designed for people; now it is starting to be designed with interfaces for agents.

As for which job will be changed first, it's too early to guess. What is certain is that the barrier to desktop automation has dropped from knowing how to write scripts to knowing how to connect a model.

Project URL: https://github.com/vercel-labs/agent-desktop