agent-desktop
Also known as: vercel agent-desktop, agent desktop CLI, desktop computer use CLI
Most computer-use tools work by taking a screenshot, figuring out where things are on screen by pixel coordinates, and clicking there. agent-desktop takes a different approach: it reads the operating system's built-in accessibility tree, which is the same structured data that screen readers use. Every button, text field, and menu item has a stable reference ID. The agent clicks by ref, not by coordinate, so a UI theme change or a DPI scaling shift can't break the automation.
Vercel Labs released agent-desktop as version 1.0.2 on October 30, 2026, under the Apache 2.0 license. It ships as a Rust CLI and a C library that runs on all three major desktop platforms. By default it operates apps headlessly, meaning it doesn't move your system pointer or steal focus while the agent works. You can opt into a headed mode for apps that require real mouse input.
The practical appeal for builders is that you can write one agent loop and point it at desktop apps without shipping a separate screen-scraping pipeline per app. The catch: apps that don't publish an accessibility tree are invisible to it, so poorly accessible software remains hard to automate. It is not an agent itself; it embeds no model. The observe, act, and verify logic stays with whatever agent is calling it via shell commands.