Computer use
Drive local desktop apps from an agent via accessibility trees, screenshots, and safe UI actions.
The orca computer CLI lets an agent inspect and control native desktop apps — list running apps, read accessibility trees, click controls, set values, type text, scroll, and take screenshots. Use it when a task needs to operate the OS or a third-party app rather than a terminal or the built-in browser.
First-time setup
Check the runtime and permissions:
orca status --json
orca computer permissions --json
orca computer capabilities --json
If permissions reports anything missing, grant Accessibility (and Screen Recording on macOS) to Orca Computer Use in System Settings, then re-run permissions --json to confirm.
Snapshot → act → snapshot
Every interaction follows the same loop: read the app's current state, act on a specific element, then re-read state to verify the result.
orca computer list-apps --json
orca computer get-app-state --app com.spotify.client --json
orca computer click --app com.spotify.client --element-index 42 --json
Element indexes are scoped to the latest get-app-state result and may be sparse. In --json output, read the tree from result.snapshot.treeText. Do not invent indexes from elementCount. Refresh state after navigation, focus changes, scrolling, or any app re-render before reusing an index.
Action results return only the accessibility-tree lines that changed since the last state for that window (~ changed, + added, - removed), marked by snapshot.treeDiff; pass --full-tree to get the whole tree. To find one control in a large window, get-app-state --query <text> prints only matching elements and their ancestors, and --max-elements <n> caps the output. A window larger than the snapshot limit shows ... N more not shown (snapshot limit) where the tree was cut; get-app-state --element-index <n> reads only that element's subtree, so pass the cut-off children's parent to see them.
Waiting for the app
Instead of sleeping between actions, wait for what the app should show next. wait returns as soon as the condition holds, prints the matching elements or windows, and fails with wait_timeout after --timeout milliseconds (default 10000, at most 120000):
orca computer wait --app com.apple.TextEdit --text "Saved" --json
orca computer wait --app com.apple.TextEdit --enabled "Continue" --json
orca computer wait --app com.apple.TextEdit --window-title "Save As" --json
--text matches element lines the same way as get-app-state --query. --enabled holds once elements containing the text exist and none of them is disabled. --window-title checks the window titles list-windows reports (on Windows, only the main window's). wait only reads the app and keeps checking while the app or window doesn't exist yet. --enabled needs a runtime whose provider marks disabled elements (supports.observation.disabledState in capabilities).
Multi-window apps
orca computer list-windows --app com.microsoft.edgemac --json
orca computer get-app-state --app com.microsoft.edgemac --window-id <id> --json
orca computer click --app com.microsoft.edgemac --window-id <id> --element-index 12 --json
Prefer stable --window-id when the listed id is not none; otherwise use --window-index.
Selecting an app
Prefer bundle IDs returned by list-apps:
orca computer get-app-state --app com.microsoft.edgemac --json
App names work when unambiguous (--app Spotify). Use --app pid:<number> only when bundle ID and name both collide.
On macOS, apps without a Dock icon, such as menu bar apps, are listed while they show a window, marked accessory: true.
Starting an app
list-apps shows only running apps. To start a closed one without pulling you away from what you're doing, use launch-app:
orca computer launch-app TextEdit --json
It asks macOS and Windows to start the app in the background and leaves an app that's already running alone. Some apps still come forward on their own, so the result's delivery says whether the app ended up in front. --activate brings it forward instead. Linux doesn't support it yet.
Available actions
orca computer click --app <app> --element-index <i> --json
orca computer set-value --app <app> --element-index <i> --value "text" --json
orca computer type-text --app <app> --text "text" --json
orca computer type-text --app <app> --element-index <i> --text "text" --json
orca computer press-key --app <app> --key Return --json
orca computer hotkey --app <app> --key CmdOrCtrl+A --json
orca computer paste-text --app <app> --text "text" --json
orca computer scroll --app <app> --element-index <i> --direction down --json
orca computer drag --app <app> --from-x 100 --from-y 100 --to-x 300 --to-y 300 --json
orca computer perform-secondary-action --app <app> --element-index <i> --action <name> --json
Prefer semantic actions (click, set-value, perform-secondary-action) over raw type-text or press-key — they target accessibility elements directly and survive focus changes that keyboard input doesn't.
hotkey and press-key refuse shortcuts that quit or close the app, log out, lock the screen, force-quit or end the session, such as Cmd+Q, Shift+Cmd+Q, Ctrl+Cmd+Q and Option+Cmd+Esc on macOS, Alt+F4 and Win+L on Windows, and Ctrl+Q, Alt+F4, Super+L and Ctrl+Alt+Delete on Linux. The error code is system_shortcut_blocked. Pass --allow-system-shortcut to send one anyway.
type-text --element-index <i> inserts the text at that field's caret without focusing it or bringing its window forward. It works on macOS, Linux and Windows. On macOS, and for a Windows field written through its UI Automation value, a field that doesn't report its caret gets the text at the end. An app can still bring its own window forward in response, so check action.delivery.
On macOS, actions run in the background: the target app isn't brought forward, your frontmost app and key window stay put, and the pointer doesn't move, so you can keep working while an agent drives another window. Each action result reports action.delivery (background or foreground). Pass --restore-window only when an app ignores background input, such as drags in Chromium or Electron apps or games; it brings the app forward and uses the real pointer and keyboard, and the pointer goes back where it was unless you moved it meanwhile. A window that won't take focus, such as some start windows, still gets the click while nothing covers the point, and your front app comes back afterwards unless the click opened something. The bundled skill tells agents never to add it on their own: they ask you first, and error messages say the same. Popup menus and web <select> controls are set with set-value instead of a click. On Linux, element clicks through accessibility actions, set-value, type-text --element-index and secondary actions run in the background; actions that need the real pointer or the focused window are refused unless you pass --restore-window. On Wayland they are refused even with it, because synthetic input never reaches native windows. On Windows, element clicks, set-value (including sliders), scrolls and secondary actions (Invoke, Toggle, Select, Expand, Collapse, ScrollToVisible) go through UI Automation when the control supports them, without moving the pointer or sending keys. type-text --element-index also works there for text boxes. Some apps still bring their own window forward when driven this way (WPF apps do for most of these), so action.delivery reports foreground whenever the target window came forward. Actions that need the real mouse, such as coordinate clicks, right-clicks and drags, are refused unless you pass --restore-window; then only the target window comes forward and the pointer goes back where it was. Keyboard input (type-text without an index, press-key, hotkey, paste-text) is refused the same way while the window isn't in front.
When accessibility targeting fails, fall back carefully to coordinates:
orca computer click --app com.apple.Safari --x 120 --y 340 --json
orca computer drag --app <app> --from-element-index 3 --to-element-index 9 --json
Sensitive input
Pass secrets through stdin so they don't land in shell history:
printf '%s' "$TEXT" | orca computer set-value \
--app com.apple.Safari --element-index 7 --value-stdin --json
--text-stdin works the same way for type-text and paste-text.
On Linux and Windows, action payloads also pass briefly through a local operation file while the helper runs. Stdin keeps secrets out of shell history, but it does not hide them from every local observer; avoid sending secrets unless the task explicitly requires it.
Screenshots
get-app-state returns an accessibility tree and, by default, a screenshot. With --json, the image bytes are written to disk and the path is returned in screenshot.path rather than embedded in the response. Pass --no-screenshot when pixels aren't needed (faster, smaller payload). To keep pixels but send fewer bytes, --screenshot-scale 1 caps the image at one pixel per window point (Retina captures are otherwise two), --screenshot-max-edge <px> caps its longer side, and --screenshot-format jpeg encodes it as JPEG; screenshot.format and screenshot.scale say what came back. On macOS, a window that is minimized or hidden has no pixels, so the screenshot fails while the accessibility tree still comes back; a window on another Space is captured.
On Wayland, screenshots work on KDE Plasma, Hyprland and Sway. On Hyprland and Sway they need grim, GdkPixbuf, and hyprctl or swaymsg; Orca finds the target window by process ID and title and captures its region, so overlapping windows can show in the image.
On KDE Plasma, Orca asks KWin where the window is, takes a screenshot of the desktop through xdg-desktop-portal (with xdg-desktop-portal-kde), and crops it to the window. The portal saves that screenshot in your Pictures folder; Orca reads it and deletes it. The first screenshot makes the desktop ask whether apps may take screenshots. Answer the prompt, then run get-app-state again. If the prompt is answered after get-app-state gave up waiting, the portal may leave that one screenshot in Pictures. If you deny it, screenshots keep failing until you allow them again in the desktop's permission settings. A KDE window that another window covers returns a screenshot error instead of an image of the other window.
GNOME doesn't tell apps where windows are, so it returns a screenshot error. On any Wayland desktop, when the window is minimized or on another workspace, matches more than one window, or has a different size than its accessibility bounds, get-app-state returns the accessibility tree with a screenshot error.
Use it from an agent
The shipped orca-computer-use skill packages the same command surface with safety guidance. Install it into the agent's skill directory:
npx skills add https://github.com/stablyai/orca --skill orca-computer-use
See Skills registry & MCP for how skills are picked up.
Next steps
- Orca CLI overview — the rest of the CLI surface (worktrees, terminal, browser).
- Skills registry & MCP — distributing this CLI to agents.
The skill was previously named computer-use. Existing Orca CLI requests using that name remain supported. Use $orca-computer-use for agent invocations. Orca migrates verified, unedited installations through its skill updater and preserves edited copies. The old shared folder may remain as a forwarding link for existing agent profiles.