Computer Use
Computer Use gives your AI assistant the ability to open, read, and operate any Mac application, even apps without a dedicated integration. It runs entirely on your Mac and works with any MCP client. The driver runs in the background, so your active window and cursor stay untouched.
Requirements
- Permission: Accessibility
- Permission: Screen Recording (required only when capturing annotated screenshots)
How AI targets elements
Macuse gives each UI element a short, stable ID so your agent can refer back to the same button, field, or menu across tool calls. The ID is a letter (by role) followed by a counter: B1, T2, L1.
| Prefix | Role |
|---|---|
B | Button |
T | Text field, text area, search field |
L | Link |
S | Slider |
C | Checkbox |
R | Radio button |
M | Menu item |
P | Popup button, combo box |
I | Image |
X | Static text |
V | List, table, outline |
G | Tab group |
D | Disclosure triangle |
Call computer_use_snapshot or computer_use_find_elements to get a refs map, then pass these IDs via the target parameter on any interaction tool.
Tool catalog
Fourteen tools, grouped by what they do.
Observe
| Tool | Description |
|---|---|
computer_use_list_apps | List running Mac applications, optionally frontmost-only. |
computer_use_list_windows | List a single app's windows. |
computer_use_snapshot | Capture refs for an app's UI, with optional screenshot and annotation. |
computer_use_find_elements | Find elements matching a text query or XPath. |
computer_use_read_text | Extract visible text under part of the UI. |
Interact
| Tool | Description |
|---|---|
computer_use_click | Click an element or screen point. Supports multi-click via click_count. |
computer_use_type_text | Type into a field, with options to clear first, press enter, and verify. |
computer_use_press_key | Press a key or chord like cmd+shift+s. |
computer_use_scroll | Scroll at an element or screen point. |
computer_use_drag | Drag from one target to another. |
Compose
| Tool | Description |
|---|---|
computer_use_window | Close, minimize, restore, fullscreen, focus, move, or resize a window. |
computer_use_menu | Walk a menu bar path like ["View", "as Icons"]. |
computer_use_dialog | Navigate, set a filename, select an existing file, confirm, or cancel an Open / Save dialog. |
Lifecycle
| Tool | Description |
|---|---|
computer_use_app | Launch or quit an application. |
Approvals
The first time your agent tries to drive a new app, Macuse asks for your approval, either through the connected AI client or in a desktop prompt. You can allow it once or always; you can revoke any approval from the Permissions settings page at any time. Read-only tools (list_apps, list_windows, snapshot, find_elements, read_text) never prompt.
For password managers and keychain apps, only one-session approval is ever offered. The "allow always" option is never available for these apps, so a runaway agent cannot turn a single moment of consent into permanent access to your secrets.
Example prompts
- "Take a snapshot of Safari and show me all the buttons."
- "Click the Submit button in the current Figma window."
- "Type 'hello world' in the search field and press enter."
- "Scroll down in the document."
- "Drag the file icon to the trash."
- "Press Command+S to save the document."
- "Select Copy from the Edit menu."
- "List all windows in VS Code, then minimize the second one."
- "Save the file as report.pdf in my Documents folder."
- "Launch Slack."
- "Find all buttons in the current app."
Related Workflows
- Batch Data Entry: Automate repetitive form filling
- Menu Navigation: Execute complex menu commands