Toolboxes
UI Automation
The UI Automation toolboxes (UI Viewer and UI Controller) provide control over any macOS application using the Accessibility API. This enables AI assistants to interact with apps that don't have dedicated integrations.
Requirements
- Permission: Accessibility
UI Viewer Tools
Inspect application interfaces without making changes.
| Tool | Description |
|---|---|
ui_viewer_list_apps | List running applications |
ui_viewer_get_frontmost | Get the frontmost app and window |
ui_viewer_get_ui_tree | Get the UI structure as XML |
ui_viewer_get_visible_text | Extract visible text from an app |
ui_viewer_find_elements | Find UI elements matching a query |
UI Controller Tools
Interact with applications by clicking, typing, and more.
| Tool | Description |
|---|---|
ui_controller_toggle_app | Open, close, hide, or activate apps |
ui_controller_click | Click on UI elements |
ui_controller_type_text | Type text into input fields |
ui_controller_press_key | Press keyboard keys or shortcuts |
ui_controller_select_menu | Select menu items |
Example Usage
- "Click the Submit button in Safari"
- "Type 'hello world' in the search field"
- "What windows are currently open?"
- "Press Command+S to save the document"
- "Select Copy from the Edit menu"
- "Find all buttons in the current app"