Working on your screen
Prepare macOS permissions and inspect tasks that operate real application interfaces.
In this topic
Computer Use lets a Mellow agent inspect and operate native applications. It is useful when the task has no suitable structured tool: moving through a settings screen, preparing a form, or reading a desktop interface. Treat it as an attended workflow until you have reviewed the agent's behavior and permissions.
Set up a first task
Grant Accessibility in the Computer Use settings, then enable Computer Use for the custom agent that will perform the work. Screen Recording is a separate permission used when screenshots are needed. An enabled capability does not override a denied macOS permission.
Choose Balanced autonomy for the first task. Keep the chat open so approval cards can appear. Start with a specific application and a read-only objective, then check the observed result before allowing edits.
In Calendar, find tomorrow's first meeting and tell me its title and start time. Do not change it.
The coordinating agent can delegate to an agent with this capability. Grant the capability to the worker that needs it rather than assuming the coordinator operates the desktop directly.
Decide what can proceed without you
| Preset | Navigation | Reviewable edits | Consequential actions |
|---|---|---|---|
| Read-only | Automatic | Blocked | Blocked |
| Cautious | Ask | Ask | Ask |
| Balanced | Automatic | Ask | Ask |
| Trusted | Automatic | Automatic | Ask |
| Autonomous | Automatic | Automatic | Automatic, subject to additional gates |
Per-app rules and an agent's autonomy ceiling can make the global preset stricter. An app allowlist can further restrict the targets. Sensitive applications have additional confirmation rules. The broadest preset is therefore not a blanket grant to every operation.
A button such as Send, Purchase, or Delete represents a different effect from scrolling. Inspect the action, target application, and proposed text in the approval. Stop the run if it is acting on the wrong window.
Read the execution feed
The worker loops through observation, proposed action, permission check, input, and verification. Accessibility information is the primary way it identifies controls. Screenshots can supplement that when necessary.
Posting an input event is not proof that the application accepted it. The run distinguishes an observed change from an action posted without an observed result. When an operation may already have committed, inspect the target application before retrying. Stop controls end further work but do not roll back completed changes.
The current loop defaults to 24 steps and a five-minute active-work budget, with additional stall detection. The worker is bounded by step, elapsed-time, and stall limits. Approval wait time is handled separately from active work time. A headless run cannot complete a step that needs an approval surface unavailable to it.
Screen context and cloud vision
Screen context is a text snapshot that helps an enabled agent understand the foreground application. It is separate from permission to click or type. Review its preview and per-agent setting before using it around private content.
A cloud-backed model can receive context through its configured request path. Local execution is not a universal guarantee that every tool result remains local. Review the selected provider and Privacy Filter when deciding what context to share.
Cloud screenshot use has a separate consent control. Mellow can mask recognized text before sending a frame; the stricter mode covers all recognized text, while selective masking depends on detection. OCR and sensitive-data detection have limits. A masked screenshot is not proof that every identifying visual detail was removed.
AppleScript assistance
Some applications expose a scripting interface that works better than repeated clicking. Where offered, an AppleScript helper prepares a script for review. macOS Automation permission may be required for each target application. Read the script and intended effect before allowing it. A script can affect several objects even when it is short.
Resolve a blocked run
- Capability unavailable: check Accessibility and the agent's Computer Use setting.
- No approval window: open the chat and retry the specific operation.
- Application not ready: open it manually and wait for startup or sign-in.
- Repeated actions without progress: stop, narrow the objective, and identify the expected control explicitly.
- Screenshot unavailable: check Screen Recording and the chosen cloud-vision consent mode.
- Remote request refused: host-only restrictions still apply to paired-device and external callers.
The model-facing operation is computer_use; it receives the overall goal and starts the worker. Use Apple apps for structured app operations and Browser Use for websites.