v1.17

epi052/feroxbusterv1.17May 23, 2026by github-actions[bot]

AI Summary

This release fully implements the `computer_use_remote` tool, enabling visual verification, multimodal screenshot capture, and platform-specific structural targeting skills for comprehensive desktop control.

Key Highlights

  • `computer_use_remote` exposed as a callable tool for live sessions.
  • Visual verification required for all state-changing desktop actions.
  • Screenshots attached as multimodal vision messages instead of text summaries.
  • Platform-specific structural targeting skills for macOS (AX), Windows (UIA), and Linux (AT-SPI).

New Features

  • Full Computer Use Remote pipeline with visual verification.
  • Multimodal screenshot capture for agent inspection.
  • Platform-specific desktop control skills (macOS, Windows, Linux).
  • Codex OAuth proxy preserving vision inputs.

Full Release Notes

# Computer Use Remote: Full Desktop Control Pipeline

This release builds out the `computer_use_remote` tool into a complete, cross-platform host desktop control system with visual verification, multimodal capture handling, and platform-specific structural targeting.

## Highlights

- **`computer_use_remote` exposed as a callable tool** — The model can now invoke `computer_use_remote` directly in live sessions. Availability, trust mode, and re-arm enforcement remain runtime checks rather than prompt-loader gates.

- **Visual verification required for all desktop actions** — State-changing desktop actions are treated as unverified attempts until a fresh screenshot visibly confirms the outcome. Agents must stop and not proceed when a screenshot is unavailable.

- **Screenshots attached as multimodal tool results** — Computer-use captures are returned as real multimodal vision messages (not just text summaries), so the model can visually inspect the screen after each action. Older capture payloads are pruned to prevent runaway context growth.

- **Host desktop cleanly separated from Xpra desktop** — `computer_use_remote` is now the sole host desktop-control path; `linux-desktop` targets only the internal Docker/Xpra environment. Host-screen queries rank ahead of the Xpra skill while explicit "Agent Zero Desktop" requests still route correctly.

- **Platform-specific structural targeting skills:**
  - **macOS** — Dedicated skill for Accessibility (AX) structural targeting with `ax_snapshot` and `ax_action` support, loaded only when the backend reports macOS capabilities.
  - **Windows** — UIA-based skill with window-management guidance, selector passthrough, and click-last workflow hints.
  - **Linux** — AT-SPI/Wayland skill with compact structural tree outlines in snapshot responses for semantic target selection.
  - Backend-specific action details are kept out of the generic prompt; the generic layer handles only backend discovery and skill loading.

- **Codex OAuth proxy preserves vision inputs** — Image content parts are now correctly converted to Responses API `input_image` parts instead of being flattened to text, with regression coverage for multimodal tool results containing screenshots.

- **macOS approval denial handled gracefully** — `COMPUTER_USE_APPROVAL_REQUIRED` responses map to the existing re-arm-required stop flow, preventing agents from retrying or falling back to screenshots when permissions haven't been granted.

- **Prompt token accounting fixed for screenshots** — Embedded base64 image data URLs are sanitized from token estimates so screenshot attachments no longer inflate context budgets.

- **Window-hide guidance updated** — Ubuntu/GNOME/Wayland sessions now prefer `Super+H` over `Alt+F9`, with a reminder that keystroke results only prove the keys were sent, not that the action succeeded.