0.83.0

takahirom/arbigent0.83.0Sep 16, 2026by takahirom

Upgrading?

AI Summary

This release extends project variables to initialization methods, enforces error handling for unresolved placeholders, and fixes a regression where the AI decision request did not include the annotated screenshot.

Key Highlights

  • Project variables (`{{name}}`) now resolve in `LaunchApp`, `CleanupData`, and `OpenLink` initialization methods.
  • Unresolved placeholders now cause a scenario failure before execution, providing clear error messages.
  • The AI decision request now includes the Set-of-Mark annotated screenshot instead of just the raw screenshot.

Breaking Changes

  • Unresolved `{{name}}` placeholders now fail the scenario before it runs.

New Features

  • Project-level variables in initialization methods
  • Unified error reporting for undefined variables before execution
  • Sending annotated screenshots in AI decision requests

Full Release Notes

This release extends `{{name}}` variables to initialization methods, makes an unresolved placeholder an error, and fixes a regression that sent the model the screenshot without the numbered boxes.

## Project Variables in Initialization Methods (#443)

`{{name}}` now resolves in `LaunchApp.packageName`, `CleanupData.packageName` and `OpenLink.link`, with defaults in the project YAML:

```yaml
settings:
  variables:
    appId: "com.example.app"
scenarios:
- id: "launch-app"
  goal: "Confirm the home screen is shown"
  initializationMethods:
  - type: "LaunchApp"
    packageName: "{{appId}}"
```

```bash
arbigent run --variables="appId=com.example.app.debug"
```

`--variables` (and `.arbigent/settings.yml`, and the UI) overrides `settings.variables` per key, so a job can swap out just the application id of a debug build. `{{inputs.*}}` resolves in those fields too, and its load-time check was extended to match.

## An Unresolved `{{name}}` Fails the Scenario Before It Runs (#444)

**Breaking change.** A goal with a `{{name}}` that nothing defines used to run with the placeholder passed straight to the AI. Now every unresolved reference is reported at once, before any device, AI or MCP work:

```
Scenario "launch-app" references variables that are not defined:
  - {{appId}} in LaunchApp packageName of scenario "launch-app"
  - {{user}} in the goal of scenario "launch-app"
Give it a default in settings.variables in the project YAML, pass it in with
--variables="appId=com.example.app", or write \{{appId}} if you meant the text itself.
```

Write `\{{name}}` for literal text. Placeholders outside the variable-name grammar, such as `{{user:id}}`, are unaffected. One rejected scenario does not stop the others, and loading a project never fails for this.

## The Model Sees the Annotated Screenshot Again (#445)

The Set-of-Mark annotated screenshot — the numbered boxes that map what the model sees onto the `<ELEMENTS>` indices — was drawn, saved and shown in the UI, but the **raw** screenshot was what went into the decision request. It had been that way for eight months. Both providers now send the annotated file; image assertions keep the raw screenshot on purpose.

The annotated image is drawn in the element coordinate system, so the boxes land on the elements. On Android that is pixels and the dimensions do not change; the drawn boxes make the PNG about 17% larger. On iOS it is logical points, so what is sent is 393x852 instead of the raw 1178x2556 — a change of coordinate system, not a quality setting. Input tokens stay about the same either way.

## What's Changed
* Support project-level variables in initialization methods by @takahirom in https://github.com/takahirom/arbigent/pull/443
* Fail a scenario with unresolved variables before it runs by @takahirom in https://github.com/takahirom/arbigent/pull/444
* Send the Set-of-Mark annotated screenshot in the decision request by @takahirom in https://github.com/takahirom/arbigent/pull/445


**Full Changelog**: https://github.com/takahirom/arbigent/compare/0.82.0...0.83.0

Related Repositories