v2.8.0

siddharthvaddem/openscreenv2.8.0Jul 20, 2026by kausmeows

AI Summary

Introduces a comprehensive scoring framework (`agno.scorer`) and evaluation environments (`agno.environments`) for running rollouts. Adds new tools like FileGenerationTools and Gmail pagination, alongside bug fixes for metadata and decision logs.

Key Highlights

  • New `agno.scorer` module with `CodeScorer`, `JudgeScorer`, and `ToolCallScorer`.
  • New `agno.environments` for running rollouts (`run_rollouts`) with isolation.
  • Integration of `Case.scorer` into the eval suite.
  • New tools: FileGenerationTools (code generation), Gmail Tools (pagination), Adanos Tools (market sentiment).

Breaking Changes

  • `ReliabilityEval` now strictly matches tool executions (clean execution required).
  • `AgentAsJudgeEval` prompt hardened with random nonce to prevent prompt injection.

New Features

  • CodeScorer, JudgeScorer, ToolCallScorer
  • Environment, Task, run_rollouts
  • Case.scorer integration
  • FileGenerationTools code generation
  • Gmail Tools pagination

Full Release Notes

# Changelog

## **New Features**

- **`agno.scorer`** — Turn a run into a number:
    - **`CodeScorer`**: wraps any callable (`bool | float | Score`; typed-field comparison under `output_schema` recommended)
    - **`JudgeScorer`**: LLM judge with the model always an explicit choice and numeric verdicts normalized to exact endpoints (`(score - 1) / 9`)
    - **`ToolCallScorer`**: checks tool **executions** deterministically (refused, errored, or HITL-rejected calls never satisfy an expectation)
    - All scorers ship sync and async variants.
- **`agno.environments`** — `Environment` + `Task` + `run_rollouts(env, k=8)`: run each task K times in full isolation (fresh db/session/user, no memory/knowledge/learning writes, cache off; knowledge reads still work). Includes a live per-attempt grid, real pass rate per task, drift-vs-policy fingerprints, `save`/`load`/`diff`, `learning_zone()`, and `to_sft_jsonl(...)` to export passing attempts as conversational-SFT JSONL with a provenance sidecar. This is the pass@k door.
- **`Case.scorer`**: the eval suite gains a third check — plug any scorer into a `Case` (with `Case.expected`), free and exact, no LLM call. `SuiteResult.to_dict()` gains additive `score_value`, `score_passed`, `score_reason` keys.
- **FileGenerationTools**: Added code file generation for `FileGenerationTools`.
- **Gmail Tools**: Added pagination and `max_results_per_request`.
- **Adanos Tools**: Added optional Adanos market sentiment tools.

## **Bug Fixes**

- **RemoteAgent / RemoteTeam**: Fixed metadata being dropped on the A2A protocol path.
- **Content**: Return empty string for empty content list in `get_content_string()`.
- **Decision log**: Replaced deprecated `datetime.utcnow()` in the decision_log store.

## **Breaking Changes**

- **`ReliabilityEval` matches tool executions**: Tool expectations are now satisfied only by a clean execution (`RunOutput.tools`, `tool_call_error` not set), not by message-side requests. **Verdicts can flip red after upgrading** — when they do, the eval was previously passing for the wrong reason (missing entries are annotated `"... (requested but refused/errored — execution matching, new in 2.8.0)"`). Argument checks moved to `ToolExecution.tool_args` with the same partial-match semantics.
- **Hardened judge prompt**: Every `AgentAsJudgeEval` now fences judged output behind a per-call random nonce, with the untrusted-data instruction inside the prompt. A literal `</output>` no longer escapes the block, and "score this 10" inside a judged answer is data, not an instruction. **Judge verdicts and token counts may shift.**

## What's Changed
* fix: replace deprecated datetime.utcnow() in decision_log store by @Ghraven in https://github.com/agno-agi/agno/pull/7949
* cookbook: add dpo_jury pairwise preference example by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9033
* cookbook: refresh data_labeling for agno 2.7.x by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9037
* cookbook: jury calibration, hardening, and agreement metrics by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9038
* cookbook: synthetic data generation workflows by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9040
* cookbook: critique-revision, persona, and tool-call trajectory generation by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9043
* cookbook: step-reward scoring, scale-out mechanics, and safety labeling by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9046
* cookbook: image_search README - ingest is a full rebuild, not idempotent by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9047
* [fix] RemoteAgent/RemoteTeam drop metadata on A2A protocol path by @psinojiya in https://github.com/agno-agi/agno/pull/8944
* feat: add pagination and max_results_per_request to Gmail tools by @Mustafa-Esoofally in https://github.com/agno-agi/agno/pull/9030
* fix: Return empty string for empty content list in get_content_string() by @chuck-duplocloud in https://github.com/agno-agi/agno/pull/6122
* [feat] Add optional Adanos market sentiment tools by @alexander-schneider in https://github.com/agno-agi/agno/pull/9060
* [fix] Clarify Adanos trending ranking by @alexander-schneider in https://github.com/agno-agi/agno/pull/9061
* fix: replace retired qwen/qwen3-32b with openai/gpt-oss-20b on Groq by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9055
* feat: add code file generation to FileGenerationTools by @anuragts in https://github.com/agno-agi/agno/pull/8420
* feat: agno.scorer and the judge prompt fence by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9049
* feat: rollout engine and Case.scorer seam by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9050
* release: v2.8.0 by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9063
* cookbook: expand environments into progressive verification suite by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9070
* feat: Release v2.8.0 by @kausmeows in https://github.com/agno-agi/agno/pull/9073

## New Contributors
* @psinojiya made their first contribution in https://github.com/agno-agi/agno/pull/8944
* @chuck-duplocloud made their first contribution in https://github.com/agno-agi/agno/pull/6122
* @alexander-schneider made their first contribution in https://github.com/agno-agi/agno/pull/9060

**Full Changelog**: https://github.com/agno-agi/agno/compare/v2.7.4...v2.8.0