v2.8.0
agno-agi/agnov2.8.0Jul 20, 2026by kausmeows
AI Summary
Introduces a robust scoring system and evaluation environment for running rollouts, alongside new file generation capabilities and Gmail pagination.
Key Highlights
- New `agno.scorer` package with CodeScorer, JudgeScorer, and ToolCallScorer
- New `agno.environments` package for running rollouts (pass@k)
- Integration of `Case.scorer` into the eval suite
- FileGenerationTools code file generation support
- Gmail Tools pagination and `max_results_per_request`
Breaking Changes
- `ReliabilityEval` now strictly matches tool executions, causing verdicts to potentially flip red
- Judge prompts are hardened with random nonces, potentially shifting verdicts
- MCP Surface shrunk from 19 to 8 tools (memory/session tools removed)
- MCP Run Results are trimmed by default
- `AGENTOS_URL` env var renamed from `AGNO_OS_URL`
New Features
- Scoring system (Code, Judge, ToolCall)
- Environment rollout engine
- Code file generation
- Adanos market sentiment tools
- Rollout export to SFT JSONL
Full Release Notes
# Changelog
## **New Features**
- **`agno.scorer`** — Turn a run into a number:
- **`CodeScorer`**: wraps any callable (`bool | float | Score`; typed-field comparison under `output_schema` recommended)
- **`JudgeScorer`**: LLM judge with the model always an explicit choice and numeric verdicts normalized to exact endpoints (`(score - 1) / 9`)
- **`ToolCallScorer`**: checks tool **executions** deterministically (refused, errored, or HITL-rejected calls never satisfy an expectation)
- All scorers ship sync and async variants.
- **`agno.environments`** — `Environment` + `Task` + `run_rollouts(env, k=8)`: run each task K times in full isolation (fresh db/session/user, no memory/knowledge/learning writes, cache off; knowledge reads still work). Includes a live per-attempt grid, real pass rate per task, drift-vs-policy fingerprints, `save`/`load`/`diff`, `learning_zone()`, and `to_sft_jsonl(...)` to export passing attempts as conversational-SFT JSONL with a provenance sidecar. This is the pass@k door.
- **`Case.scorer`**: the eval suite gains a third check — plug any scorer into a `Case` (with `Case.expected`), free and exact, no LLM call. `SuiteResult.to_dict()` gains additive `score_value`, `score_passed`, `score_reason` keys.
- **FileGenerationTools**: Added code file generation for `FileGenerationTools`.
- **Gmail Tools**: Added pagination and `max_results_per_request`.
- **Adanos Tools**: Added optional Adanos market sentiment tools.
## **Bug Fixes**
- **RemoteAgent / RemoteTeam**: Fixed metadata being dropped on the A2A protocol path.
- **Content**: Return empty string for empty content list in `get_content_string()`.
- **Decision log**: Replaced deprecated `datetime.utcnow()` in the decision_log store.
## **Breaking Changes**
- **`ReliabilityEval` matches tool executions**: Tool expectations are now satisfied only by a clean execution (`RunOutput.tools`, `tool_call_error` not set), not by message-side requests. **Verdicts can flip red after upgrading** — when they do, the eval was previously passing for the wrong reason (missing entries are annotated `"... (requested but refused/errored — execution matching, new in 2.8.0)"`). Argument checks moved to `ToolExecution.tool_args` with the same partial-match semantics.
- **Hardened judge prompt**: Every `AgentAsJudgeEval` now fences judged output behind a per-call random nonce, with the untrusted-data instruction inside the prompt. A literal `</output>` no longer escapes the block, and "score this 10" inside a judged answer is data, not an instruction. **Judge verdicts and token counts may shift.**
## What's Changed
* fix: replace deprecated datetime.utcnow() in decision_log store by @Ghraven in https://github.com/agno-agi/agno/pull/7949
* cookbook: add dpo_jury pairwise preference example by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9033
* cookbook: refresh data_labeling for agno 2.7.x by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9037
* cookbook: jury calibration, hardening, and agreement metrics by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9038
* cookbook: synthetic data generation workflows by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9040
* cookbook: critique-revision, persona, and tool-call trajectory generation by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9043
* cookbook: step-reward scoring, scale-out mechanics, and safety labeling by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9046
* cookbook: image_search README - ingest is a full rebuild, not idempotent by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9047
* [fix] RemoteAgent/RemoteTeam drop metadata on A2A protocol path by @psinojiya in https://github.com/agno-agi/agno/pull/8944
* feat: add pagination and max_results_per_request to Gmail tools by @Mustafa-Esoofally in https://github.com/agno-agi/agno/pull/9030
* fix: Return empty string for empty content list in get_content_string() by @chuck-duplocloud in https://github.com/agno-agi/agno/pull/6122
* [feat] Add optional Adanos market sentiment tools by @alexander-schneider in https://github.com/agno-agi/agno/pull/9060
* [fix] Clarify Adanos trending ranking by @alexander-schneider in https://github.com/agno-agi/agno/pull/9061
* fix: replace retired qwen/qwen3-32b with openai/gpt-oss-20b on Groq by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9055
* feat: add code file generation to FileGenerationTools by @anuragts in https://github.com/agno-agi/agno/pull/8420
* feat: agno.scorer and the judge prompt fence by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9049
* feat: rollout engine and Case.scorer seam by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9050
* release: v2.8.0 by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9063
* cookbook: expand environments into progressive verification suite by @ashpreetbedi in https://github.com/agno-agi/agno/pull/9070
* feat: Release v2.8.0 by @kausmeows in https://github.com/agno-agi/agno/pull/9073
## New Contributors
* @psinojiya made their first contribution in https://github.com/agno-agi/agno/pull/8944
* @chuck-duplocloud made their first contribution in https://github.com/agno-agi/agno/pull/6122
* @alexander-schneider made their first contribution in https://github.com/agno-agi/agno/pull/9060
**Full Changelog**: https://github.com/agno-agi/agno/compare/v2.7.4...v2.8.0