Starstruk Logo MarkStarstruk

v0.15.0

a16z-infra/llm-app-stackv0.15.0May 1, 2026by jmartin-tech

AI Summary

This update focuses on adding new security probes and detectors for modern AI models, including support for Agent testing and ModernBERT refusal detection. It also includes improvements to the reporting system and documentation structure.

Key Highlights

  • New 'Agent breaker' probe to test tools available to target systems.
  • Support for ModernBERT refusal detection and NeMoGuardrails server generation.
  • Enhanced mTLS client certificate authentication for REST generators.
  • Improved reporting with snapshot plugin cache into JSONL format.
  • Documentation restructured into a tree format.

New Features

  • Multi-turn GOAT probe.
  • System prompt extraction probe.
  • Homoglyph obfuscation prompts in smuggling probe.
  • Bedrock generator support for reasoning models.
  • Snapshot plugin cache into report JSONL.

Full Release Notes

## What's Changed

### New plugins
* probes: Add multi-turn GOAT probe by @saichandrapandraju in https://github.com/NVIDIA/garak/pull/1424
* probes: Agent breaker by @eliyacohen-hub in https://github.com/NVIDIA/garak/pull/1628
* probes: Added system prompt extraction probe by @Nakul-Rajpal in https://github.com/NVIDIA/garak/pull/1538
* probes: add homoglyph obfuscation prompts to smuggling probe by @dentity007 in https://github.com/NVIDIA/garak/pull/1660
* detectors: Add ModernBERT refusal detector by @patriciapampanelli in https://github.com/NVIDIA/garak/pull/1650
* generators: NeMoGuardrails server support by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1675
* generators: test generator with reasoning traces by @leondz in https://github.com/NVIDIA/garak/pull/1697

### New features
* Agent breaker probe by @eliyacohen-hub in https://github.com/NVIDIA/garak/pull/1628 adds support for testing tools available to target systems. 

### Improved plugins
* generators: add mTLS client certificate authentication to REST generator by @JakeBx in https://github.com/NVIDIA/garak/pull/1681
* generators: Add support for reasoning models to Bedrock generator by @TheTeaCat in https://github.com/NVIDIA/garak/pull/1642
* detectors: limit tensor length passed to roberta by @leondz in https://github.com/NVIDIA/garak/pull/1715
* probes: Enforce soft_probe_prompt_cap in GCGCached probe by @precognitivem0nk in https://github.com/NVIDIA/garak/pull/1665
* detectors: fix logic flip bug in encoding detection, add tests to catch by @leondz in https://github.com/NVIDIA/garak/pull/1711

### Documentation
* docs: structure docs out of flat dir into tree by @leondz in https://github.com/NVIDIA/garak/pull/1662
* docs: explicitly enumerate configurable params in docs by @leondz in https://github.com/NVIDIA/garak/pull/1683

### Tuning & fixes
* fix: prevent semaphore leak by @r3v5 in https://github.com/NVIDIA/garak/pull/1657
* Replace base2048 with mikeshardmind-base2048 and add spec tests by @onceandtwice in https://github.com/NVIDIA/garak/pull/1518
* fix: remove duplicate status:2 attempt writes in TreeSearchProbe by @Christbowel in https://github.com/NVIDIA/garak/pull/1655
* task: dependency updates by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1678
* fix: report_digest -t taxonomy option not working by @HayatoFujihara in https://github.com/NVIDIA/garak/pull/1672
* fix: use magic.from_file() instead of from_buffer() for libmagic 5.47 compatibility by @HayatoFujihara in https://github.com/NVIDIA/garak/pull/1671
* housekeeping: prune phantom rapidfuzz dep by @leondz in https://github.com/NVIDIA/garak/pull/1701
* housekeeping: supply hook for signing, document signing prefs by @leondz in https://github.com/NVIDIA/garak/pull/1702
* first version of AGENTS.md by @leondz in https://github.com/NVIDIA/garak/pull/1698
* bug: avoid TBSA double horizontal rule by @leondz in https://github.com/NVIDIA/garak/pull/1712
* Handle unrecognized probes and detectors in report digest generation by @precognitivem0nk in https://github.com/NVIDIA/garak/pull/1663
* report: snapshot plugin cache into report jsonl by @patriciapampanelli in https://github.com/NVIDIA/garak/pull/1723
* docs: add build test for docs by @leondz in https://github.com/NVIDIA/garak/pull/1713
* improve compatibility with older reports by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1728

## New Contributors
* @r3v5 made their first contribution in https://github.com/NVIDIA/garak/pull/1657
* @onceandtwice made their first contribution in https://github.com/NVIDIA/garak/pull/1518
* @TheTeaCat made their first contribution in https://github.com/NVIDIA/garak/pull/1642
* @Christbowel made their first contribution in https://github.com/NVIDIA/garak/pull/1655
* @HayatoFujihara made their first contribution in https://github.com/NVIDIA/garak/pull/1672
* @precognitivem0nk made their first contribution in https://github.com/NVIDIA/garak/pull/1665
* @JakeBx made their first contribution in https://github.com/NVIDIA/garak/pull/1681
* @eliyacohen-hub made their first contribution in https://github.com/NVIDIA/garak/pull/1628
* @dentity007 made their first contribution in https://github.com/NVIDIA/garak/pull/1660

**Full Changelog**: https://github.com/NVIDIA/garak/compare/v0.14.1...v0.15.0