Starstruk Logo MarkStarstruk

v0.9.0.10

NVIDIA/garakv0.9.0.10Dec 20, 2023by leondz

AI Summary

Overhauls reporting with automatic HTML report creation and grouping by tags. Adds new buffs (Base64, CharCode), NeMo Guardrails generator, and AutoDAN probe.

Key Highlights

  • Automatic HTML report creation
  • Grouping reports by probe tag category
  • New buffs: Base64 and CharCode
  • NeMo Guardrails generator
  • AutoDAN probe

Breaking Changes

  • Renamed `atk` module to `atkgen`

New Features

  • HTML report generation
  • Tag selection (MISP tags)
  • Base64 and CharCode buffs
  • NeMo Guardrails generator
  • AutoDAN probe

Full Release Notes

* Probes can now be selected by MISP tag, e.g. owasp:llm01
* garak now automatically creates an HTML report on completion
* HTML reports can be grouped by module but also by probe tag category, so you can see e.g. top-level scores for prompt injection, hallucination, and so on
* logs now go to a dedicated log dir by default, to keep things clean
* new buffs: encoding.Base64, encoding.CharCode
* new generator: [NeMo guardrails](https://github.com/NVIDIA/NeMo-Guardrails)
* new probe: [AutoDAN](https://arxiv.org/abs/2310.04451)
* RealToxicityProbes now only loads local lists, much faster
* update OpenAI models list
* fix attempt parameter stability
* better logging of config params
* atk is now atkgen

Contributions from @erickgalinkin , @drazvan . Enjoy & Happy holidays! 🎅🎄

## What's Changed
* Attempt no longer uses mutable type defaults by @leondz in https://github.com/leondz/garak/pull/360
* Add NeMoGuardrails generator (WIP). by @drazvan in https://github.com/leondz/garak/pull/345
* add test for mutable defaults bug in attempt.Attempt by @leondz in https://github.com/leondz/garak/pull/362
* refresh openai model name list by @leondz in https://github.com/leondz/garak/pull/363
* speed up realtoxicityprompts loading by @leondz in https://github.com/leondz/garak/pull/364
* Feature/digest report 231212 by @leondz in https://github.com/leondz/garak/pull/365
* Autodan by @erickgalinkin in https://github.com/leondz/garak/pull/367
* Auto-reporting by @leondz in https://github.com/leondz/garak/pull/368
* add guardrails doc connection by @leondz in https://github.com/leondz/garak/pull/369
* Feature/digest plugin descrs by @leondz in https://github.com/leondz/garak/pull/370
* Add Base64 and CharCode buffs by @erickgalinkin in https://github.com/leondz/garak/pull/372
* tidy buffs, add test for buff config loading by @leondz in https://github.com/leondz/garak/pull/376
* Feature/tag selection by @leondz in https://github.com/leondz/garak/pull/383
* set default for probe_tags in core config; use this as default cli arg by @leondz in https://github.com/leondz/garak/pull/386
* hitlogs should use same paths as other reporting. add test for this by @leondz in https://github.com/leondz/garak/pull/387
* Feature/reporting categories by @leondz in https://github.com/leondz/garak/pull/389

## New Contributors
* @drazvan made their first contribution in https://github.com/leondz/garak/pull/345

**Full Changelog**: https://github.com/leondz/garak/compare/v0.9.0.9...v0.9.0.10