Starstruk Logo MarkStarstruk

v0.11.0

NVIDIA/garakv0.11.0May 28, 2025by jmartin-tech

AI Summary

Adds Template/SQL injection probes, Mistral generator support, and multilingual machine translation capabilities. The release also introduces probe tiers and refines reporting aggregation methods.

Key Highlights

  • Template / SQL injection probes
  • Mistral generator support
  • Multilingual machine translation support
  • Introduction of probe tiers

New Features

  • Template/SQL injection probes
  • Mistral generator
  • Multilingual translation
  • Probe tiers
  • Improved reporting aggregation

Full Release Notes

## What's Changed

### New plugins
* Template / SQL injection probes by @erickgalinkin in https://github.com/NVIDIA/garak/pull/1138
* Feature/add mistral generator by @dimensi0n in https://github.com/NVIDIA/garak/pull/1135

### New features
* feature: update default toxicity detector by @leondz in https://github.com/NVIDIA/garak/pull/1106
* feature: lightweight probe defaults by @leondz in https://github.com/NVIDIA/garak/pull/1116
* feature: max_workers / give kinda helpful message if too many open files by @leondz in https://github.com/NVIDIA/garak/pull/1110
* Multiprocess enabled logging config by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1140
* Feature: multilingual machine translation by @SnowMasaya in https://github.com/NVIDIA/garak/pull/943
* Support stripping until end think token given empty skip_seq_start in config by @aishwaryap in https://github.com/NVIDIA/garak/pull/1185
* update: add probe tiers by @leondz in https://github.com/NVIDIA/garak/pull/1151
* update: promptinject detector now accepts multiple triggers by @leondz in https://github.com/NVIDIA/garak/pull/1148
* update: rename atkgen probe model to be clear about toxicity by @leondz in https://github.com/NVIDIA/garak/pull/1149
* update: remove ambiguous terms from `slur_terms_en` payload by @leondz in https://github.com/NVIDIA/garak/pull/1150
* reporting: update report aggregation funcs by @leondz in https://github.com/NVIDIA/garak/pull/1156
* script: qualitative review output by @leondz in https://github.com/NVIDIA/garak/pull/1144
* Add -no-cnv flag support to ggml generators by @IanYHChu in https://github.com/NVIDIA/garak/pull/1189
* reporting: add option for no group score by @leondz in https://github.com/NVIDIA/garak/pull/1194
* reporting: aggregate probe as min by @leondz in https://github.com/NVIDIA/garak/pull/1218
* reporting: add defcon lozenges for relative & absolute scores by @leondz in https://github.com/NVIDIA/garak/pull/1216
* Update/refactor specialwords by @leondz in https://github.com/NVIDIA/garak/pull/1178
* reporting: smooth z-score wildness by @leondz in https://github.com/NVIDIA/garak/pull/1212
* Task: 2025 Q2 scoring calibration by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1231 (thanks to [Vijil.ai](https://vijil.ai) for data contributions)
* update calibration data for additional probes by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1236
* reporting: change default aggregation by @leondz in https://github.com/NVIDIA/garak/pull/1234

### Documentation
* Fix typo in README for leak replay probe by @arjun-krishna1 in https://github.com/NVIDIA/garak/pull/1142
* docs: split 'extending' docs out from 'contributing' by @leondz in https://github.com/NVIDIA/garak/pull/1146
* doc file class corrections by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1200
* docs: formatting fixes by @leondz in https://github.com/NVIDIA/garak/pull/1215

### Tuning & fixes
* clear pip cached files by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1129
* set a default soft_probe_prompt_cap in `_config` by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1133
* enhance response type support from local NeMo-Guardrails by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1131
* bugfix: encoding detection generating false positives by @leondz in https://github.com/NVIDIA/garak/pull/1130
* update: unify on `attempt.notes["triggers"]` by @leondz in https://github.com/NVIDIA/garak/pull/1147
* Bump datasets version by @JanetVictorious in https://github.com/NVIDIA/garak/pull/1137
* make all workflow permissions explicit by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1162
* update: add soft prompt caps to encoding probes by @leondz in https://github.com/NVIDIA/garak/pull/1154
* update: rename `bcp47` to `lang` by @leondz in https://github.com/NVIDIA/garak/pull/1164
* one detection result per output when testing regex based matches in `exploitation` by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1167
* Removed detector prefix from eval records by @mrowebot in https://github.com/NVIDIA/garak/pull/1157
* bugfix: HF Detector exceptions now handled gracefully by default by @leondz in https://github.com/NVIDIA/garak/pull/1170
* cache workflow resources by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1173
* refactor probe `tier` as enum with value in plugin cache by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1159
* update: more meaningful values in tier enums by @leondz in https://github.com/NVIDIA/garak/pull/1176
* block failing litellm 1.67.2 by @leondz in https://github.com/NVIDIA/garak/pull/1179
* ux: give more verbose message for CLI typos by @leondz in https://github.com/NVIDIA/garak/pull/1182
* refactor `LatentInjection` by @leondz in https://github.com/NVIDIA/garak/pull/1152
* cap `litellm` max version to avoid their windows bug by @leondz in https://github.com/NVIDIA/garak/pull/1186
* update: rename `Translator` -> `LangProvider` and associated elements by @leondz in https://github.com/NVIDIA/garak/pull/1183
* bugfix: reduce latent optimisation permutation explosion by @leondz in https://github.com/NVIDIA/garak/pull/1181
* replicate generator pickle support improvements by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1190
* Fix ambiguous series value error when running --report by @marcorosa in https://github.com/NVIDIA/garak/pull/1171
* add arm64 runner to Linux testing by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1196
* Testing: storage reduction by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1204
* remove unused tooling to free space by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1206
* update deps away from insecure versions by @leondz in https://github.com/NVIDIA/garak/pull/1207
* update `Tier` impl by @leondz in https://github.com/NVIDIA/garak/pull/1205
* config: sync probe active defaults with default config used in practice by @leondz in https://github.com/NVIDIA/garak/pull/1214
* update: revert default `_config.run.generations` to `5` by @leondz in https://github.com/NVIDIA/garak/pull/1227
* fix: stop `atkgen` turn count variation in test relying on fixed turn count by @leondz in https://github.com/NVIDIA/garak/pull/1226
* fix plugin cache tests by @emmanuel-ferdman in https://github.com/NVIDIA/garak/pull/1229
* ux: move translator load msg into translator instantiation by @leondz in https://github.com/NVIDIA/garak/pull/1184
* extract text when processing multi-modal prompts by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1228

## New Contributors
* @JanetVictorious made their first contribution in https://github.com/NVIDIA/garak/pull/1137
* @SnowMasaya made their first contribution in https://github.com/NVIDIA/garak/pull/943
* @dimensi0n made their first contribution in https://github.com/NVIDIA/garak/pull/1135
* @mrowebot made their first contribution in https://github.com/NVIDIA/garak/pull/1157
* @aishwaryap made their first contribution in https://github.com/NVIDIA/garak/pull/1185
* @marcorosa made their first contribution in https://github.com/NVIDIA/garak/pull/1171
* @IanYHChu made their first contribution in https://github.com/NVIDIA/garak/pull/1189

**Full Changelog**: https://github.com/NVIDIA/garak/compare/v0.10.3.1...v0.11.0