v0.9.0.14

NVIDIA/garakv0.9.0.14Jul 29, 2024by leondz

AI Summary

Adds BEAST Attack Implementation and Grandma attack probes, along with configurable plugins and Windows/OSX support. Includes JSON Path support for the REST generator and support for GPT-4o models.

Key Highlights

  • BEAST Attack Implementation probe
  • Grandma attack probe
  • Configurable plugins feature
  • Windows & OSX support
  • JSON Path support for RestGenerator

New Features

  • BEAST Attack probe
  • Grandma attack probe
  • Configurable plugins
  • GPT-4o and GPT-4o-mini support
  • Windows/OSX testing support

Full Release Notes

## What's Changed

### New plugins
* BEAST Attack Implementation by @erickgalinkin in https://github.com/leondz/garak/pull/728
* add probe for "grandma" attack by @leondz in https://github.com/leondz/garak/pull/740
* generator: lorem ipsum by @leondz in https://github.com/leondz/garak/pull/776

### New features
* Feature: configurable plugins by @jmartin-tech in https://github.com/leondz/garak/pull/711
* Add JSON Path support for RestGenerator by @leondz in https://github.com/leondz/garak/pull/651
* Make banner line test dynamic by @jmartin-tech in https://github.com/leondz/garak/pull/670
* Tutorial for contributing a generator by @leondz in https://github.com/leondz/garak/pull/688
* add 'complete' message to report at end of run by @leondz in https://github.com/leondz/garak/pull/735
* support chat modality in attempts by @leondz in https://github.com/leondz/garak/pull/644
* add grandma substance synth probe, update docs and var names by @leondz in https://github.com/leondz/garak/pull/763
* update NVCF to support versioning, custom payload keys by @leondz in https://github.com/leondz/garak/pull/772
* WINDOWS & OSX SUPPORT: test by default on linux, osx, win by @leondz in https://github.com/leondz/garak/pull/793
* enable nested setting of plugin configs by @leondz in https://github.com/leondz/garak/pull/773
* always log nvcf payload by @leondz in https://github.com/leondz/garak/pull/778
* update ecoji for windows support by @jmartin-tech in https://github.com/leondz/garak/pull/787
* remove extraneous default params for nims that expect conservative pa… by @leondz in https://github.com/leondz/garak/pull/749
* require probes to match input modality of the generator by @jmartin-tech in https://github.com/leondz/garak/pull/738
* Improve cloze data for `leakreplay` probe by @leondz in https://github.com/leondz/garak/pull/708
* add CONTRIBUTING.md, update contributing guide in docs by @leondz in https://github.com/leondz/garak/pull/706
* add DEFAULT_PARAMS to base types that are configurables by @leondz in https://github.com/leondz/garak/pull/771
* atkgen - add configurability, allow config of which adversarial generator to use by @leondz in https://github.com/leondz/garak/pull/777
* enhancement: enable lazy setting of nested dicts by @leondz in https://github.com/leondz/garak/pull/775
* convert mutation_generator to use `load_plugin` by @jmartin-tech in https://github.com/leondz/garak/pull/795
* Add support for gpt-4o, gpt-4o-mini by @erickgalinkin in https://github.com/leondz/garak/pull/798
* bump transformers ver to min reqd for llama 3.1 by @leondz in https://github.com/leondz/garak/pull/800
* add mitigation strings, add test for string-based mitigation skipping by @leondz in https://github.com/leondz/garak/pull/762
* speedup report generation by using class attribs instead of instantiating by @leondz in https://github.com/leondz/garak/pull/707
* cached plugin enum by @jmartin-tech in https://github.com/leondz/garak/pull/768

### Tuning & fixes
* add some actionable opinions about linting by @leondz in https://github.com/leondz/garak/pull/668
* handle target model returning None in atkgen by @leondz in https://github.com/leondz/garak/pull/678
* bump & fix black version by @leondz in https://github.com/leondz/garak/pull/686
* Bump/black 24.4.2 -- and apply formatting by @leondz in https://github.com/leondz/garak/pull/687
* convert module-level constant names to uppercase by @leondz in https://github.com/leondz/garak/pull/685
* Guard cli run with invalid or incomplete config by @jmartin-tech in https://github.com/leondz/garak/pull/677
* Catch generic exception from transformers in huggingface by @jmartin-tech in https://github.com/leondz/garak/pull/690
* stablize openai parallel by @jmartin-tech in https://github.com/leondz/garak/pull/689
* rest generator response must be a list by @jmartin-tech in https://github.com/leondz/garak/pull/693
* Set generator `_call_model()` and `_generate()` type hints; amend functions by @leondz in https://github.com/leondz/garak/pull/694
* typo in racial slur prompt by @leondz in https://github.com/leondz/garak/pull/697
* Generator streamlining, docs by @leondz in https://github.com/leondz/garak/pull/682
* Split NVCF generator into completion chat by @leondz in https://github.com/leondz/garak/pull/696
* Update the quality taxonomy by @leondz in https://github.com/leondz/garak/pull/703
* generator & detector structure validation by @leondz in https://github.com/leondz/garak/pull/702
* doc & inheritance fixes to DanInTheWildMini by @leondz in https://github.com/leondz/garak/pull/712
* Remove support for openai api v0 by @leondz in https://github.com/leondz/garak/pull/718
* restore `req_template_json_object` lost in refactor by @jmartin-tech in https://github.com/leondz/garak/pull/729
* utilize GarakException in cli by @jmartin-tech in https://github.com/leondz/garak/pull/730
* ensure report is printed in `utf-8` for Windows by @jmartin-tech in https://github.com/leondz/garak/pull/732
* Linting - 20240612 by @leondz in https://github.com/leondz/garak/pull/733
* add garak preprint paper by @leondz in https://github.com/leondz/garak/pull/739
* only set generations if still default by @jmartin-tech in https://github.com/leondz/garak/pull/741
* return value from `batch_decode` is already a list by @jmartin-tech in https://github.com/leondz/garak/pull/743
* Handle openai 400 errors more gracefully by @leondz in https://github.com/leondz/garak/pull/744
* Refactor `huggingface` config support by @jmartin-tech in https://github.com/leondz/garak/pull/742
* rename class level probe doc_uri by @jmartin-tech in https://github.com/leondz/garak/pull/759
* retry openaicompatible requests if invalid content received by @leondz in https://github.com/leondz/garak/pull/761
* add colour theme config by @leondz in https://github.com/leondz/garak/pull/769
* reduce zalgo-ness by @leondz in https://github.com/leondz/garak/pull/770
* move REST generator exception into garak.exceptions by @leondz in https://github.com/leondz/garak/pull/779
* defensive coding: allow python generators more places by @leondz in https://github.com/leondz/garak/pull/782
* stop encoding.payloads accumulating each load by @leondz in https://github.com/leondz/garak/pull/780
* correct sorting for set json in plugin_cache.json by @jmartin-tech in https://github.com/leondz/garak/pull/794
* corrections to config format for TAP internal generators by @jmartin-tech in https://github.com/leondz/garak/pull/796
* add missing comma, extra test case in probes.test.Test by @leondz in https://github.com/leondz/garak/pull/797
* Bugfix/leakreplay output setting by @leondz in https://github.com/leondz/garak/pull/790


**Full Changelog**: https://github.com/leondz/garak/compare/v0.9.0.13...v0.9.0.14