v0.9.0.14
NVIDIA/garakv0.9.0.14Jul 29, 2024by leondz
AI Summary
Adds BEAST Attack Implementation and Grandma attack probes, along with configurable plugins and Windows/OSX support. Includes JSON Path support for the REST generator and support for GPT-4o models.
Key Highlights
- BEAST Attack Implementation probe
- Grandma attack probe
- Configurable plugins feature
- Windows & OSX support
- JSON Path support for RestGenerator
New Features
- BEAST Attack probe
- Grandma attack probe
- Configurable plugins
- GPT-4o and GPT-4o-mini support
- Windows/OSX testing support
Full Release Notes
## What's Changed ### New plugins * BEAST Attack Implementation by @erickgalinkin in https://github.com/leondz/garak/pull/728 * add probe for "grandma" attack by @leondz in https://github.com/leondz/garak/pull/740 * generator: lorem ipsum by @leondz in https://github.com/leondz/garak/pull/776 ### New features * Feature: configurable plugins by @jmartin-tech in https://github.com/leondz/garak/pull/711 * Add JSON Path support for RestGenerator by @leondz in https://github.com/leondz/garak/pull/651 * Make banner line test dynamic by @jmartin-tech in https://github.com/leondz/garak/pull/670 * Tutorial for contributing a generator by @leondz in https://github.com/leondz/garak/pull/688 * add 'complete' message to report at end of run by @leondz in https://github.com/leondz/garak/pull/735 * support chat modality in attempts by @leondz in https://github.com/leondz/garak/pull/644 * add grandma substance synth probe, update docs and var names by @leondz in https://github.com/leondz/garak/pull/763 * update NVCF to support versioning, custom payload keys by @leondz in https://github.com/leondz/garak/pull/772 * WINDOWS & OSX SUPPORT: test by default on linux, osx, win by @leondz in https://github.com/leondz/garak/pull/793 * enable nested setting of plugin configs by @leondz in https://github.com/leondz/garak/pull/773 * always log nvcf payload by @leondz in https://github.com/leondz/garak/pull/778 * update ecoji for windows support by @jmartin-tech in https://github.com/leondz/garak/pull/787 * remove extraneous default params for nims that expect conservative pa… by @leondz in https://github.com/leondz/garak/pull/749 * require probes to match input modality of the generator by @jmartin-tech in https://github.com/leondz/garak/pull/738 * Improve cloze data for `leakreplay` probe by @leondz in https://github.com/leondz/garak/pull/708 * add CONTRIBUTING.md, update contributing guide in docs by @leondz in https://github.com/leondz/garak/pull/706 * add DEFAULT_PARAMS to base types that are configurables by @leondz in https://github.com/leondz/garak/pull/771 * atkgen - add configurability, allow config of which adversarial generator to use by @leondz in https://github.com/leondz/garak/pull/777 * enhancement: enable lazy setting of nested dicts by @leondz in https://github.com/leondz/garak/pull/775 * convert mutation_generator to use `load_plugin` by @jmartin-tech in https://github.com/leondz/garak/pull/795 * Add support for gpt-4o, gpt-4o-mini by @erickgalinkin in https://github.com/leondz/garak/pull/798 * bump transformers ver to min reqd for llama 3.1 by @leondz in https://github.com/leondz/garak/pull/800 * add mitigation strings, add test for string-based mitigation skipping by @leondz in https://github.com/leondz/garak/pull/762 * speedup report generation by using class attribs instead of instantiating by @leondz in https://github.com/leondz/garak/pull/707 * cached plugin enum by @jmartin-tech in https://github.com/leondz/garak/pull/768 ### Tuning & fixes * add some actionable opinions about linting by @leondz in https://github.com/leondz/garak/pull/668 * handle target model returning None in atkgen by @leondz in https://github.com/leondz/garak/pull/678 * bump & fix black version by @leondz in https://github.com/leondz/garak/pull/686 * Bump/black 24.4.2 -- and apply formatting by @leondz in https://github.com/leondz/garak/pull/687 * convert module-level constant names to uppercase by @leondz in https://github.com/leondz/garak/pull/685 * Guard cli run with invalid or incomplete config by @jmartin-tech in https://github.com/leondz/garak/pull/677 * Catch generic exception from transformers in huggingface by @jmartin-tech in https://github.com/leondz/garak/pull/690 * stablize openai parallel by @jmartin-tech in https://github.com/leondz/garak/pull/689 * rest generator response must be a list by @jmartin-tech in https://github.com/leondz/garak/pull/693 * Set generator `_call_model()` and `_generate()` type hints; amend functions by @leondz in https://github.com/leondz/garak/pull/694 * typo in racial slur prompt by @leondz in https://github.com/leondz/garak/pull/697 * Generator streamlining, docs by @leondz in https://github.com/leondz/garak/pull/682 * Split NVCF generator into completion chat by @leondz in https://github.com/leondz/garak/pull/696 * Update the quality taxonomy by @leondz in https://github.com/leondz/garak/pull/703 * generator & detector structure validation by @leondz in https://github.com/leondz/garak/pull/702 * doc & inheritance fixes to DanInTheWildMini by @leondz in https://github.com/leondz/garak/pull/712 * Remove support for openai api v0 by @leondz in https://github.com/leondz/garak/pull/718 * restore `req_template_json_object` lost in refactor by @jmartin-tech in https://github.com/leondz/garak/pull/729 * utilize GarakException in cli by @jmartin-tech in https://github.com/leondz/garak/pull/730 * ensure report is printed in `utf-8` for Windows by @jmartin-tech in https://github.com/leondz/garak/pull/732 * Linting - 20240612 by @leondz in https://github.com/leondz/garak/pull/733 * add garak preprint paper by @leondz in https://github.com/leondz/garak/pull/739 * only set generations if still default by @jmartin-tech in https://github.com/leondz/garak/pull/741 * return value from `batch_decode` is already a list by @jmartin-tech in https://github.com/leondz/garak/pull/743 * Handle openai 400 errors more gracefully by @leondz in https://github.com/leondz/garak/pull/744 * Refactor `huggingface` config support by @jmartin-tech in https://github.com/leondz/garak/pull/742 * rename class level probe doc_uri by @jmartin-tech in https://github.com/leondz/garak/pull/759 * retry openaicompatible requests if invalid content received by @leondz in https://github.com/leondz/garak/pull/761 * add colour theme config by @leondz in https://github.com/leondz/garak/pull/769 * reduce zalgo-ness by @leondz in https://github.com/leondz/garak/pull/770 * move REST generator exception into garak.exceptions by @leondz in https://github.com/leondz/garak/pull/779 * defensive coding: allow python generators more places by @leondz in https://github.com/leondz/garak/pull/782 * stop encoding.payloads accumulating each load by @leondz in https://github.com/leondz/garak/pull/780 * correct sorting for set json in plugin_cache.json by @jmartin-tech in https://github.com/leondz/garak/pull/794 * corrections to config format for TAP internal generators by @jmartin-tech in https://github.com/leondz/garak/pull/796 * add missing comma, extra test case in probes.test.Test by @leondz in https://github.com/leondz/garak/pull/797 * Bugfix/leakreplay output setting by @leondz in https://github.com/leondz/garak/pull/790 **Full Changelog**: https://github.com/leondz/garak/compare/v0.9.0.13...v0.9.0.14