v0.10.0
NVIDIA/garakv0.10.0Oct 31, 2024by jmartin-tech
AI Summary
Adds 'LLM as a Judge' detector, O1 model support for OpenAI, Ollama generator, Azure OpenAI support, and Vision NIMs. Introduces payload/probe separation and various documentation updates.
Key Highlights
- LLM as a Judge detector
- O1 model support for OpenAI generator
- Ollama generator support
- Azure OpenAI support
- Vision NIM support
- Payload/probe separation
New Features
- LLM as a Judge detector
- O1 model support
- Ollama generator
- Azure OpenAI support
- Vision NIMs
- Payload/probe separation
Full Release Notes
## What's Changed ### New plugins * detector: llm as a judge by @jmartin-tech in https://github.com/leondz/garak/pull/956 * generator: update `openai` for o1 models by @leondz in https://github.com/leondz/garak/pull/922 * generator: Ollama by @martinebl in https://github.com/leondz/garak/pull/876 * generator: support of azure openai by @eric-therond in https://github.com/leondz/garak/pull/817 * generator: vision nims by @leondz in https://github.com/leondz/garak/pull/959 * probe: add whois injection to latentinjection by @leondz in https://github.com/leondz/garak/pull/947 ### New features * payload/probe separation by @leondz in https://github.com/leondz/garak/pull/870 * data file override support with precedence by @jmartin-tech in https://github.com/leondz/garak/pull/916 * cli: `--parallel_attempts` hint by @leondz in https://github.com/leondz/garak/pull/932 * reporting: tidy config details, add payload audit info by @leondz in https://github.com/leondz/garak/pull/936 * payloads: web-related, incl. `probes.encoding` migration by @leondz in https://github.com/leondz/garak/pull/933 ### Documentation * Document configuration of garak using YAML by @leondz in https://github.com/leondz/garak/pull/911 * RST formatting updates to restgenerator page by @leondz in https://github.com/leondz/garak/pull/912 * doc: give generatiors.rest list the space it wants and deserves by @leondz in https://github.com/leondz/garak/pull/914 * latent injection doc fix by @leondz in https://github.com/leondz/garak/pull/921 * Contributing Probes documentation by @erickgalinkin in https://github.com/leondz/garak/pull/919 * docs: add example of using `Configurable` to specify a generator name for `load_plugin` by @leondz in https://github.com/leondz/garak/pull/929 * docs: NVIDIA generators by @leondz in https://github.com/leondz/garak/pull/917 * update bag reference by @emmanuel-ferdman in https://github.com/leondz/garak/pull/953 * docs: update faq by @leondz in https://github.com/leondz/garak/pull/957 * docs: describe top-level objectives in reference guide by @leondz in https://github.com/leondz/garak/pull/958 * docs: centralise faq by @leondz in https://github.com/leondz/garak/pull/965 * docs: make project scope clear and note that contributors should check it by @leondz in https://github.com/leondz/garak/pull/964 * doc: update to use rst toctree href syntax by @leondz in https://github.com/leondz/garak/pull/967 * docs: clarify "contributing" document by @leondz in https://github.com/leondz/garak/pull/968 * docs: require core config top- and second-level params to be documented by @leondz in https://github.com/leondz/garak/pull/966 ### Tuning & fixes * plugin classes should not access `DEFAULT_PARAMS` by @jmartin-tech in https://github.com/leondz/garak/pull/906 * add test status lozenges for lin/win/osx by @leondz in https://github.com/leondz/garak/pull/903 * utilized the cache to filter `base` classes by @jmartin-tech in https://github.com/leondz/garak/pull/905 * paraphrase fast consistent model device by @jmartin-tech in https://github.com/leondz/garak/pull/898 * Force cache build in automation by @jmartin-tech in https://github.com/leondz/garak/pull/907 * tweak: defer import of slow Llava* classes by @leondz in https://github.com/leondz/garak/pull/908 * add issue and PR templates by @jmartin-tech in https://github.com/leondz/garak/pull/909 * tweak: new latentinjection probe, add configurability to mini version by @leondz in https://github.com/leondz/garak/pull/910 * bug: set match type correctly in `riskywords` detectors by @leondz in https://github.com/leondz/garak/pull/918 * bugfix: enable latent injection mini latent jailbreak probe by default by @leondz in https://github.com/leondz/garak/pull/923 * tweak: update migitation strings by @leondz in https://github.com/leondz/garak/pull/925 * bugfix: use distinct report entry type for payload init by @leondz in https://github.com/leondz/garak/pull/930 * add code coverage options by @jmartin-tech in https://github.com/leondz/garak/pull/934 * reporting: update link to bag by @leondz in https://github.com/leondz/garak/pull/935 * probe: missing newline in `latentinjection.LatentInjectionFactSnippetEiffel` by @leondz in https://github.com/leondz/garak/pull/938 * update: extend calibration with more probes by @leondz in https://github.com/leondz/garak/pull/939 * limit project actions to primary repo by @jmartin-tech in https://github.com/leondz/garak/pull/940 * Add error handling for empty `node_results` to `base.py`. by @erickgalinkin in https://github.com/leondz/garak/pull/942 * update: rename `replay` plugins to `divergence` by @leondz in https://github.com/leondz/garak/pull/945 * update hint to reference --parallel_attempts by @jmartin-tech in https://github.com/leondz/garak/pull/951 * Configurable TAP probe, refactor judge resources to shared red team by @jmartin-tech in https://github.com/leondz/garak/pull/949 * architecture: factor HFCompatible out by @leondz in https://github.com/leondz/garak/pull/954 * generators: fix rasa issues #961 & #962 by @rgstephens in https://github.com/leondz/garak/pull/963 * fix: add garak/data to pip package by @jmartin-tech in https://github.com/leondz/garak/pull/969 ## New Contributors * @martinebl made their first contribution in https://github.com/leondz/garak/pull/876 * @emmanuel-ferdman made their first contribution in https://github.com/leondz/garak/pull/953 * @eric-therond made their first contribution in https://github.com/leondz/garak/pull/817 **Full Changelog**: https://github.com/leondz/garak/compare/v0.9.0.16...v0.10.0