v0.9.0.13
NVIDIA/garakv0.9.0.13May 10, 2024by leondz
AI Summary
Marks the project's 1st birthday with major additions including multimodal support (LLaVa), LiteLLM generator, DoNotAnswer probe, and NIM generators. Adds parallel OpenAI calls and enhanced Windows compatibility.
Key Highlights
- Multimodal support (LLaVa + FigStep)
- LiteLLM generator support
- DoNotAnswer probe
- NIM (NVIDIA Inference Microservices) generators
- Parallel OpenAI calls support
Breaking Changes
- Drop support for Python 3.9 (now supports 3.10-3.12)
New Features
- Multimodal jailbreaking attack
- LiteLLM integration
- DoNotAnswer probe
- LangChain Serve generator
- Windows compatibility
Full Release Notes
## What's Changed garak's 1st birthday! 🎂 Headlines in this birthday release: * Multimodal support! LLaVa + FigStep - HUGE thanks to @DavidLee528 * LiteLLM generator support - thanks to @Tien-Cheng! * DoNotAnswer probe for prompts an LLM should decline - thanks to @AhsanAyub! * LangChain Serve generator support - thanks to @GustavFredrikson! * Support for NIM generators, NVIDIA Inference Microservices * OpenAI abstraction with parallelisation support * Windows compatibility enhanced, with optional testing in the workflow * Hugging Face speedup ### plugins * LiteLLM generator by @Tien-Cheng in https://github.com/leondz/garak/pull/572 * Support for using 'langchain serve' endpoints by @GustavFredrikson in https://github.com/leondz/garak/pull/588 * Enable parallel OpenAI calls by @jmartin-tech in https://github.com/leondz/garak/pull/645 * Multi-modal Jailbreaking Attack on LLaVA by @DavidLee528 in https://github.com/leondz/garak/pull/587 * bump openai module version to match paths in latest litellm by @leondz in https://github.com/leondz/garak/pull/664 * generator: NIM by @leondz in https://github.com/leondz/garak/pull/637 * Probe: Do Not Answer by @AhsanAyub in https://github.com/leondz/garak/pull/608 ### architecture * change supported pythons to 3.10-3.12 by @leondz in https://github.com/leondz/garak/pull/503 * add more detailed TAP docs by @leondz in https://github.com/leondz/garak/pull/504 * add multiple-result aggregator by @leondz in https://github.com/leondz/garak/pull/505 * add post buff hook by @erickgalinkin in https://github.com/leondz/garak/pull/506 * Add fleshed-out docs to all probes by @leondz in https://github.com/leondz/garak/pull/507 * add bibtex under citation info in readme by @leondz in https://github.com/leondz/garak/pull/511 * define broad test, all probes, 1 gen per by @leondz in https://github.com/leondz/garak/pull/514 * Feature/taxonomy payloads by @leondz in https://github.com/leondz/garak/pull/519 * include paraphrasing in broad conf by @leondz in https://github.com/leondz/garak/pull/521 * choose whether buffing will also include the original prompt by @leondz in https://github.com/leondz/garak/pull/523 * add config var for capping max # buffed prompts to add per buff by @leondz in https://github.com/leondz/garak/pull/526 * document Probe.probe(); skip a buff hook if no buffs by @leondz in https://github.com/leondz/garak/pull/527 * add type hints to base.Probe; fix base probe rst by @leondz in https://github.com/leondz/garak/pull/528 * Bump datasets package by @shubhobm in https://github.com/leondz/garak/pull/536 * Add ConversationalPipeline for huggingface models by @erickgalinkin in https://github.com/leondz/garak/pull/539 * add generator for supporting openai module v0.x by @leondz in https://github.com/leondz/garak/pull/553 * Update README.md by @erickgalinkin in https://github.com/leondz/garak/pull/558 * Minor typo in FAQ by @jmartin-tech in https://github.com/leondz/garak/pull/562 * Add additional error message when __doc__ is None type by @DavidLee528 in https://github.com/leondz/garak/pull/566 * shared constant & string literal by @jmartin-tech in https://github.com/leondz/garak/pull/571 * Spelling corrections for multiple locations by @jmartin-tech in https://github.com/leondz/garak/pull/564 * Reduce Huggingface GPU utilization by @erickgalinkin in https://github.com/leondz/garak/pull/567 * skip `verbose` flag in secondary parser by @jmartin-tech in https://github.com/leondz/garak/pull/576 * Added project twitter link and corrected a grammatic error by @codebrain001 in https://github.com/leondz/garak/pull/578 * Convert GGML to expect GGUF format by @jmartin-tech in https://github.com/leondz/garak/pull/581 * Update workflows: CLA asst bump, PR & manual testing by @leondz in https://github.com/leondz/garak/pull/591 * add test de-duping using skip-duplicate-actions by @leondz in https://github.com/leondz/garak/pull/597 * Remove `#!` entries from files not intended as executables by @jmartin-tech in https://github.com/leondz/garak/pull/612 * Further align shebangs with code that has executable entry points by @leondz in https://github.com/leondz/garak/pull/613 * interactive mode intro by @leondz in https://github.com/leondz/garak/pull/614 * add tests for `ggml` generator by @jmartin-tech in https://github.com/leondz/garak/pull/618 * add var for generator context_len and populate this for some generators by @leondz in https://github.com/leondz/garak/pull/616 * allow generators.Base.generate() to take an optional param specifying generation count by @leondz in https://github.com/leondz/garak/pull/600 * Enable windows tests as github action by @jmartin-tech in https://github.com/leondz/garak/pull/626 * add on-demand macos testing by @leondz in https://github.com/leondz/garak/pull/631 * macOS test install from correct path by @jmartin-tech in https://github.com/leondz/garak/pull/633 * consolidate test file cleanup by @jmartin-tech in https://github.com/leondz/garak/pull/634 * bump discord link by @leondz in https://github.com/leondz/garak/pull/648 * meta the arguments a bit for GET vs other request types by @jmartin-tech in https://github.com/leondz/garak/pull/640 ### stability * fail gracefully if nvcf rejects input; compact zalgo prompts by @leondz in https://github.com/leondz/garak/pull/509 * log & skip past NVCF 4xx errors by @leondz in https://github.com/leondz/garak/pull/533 * fix empty autodan prompts & poor detector behaviour by @leondz in https://github.com/leondz/garak/pull/534 * Fix AutoDAN issues by @erickgalinkin in https://github.com/leondz/garak/pull/537 * fix bad nonetype handling in atkgen probe by @leondz in https://github.com/leondz/garak/pull/538 * Division by zero error fixed in HTML report generation by @CoderMayhem in https://github.com/leondz/garak/pull/545 * cap cohere lib version by @leondz in https://github.com/leondz/garak/pull/569 * rm deprecated model from example by @leondz in https://github.com/leondz/garak/pull/575 * Attack fixes by @erickgalinkin in https://github.com/leondz/garak/pull/555 * More regex as string literal by @jmartin-tech in https://github.com/leondz/garak/pull/586 * Bugfix/action dedupe by @leondz in https://github.com/leondz/garak/pull/598 * wrap cli exec to gracefully catch keyboard exit signal by @jmartin-tech in https://github.com/leondz/garak/pull/603 * Enforce warning output for `garak` classes. by @jmartin-tech in https://github.com/leondz/garak/pull/605 * bump hf transformers v to avoid transformers#30076 by @leondz in https://github.com/leondz/garak/pull/636 * update avidtools to remove typing reference by @jmartin-tech in https://github.com/leondz/garak/pull/639 * torch v bump by @leondz in https://github.com/leondz/garak/pull/649 * Pause FigStepTiny by @leondz in https://github.com/leondz/garak/pull/652 * Bugfix/visual jailbreak pause by @leondz in https://github.com/leondz/garak/pull/653 * limit push test to main by @jmartin-tech in https://github.com/leondz/garak/pull/661 * Update MANIFEST.in so all resources are installed by @JKL98ISR in https://github.com/leondz/garak/pull/660 * handle extant but closed `hitlogfile` file by @leondz in https://github.com/leondz/garak/pull/665 ## New Contributors * @CoderMayhem made their first contribution in https://github.com/leondz/garak/pull/545 * @jmartin-tech made their first contribution in https://github.com/leondz/garak/pull/562 * @codebrain001 made their first contribution in https://github.com/leondz/garak/pull/578 * @Tien-Cheng made their first contribution in https://github.com/leondz/garak/pull/572 * @GustavFredrikson made their first contribution in https://github.com/leondz/garak/pull/588 * @JKL98ISR made their first contribution in https://github.com/leondz/garak/pull/660 * @AhsanAyub made their first contribution in https://github.com/leondz/garak/pull/608 **Full Changelog**: https://github.com/leondz/garak/compare/v0.9.0.12...v0.9.0.13