0.2.1
Xiaojiu-z/EasyControl0.2.1Jun 4, 2024by h2o-ops
AI Summary
This release focuses on enhancing vision capabilities, integrating Qdrant vector store, and improving OpenAI proxy features for text-to-speech and speech-to-text.
Key Highlights
- Added vision model support (Claude-3, GPT-4 Vision, Gemini Pro Vision)
- Qdrant vector store integration
- OpenAI Proxy TTS and STT support
- Security vulnerability fixes
New Features
- Vision model support for Claude-3, GPT-4 Vision, and Gemini Pro Vision
- Qdrant vector store
- JSON mode implementation
- Function server
- OpenAI Proxy Text-to-Speech (TTS)
- OpenAI Proxy Speech-to-Text (STT)
Full Release Notes
Official Release for h2oGPT 0.2.1 ## What's Changed * Update Mac One click installer Mar 08, 2024 by @Mathanraj-Sharma in https://github.com/h2oai/h2ogpt/pull/1456 * Update gradio constraint to 4.20.1 by @Mathanraj-Sharma in https://github.com/h2oai/h2ogpt/pull/1457 * General chat_template handling + Sealion by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1463 * Update linux install script by @Mathanraj-Sharma in https://github.com/h2oai/h2ogpt/pull/1462 * CohereForAI/c4ai-command-r-v01 by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1467 * Set OpenAI proxy port to 5001 for MacOS by @Mathanraj-Sharma in https://github.com/h2oai/h2ogpt/pull/1468 * Do map[0] instead of map_reduce if all fits into context by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1475 * Image change style by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1444 * Fix history with images by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1479 * Support claude-3 as vision models by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1480 * Add gpt-4-vision support as vision model by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1481 * Support gemini-vision-pro as vision model by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1482 * [DOCS] Correct typos in FAQ and improve readability by @zainhaq-h2o in https://github.com/h2oai/h2ogpt/pull/1487 * Google auth by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1486 * Handle multiple images for gpt4-vision-preview, gemini-pro-vision and claude-3's by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1490 * [DOCS] Fix typos on Links page by @zainhaq-h2o in https://github.com/h2oai/h2ogpt/pull/1488 * clean-up use of grclient by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1493 * feat: Qdrant vector store by @Anush008 in https://github.com/h2oai/h2ogpt/pull/1460 * [Snyk] Fix for 3 vulnerabilities by @smg478 in https://github.com/h2oai/h2ogpt/pull/1498 * [Snyk] Fix for 15 vulnerabilities by @smg478 in https://github.com/h2oai/h2ogpt/pull/1501 * Fix trust by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1505 * Makelist by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1511 * Pass number of prompt tokens and prompt_raw back by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1514 * Handle multiple images for llava by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1516 * fix: Ignore Qdrant scroll offset gpt_langchain.py by @Anush008 in https://github.com/h2oai/h2ogpt/pull/1524 * JSON mode by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1527 * Gradio 4.25.0 by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1510 * Fix grounded template token counting by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1533 * Fix llava token counting by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1534 * Fixdocker by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1538 * Update docker_build_script_ubuntu.sh by @achraf-mer in https://github.com/h2oai/h2ogpt/pull/1541 * Check and version by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1542 * Back to gradio 4.20.1, 4.25.0 really bad in terms of speed and overall stressed performance. Eventually hangs server too easily by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1546 * Restore gradio 4.26.0 but no heartbeat by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1562 * Repair json if required, also pass back raw response without extraction by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1568 * Faster auth access using sqlite3 instead of full json load/change every minor operation by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1569 * Stream in async for summary/extract by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1575 * Isolate JSON prompts so can change language etc. by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1581 * Ensure llama-3 or other chat template based models handled by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1588 * Clean-up stopping to avoid hard-coded things for llama-3 as it was fixed 11 days ago. by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1590 * remove vllm-check/tgi-check init-container by @robinliubin in https://github.com/h2oai/h2ogpt/pull/1605 * Together.ai support and remove old chroma migration by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1607 * [HELM] Fixes - Add Args when running h2oGPT only by @EshamAaqib in https://github.com/h2oai/h2ogpt/pull/1610 * Improve split and merge by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1612 * [DOCS] Minor FAQ improvements by @zainhaq-h2o in https://github.com/h2oai/h2ogpt/pull/1613 * At least provide rules even if no schema by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1620 * Add OpenAI Proxy TTS by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1621 * set podSecurityContext to null, so umbrella can overwrite on openshift by @robinliubin in https://github.com/h2oai/h2ogpt/pull/1618 * guided_whitespace_pattern by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1625 * OpenAI proxy STT by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1622 * Refactor gradio tools to isolate non-gradio functions. Fix audio streaming for TTS through OpenAI. WIP for direct OpenAI nochat call without gradio. by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1543 * Add support for idefics2 vision model via TGI client by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1629 * Put file lock as deep as possible to avoid over locking by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1640 * Function server by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1641 * Use gunicorn so dead workers restart unlike uvicorn by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1645 * Cogvlm2 by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1651 * Fix asyncio sglang use by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1654 * Repair json work around by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1658 * Add function calling for mistralai for better json mode by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1659 ## New Contributors * @Anush008 made their first contribution in https://github.com/h2oai/h2ogpt/pull/1460 * @smg478 made their first contribution in https://github.com/h2oai/h2ogpt/pull/1498 **Full Changelog**: https://github.com/h2oai/h2ogpt/compare/0.2.0...0.2.1