Starstruk Logo MarkStarstruk

0.2.1

h2oai/h2ogpt0.2.1Jun 4, 2024by h2o-ops

AI Summary

This release adds extensive vision model support including Claude-3, GPT-4-Vision, Gemini Vision Pro, Llava, Cogvlm2, and Idefics2. It also introduces Qdrant vector store support, JSON mode, function calling for MistralAI, and various performance improvements including faster auth using SQLite and gunicorn for better worker management.

Key Highlights

  • Vision model support (Claude-3, GPT-4-Vision, Gemini-vision-pro, Llava, Cogvlm2, Idefics2)
  • Qdrant vector store integration
  • JSON mode and function calling for MistralAI
  • Google authentication support
  • Together.ai inference support
  • gunicorn for better server worker management

New Features

  • Support for Claude-3 vision models
  • Support for GPT-4-Vision as vision model
  • Support for Gemini-vision-pro
  • Support for multiple images with GPT4-vision-preview, Gemini-pro-vision, and Claude-3
  • Qdrant vector store support
  • JSON mode with guided_whitespace_pattern
  • Function calling for MistralAI
  • OpenAI Proxy TTS (Text-to-Speech)
  • OpenAI Proxy STT (Speech-to-Text)
  • Google authentication
  • Faster auth using SQLite3 instead of JSON
  • gunicorn instead of uvicorn for dead worker restart
  • Chat template handling for Llama-3 and other models
  • Improved token counting for grounded template and Llava

Full Release Notes

Official Release for h2oGPT 0.2.1

## What's Changed
* Update Mac One click installer Mar 08, 2024 by @Mathanraj-Sharma in https://github.com/h2oai/h2ogpt/pull/1456
* Update gradio constraint to 4.20.1 by @Mathanraj-Sharma in https://github.com/h2oai/h2ogpt/pull/1457
* General chat_template handling + Sealion by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1463
* Update linux install script by @Mathanraj-Sharma in https://github.com/h2oai/h2ogpt/pull/1462
* CohereForAI/c4ai-command-r-v01 by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1467
* Set OpenAI proxy port to 5001 for MacOS by @Mathanraj-Sharma in https://github.com/h2oai/h2ogpt/pull/1468
* Do map[0] instead of map_reduce if all fits into context by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1475
* Image change style by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1444
* Fix history with images by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1479
* Support claude-3 as vision models by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1480
* Add gpt-4-vision support as vision model by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1481
* Support gemini-vision-pro as vision model by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1482
* [DOCS] Correct typos in FAQ and improve readability by @zainhaq-h2o in https://github.com/h2oai/h2ogpt/pull/1487
* Google auth by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1486
* Handle multiple images for gpt4-vision-preview, gemini-pro-vision and claude-3's by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1490
* [DOCS] Fix typos on Links page by @zainhaq-h2o in https://github.com/h2oai/h2ogpt/pull/1488
* clean-up use of grclient by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1493
* feat: Qdrant vector store by @Anush008 in https://github.com/h2oai/h2ogpt/pull/1460
* [Snyk] Fix for 3 vulnerabilities by @smg478 in https://github.com/h2oai/h2ogpt/pull/1498
* [Snyk] Fix for 15 vulnerabilities by @smg478 in https://github.com/h2oai/h2ogpt/pull/1501
* Fix trust by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1505
* Makelist by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1511
* Pass number of prompt tokens and prompt_raw back by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1514
* Handle multiple images for llava by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1516
* fix: Ignore Qdrant scroll offset gpt_langchain.py by @Anush008 in https://github.com/h2oai/h2ogpt/pull/1524
* JSON mode by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1527
* Gradio 4.25.0 by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1510
* Fix grounded template token counting by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1533
* Fix llava token counting by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1534
* Fixdocker by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1538
* Update docker_build_script_ubuntu.sh by @achraf-mer in https://github.com/h2oai/h2ogpt/pull/1541
* Check and version by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1542
* Back to gradio 4.20.1, 4.25.0 really bad in terms of speed and overall stressed performance.  Eventually hangs server too easily by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1546
* Restore gradio 4.26.0 but no heartbeat by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1562
* Repair json if required, also pass back raw response without extraction by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1568
* Faster auth access using sqlite3 instead of full json load/change every minor operation by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1569
* Stream in async for summary/extract by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1575
* Isolate JSON prompts so can change language etc. by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1581
* Ensure llama-3 or other chat template based models handled by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1588
* Clean-up stopping to avoid hard-coded things for llama-3 as it was fixed 11 days ago. by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1590
* remove vllm-check/tgi-check init-container by @robinliubin in https://github.com/h2oai/h2ogpt/pull/1605
* Together.ai support and remove old chroma migration by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1607
* [HELM] Fixes - Add Args when running h2oGPT only by @EshamAaqib in https://github.com/h2oai/h2ogpt/pull/1610
* Improve split and merge by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1612
* [DOCS] Minor FAQ improvements by @zainhaq-h2o in https://github.com/h2oai/h2ogpt/pull/1613
* At least provide rules even if no schema by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1620
* Add OpenAI Proxy TTS by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1621
* set podSecurityContext to null, so umbrella can overwrite on openshift by @robinliubin in https://github.com/h2oai/h2ogpt/pull/1618
* guided_whitespace_pattern by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1625
* OpenAI proxy STT by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1622
* Refactor gradio tools to isolate non-gradio functions.  Fix audio streaming for TTS through OpenAI.  WIP for direct OpenAI nochat call without gradio. by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1543
* Add support for idefics2 vision model via TGI client by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1629
* Put file lock as deep as possible to avoid over locking by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1640
* Function server by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1641
* Use gunicorn so dead workers restart unlike uvicorn by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1645
* Cogvlm2 by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1651
* Fix asyncio sglang use by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1654
* Repair json work around by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1658
* Add function calling for mistralai for better json mode by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1659

## New Contributors
* @Anush008 made their first contribution in https://github.com/h2oai/h2ogpt/pull/1460
* @smg478 made their first contribution in https://github.com/h2oai/h2ogpt/pull/1498

**Full Changelog**: https://github.com/h2oai/h2ogpt/compare/0.2.0...0.2.1