Starstruk Logo MarkStarstruk

0.2.1

juliangarnier/anime0.2.1Jun 4, 2024by h2o-ops

AI Summary

A feature-rich release for h2oGPT adding extensive vision model support (Claude, Gemini, GPT-4), image handling improvements, Qdrant vector store integration, and enhanced JSON mode and function calling capabilities.

Key Highlights

  • Added support for multiple vision models (Claude-3, GPT-4-Vision, Gemini-Pro-Vision).
  • Integrated Qdrant vector store.
  • Enhanced JSON mode and function calling for MistralAI.
  • Added OpenAI Proxy TTS and STT support.
  • Improved image handling for various vision models.

New Features

  • Vision model support (Claude, GPT-4, Gemini)
  • Qdrant vector store integration
  • JSON mode
  • Function calling
  • TTS and STT support

Full Release Notes

Official Release for h2oGPT 0.2.1

## What's Changed
* Update Mac One click installer Mar 08, 2024 by @Mathanraj-Sharma in https://github.com/h2oai/h2ogpt/pull/1456
* Update gradio constraint to 4.20.1 by @Mathanraj-Sharma in https://github.com/h2oai/h2ogpt/pull/1457
* General chat_template handling + Sealion by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1463
* Update linux install script by @Mathanraj-Sharma in https://github.com/h2oai/h2ogpt/pull/1462
* CohereForAI/c4ai-command-r-v01 by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1467
* Set OpenAI proxy port to 5001 for MacOS by @Mathanraj-Sharma in https://github.com/h2oai/h2ogpt/pull/1468
* Do map[0] instead of map_reduce if all fits into context by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1475
* Image change style by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1444
* Fix history with images by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1479
* Support claude-3 as vision models by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1480
* Add gpt-4-vision support as vision model by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1481
* Support gemini-vision-pro as vision model by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1482
* [DOCS] Correct typos in FAQ and improve readability by @zainhaq-h2o in https://github.com/h2oai/h2ogpt/pull/1487
* Google auth by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1486
* Handle multiple images for gpt4-vision-preview, gemini-pro-vision and claude-3's by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1490
* [DOCS] Fix typos on Links page by @zainhaq-h2o in https://github.com/h2oai/h2ogpt/pull/1488
* clean-up use of grclient by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1493
* feat: Qdrant vector store by @Anush008 in https://github.com/h2oai/h2ogpt/pull/1460
* [Snyk] Fix for 3 vulnerabilities by @smg478 in https://github.com/h2oai/h2ogpt/pull/1498
* [Snyk] Fix for 15 vulnerabilities by @smg478 in https://github.com/h2oai/h2ogpt/pull/1501
* Fix trust by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1505
* Makelist by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1511
* Pass number of prompt tokens and prompt_raw back by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1514
* Handle multiple images for llava by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1516
* fix: Ignore Qdrant scroll offset gpt_langchain.py by @Anush008 in https://github.com/h2oai/h2ogpt/pull/1524
* JSON mode by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1527
* Gradio 4.25.0 by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1510
* Fix grounded template token counting by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1533
* Fix llava token counting by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1534
* Fixdocker by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1538
* Update docker_build_script_ubuntu.sh by @achraf-mer in https://github.com/h2oai/h2ogpt/pull/1541
* Check and version by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1542
* Back to gradio 4.20.1, 4.25.0 really bad in terms of speed and overall stressed performance.  Eventually hangs server too easily by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1546
* Restore gradio 4.26.0 but no heartbeat by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1562
* Repair json if required, also pass back raw response without extraction by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1568
* Faster auth access using sqlite3 instead of full json load/change every minor operation by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1569
* Stream in async for summary/extract by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1575
* Isolate JSON prompts so can change language etc. by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1581
* Ensure llama-3 or other chat template based models handled by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1588
* Clean-up stopping to avoid hard-coded things for llama-3 as it was fixed 11 days ago. by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1590
* remove vllm-check/tgi-check init-container by @robinliubin in https://github.com/h2oai/h2ogpt/pull/1605
* Together.ai support and remove old chroma migration by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1607
* [HELM] Fixes - Add Args when running h2oGPT only by @EshamAaqib in https://github.com/h2oai/h2ogpt/pull/1610
* Improve split and merge by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1612
* [DOCS] Minor FAQ improvements by @zainhaq-h2o in https://github.com/h2oai/h2ogpt/pull/1613
* At least provide rules even if no schema by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1620
* Add OpenAI Proxy TTS by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1621
* set podSecurityContext to null, so umbrella can overwrite on openshift by @robinliubin in https://github.com/h2oai/h2ogpt/pull/1618
* guided_whitespace_pattern by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1625
* OpenAI proxy STT by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1622
* Refactor gradio tools to isolate non-gradio functions.  Fix audio streaming for TTS through OpenAI.  WIP for direct OpenAI nochat call without gradio. by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1543
* Add support for idefics2 vision model via TGI client by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1629
* Put file lock as deep as possible to avoid over locking by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1640
* Function server by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1641
* Use gunicorn so dead workers restart unlike uvicorn by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1645
* Cogvlm2 by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1651
* Fix asyncio sglang use by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1654
* Repair json work around by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1658
* Add function calling for mistralai for better json mode by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1659

## New Contributors
* @Anush008 made their first contribution in https://github.com/h2oai/h2ogpt/pull/1460
* @smg478 made their first contribution in https://github.com/h2oai/h2ogpt/pull/1498

**Full Changelog**: https://github.com/h2oai/h2ogpt/compare/0.2.0...0.2.1