Starstruk Logo MarkStarstruk

0.2.1

TimoBolkart/FLAME-Universe0.2.1Jun 4, 2024by h2o-ops

AI Summary

This release focuses on expanding vision capabilities by adding support for GPT-4 Vision, Claude-3, and Gemini, alongside the introduction of Qdrant vector store and enhanced JSON mode. Performance optimizations include faster auth access and stability fixes for the Gradio interface.

Key Highlights

  • Support for multiple vision models (GPT-4V, Claude-3, Gemini)
  • Addition of Qdrant vector store
  • Improved JSON mode and function calling
  • Faster auth access using SQLite3
  • Enhanced image handling and history support

New Features

  • Qdrant vector store
  • OpenAI Proxy TTS (Text-to-Speech)
  • OpenAI Proxy STT (Speech-to-Text)
  • Function server
  • idefics2 vision model support
  • Together.ai support
  • Async streaming for summary/extract

Full Release Notes

Official Release for h2oGPT 0.2.1

## What's Changed
* Update Mac One click installer Mar 08, 2024 by @Mathanraj-Sharma in https://github.com/h2oai/h2ogpt/pull/1456
* Update gradio constraint to 4.20.1 by @Mathanraj-Sharma in https://github.com/h2oai/h2ogpt/pull/1457
* General chat_template handling + Sealion by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1463
* Update linux install script by @Mathanraj-Sharma in https://github.com/h2oai/h2ogpt/pull/1462
* CohereForAI/c4ai-command-r-v01 by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1467
* Set OpenAI proxy port to 5001 for MacOS by @Mathanraj-Sharma in https://github.com/h2oai/h2ogpt/pull/1468
* Do map[0] instead of map_reduce if all fits into context by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1475
* Image change style by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1444
* Fix history with images by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1479
* Support claude-3 as vision models by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1480
* Add gpt-4-vision support as vision model by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1481
* Support gemini-vision-pro as vision model by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1482
* [DOCS] Correct typos in FAQ and improve readability by @zainhaq-h2o in https://github.com/h2oai/h2ogpt/pull/1487
* Google auth by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1486
* Handle multiple images for gpt4-vision-preview, gemini-pro-vision and claude-3's by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1490
* [DOCS] Fix typos on Links page by @zainhaq-h2o in https://github.com/h2oai/h2ogpt/pull/1488
* clean-up use of grclient by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1493
* feat: Qdrant vector store by @Anush008 in https://github.com/h2oai/h2ogpt/pull/1460
* [Snyk] Fix for 3 vulnerabilities by @smg478 in https://github.com/h2oai/h2ogpt/pull/1498
* [Snyk] Fix for 15 vulnerabilities by @smg478 in https://github.com/h2oai/h2ogpt/pull/1501
* Fix trust by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1505
* Makelist by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1511
* Pass number of prompt tokens and prompt_raw back by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1514
* Handle multiple images for llava by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1516
* fix: Ignore Qdrant scroll offset gpt_langchain.py by @Anush008 in https://github.com/h2oai/h2ogpt/pull/1524
* JSON mode by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1527
* Gradio 4.25.0 by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1510
* Fix grounded template token counting by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1533
* Fix llava token counting by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1534
* Fixdocker by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1538
* Update docker_build_script_ubuntu.sh by @achraf-mer in https://github.com/h2oai/h2ogpt/pull/1541
* Check and version by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1542
* Back to gradio 4.20.1, 4.25.0 really bad in terms of speed and overall stressed performance.  Eventually hangs server too easily by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1546
* Restore gradio 4.26.0 but no heartbeat by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1562
* Repair json if required, also pass back raw response without extraction by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1568
* Faster auth access using sqlite3 instead of full json load/change every minor operation by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1569
* Stream in async for summary/extract by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1575
* Isolate JSON prompts so can change language etc. by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1581
* Ensure llama-3 or other chat template based models handled by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1588
* Clean-up stopping to avoid hard-coded things for llama-3 as it was fixed 11 days ago. by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1590
* remove vllm-check/tgi-check init-container by @robinliubin in https://github.com/h2oai/h2ogpt/pull/1605
* Together.ai support and remove old chroma migration by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1607
* [HELM] Fixes - Add Args when running h2oGPT only by @EshamAaqib in https://github.com/h2oai/h2ogpt/pull/1610
* Improve split and merge by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1612
* [DOCS] Minor FAQ improvements by @zainhaq-h2o in https://github.com/h2oai/h2ogpt/pull/1613
* At least provide rules even if no schema by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1620
* Add OpenAI Proxy TTS by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1621
* set podSecurityContext to null, so umbrella can overwrite on openshift by @robinliubin in https://github.com/h2oai/h2ogpt/pull/1618
* guided_whitespace_pattern by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1625
* OpenAI proxy STT by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1622
* Refactor gradio tools to isolate non-gradio functions.  Fix audio streaming for TTS through OpenAI.  WIP for direct OpenAI nochat call without gradio. by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1543
* Add support for idefics2 vision model via TGI client by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1629
* Put file lock as deep as possible to avoid over locking by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1640
* Function server by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1641
* Use gunicorn so dead workers restart unlike uvicorn by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1645
* Cogvlm2 by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1651
* Fix asyncio sglang use by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1654
* Repair json work around by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1658
* Add function calling for mistralai for better json mode by @pseudotensor in https://github.com/h2oai/h2ogpt/pull/1659

## New Contributors
* @Anush008 made their first contribution in https://github.com/h2oai/h2ogpt/pull/1460
* @smg478 made their first contribution in https://github.com/h2oai/h2ogpt/pull/1498

**Full Changelog**: https://github.com/h2oai/h2ogpt/compare/0.2.0...0.2.1