Starstruk Logo MarkStarstruk

v1.0.7

feyninc/chonkiev1.0.7May 6, 2025by chonknick

AI Summary

Introduces core infrastructure for data storage (Handshakes/Porters), new embedding providers, and visualizer themes.

Key Highlights

  • Added `Handshakes` (Chroma, Qdrant, Turbopuffer) and `Porters` (JSONPorter) for data management.
  • Added support for `VoyageAIEmbeddings` and `OpenAIGenie`.
  • Added Dark mode and TikTokenizer theme to the `Visualizer`.

New Features

  • Handshake and Porter infrastructure.
  • VoyageAI and OpenAIGenie embeddings support.
  • Visualizer themes (Dark mode, TikTokenizer).
  • New one-page documentation.

Full Release Notes

## ✨ Highlights

- Added initial support for `Handshakes` and `Porters` with 1 new `Porter` (`JSONPorter`) as well as 3 new `Handshakes` (`ChromaHandshake`, `QdrantHandshake` and `TurbopufferHandshake`)! Extemely simple to use, you can just import, init and call on your `List[Chunks]` that you get out from your chunkers/refineries and query them for search!
- Added support for `OpenAIGenie` which allows you to use the `OpenAI` models with the `SlumberChunker` and many more! Any API that supports the `OpenAI API` format will be usable with the `base_url` changed. Try out various LLMs with the `SlumberChunker` to see which works best for you!
- Added support for `VoyageAIEmbeddings` (thanks to @Pratik960): Use `VoyageAI` models with `EmbeddingsRefinery` and `SemanticChunkers` for your ingestion~
- Added new themes to the `Visualizer`:  Dark mode and a retro `tiktokenizer` theme (thanks to @Udayk02) 
- Fixes and perfomance improvements in the `NeuralChunker` 
- New one-page markdown `DOCS.md` for easy answers about Chonkie with LLMs. 

## What's Changed
* Feat: Add `tiktokenizer` theme to the `Visualizer` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/81
* Feat: Added support for VoyageAI embeddings by @Pratik960 in https://github.com/chonkie-inc/chonkie/pull/67
* Feat: Add initial support for `VoyageAIEmbeddings` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/85
* Fix: Enhance `NeuralChunker` by adding support for multiple models, custom… by @chonknick in https://github.com/chonkie-inc/chonkie/pull/86
* Add the local DOCS.md for consistency and ease by @chonknick in https://github.com/chonkie-inc/chonkie/pull/89
* Update DOCS.md by @chonknick in https://github.com/chonkie-inc/chonkie/pull/90
* Add `BaseHandshake` and `BasePorter` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/91
* Update cookbook by @chonknick in https://github.com/chonkie-inc/chonkie/pull/92
* Feat: Add `ChromaHandshake` + `QdrantHandshake` + `Document` + `Chomp` + More by @chonknick in https://github.com/chonkie-inc/chonkie/pull/93
* Feat: Add support for `OpenAIGenie` + `TurbopufferHandshake` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/95
* Update README.md by @chonknick in https://github.com/chonkie-inc/chonkie/pull/96
* Add warning for experimental status of TurbopufferHandshake in turbop… by @chonknick in https://github.com/chonkie-inc/chonkie/pull/97
* Feat: Viz: Added a new dark theme, separated the light and the dark themes by @Udayk02 in https://github.com/chonkie-inc/chonkie/pull/82
* Feat: Add `dark` mode to chonkie's `Visualizer` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/98
* Update DOCS.md by @chonknick in https://github.com/chonkie-inc/chonkie/pull/99

## New Contributors
* @Udayk02 made their first contribution in https://github.com/chonkie-inc/chonkie/pull/82

**Full Changelog**: https://github.com/chonkie-inc/chonkie/compare/v1.0.6...v1.0.7