Starstruk Logo MarkStarstruk

v1.0.8

feyninc/chonkiev1.0.8May 22, 2025by chonknick

AI Summary

Expands API compatibility and adds full support for cloud chunkers and URI-based model selection.

Key Highlights

  • Added `base_url` support for OpenAIEmbeddings to use compatible API services.
  • Added URI support for `AutoEmbeddings` to easily switch between providers (e.g., model2vec, sentence_transformers).
  • Full support for `NeuralChunker` and `SlumberChunker` in the cloud module.

New Features

  • OpenAI-compatible API support.
  • URI-based model selection.
  • Cloud support for NeuralChunker and SlumberChunker.

Full Release Notes

## ✨ Highlights

* Use `base_url` with `OpenAIEmbeddings` to use OpenAI API compatible embedding services!
* You can now provide the `AutoEmbeddings` a URI string with the alias to choose different providers for a model simply and easily. Just do `AutoEmbeddings.get_embedding("model2vec://minishlab/potion-base-8M")` or equivalently for the sentence_transformers version do `AutoEmbeddings.get_embedding("st://minishlab/potion-base-8M")`. This would work with all the supported embeddings in Chonkie. As Chonkie grows, it would support various providers and these notations help you choose between them quite easily. 
* Added full support for the `chonkie.cloud` chunkers with updated support for `NeuralChunker` and `SlumberChunker`. 

## What's Changed
* Tutorial: Add example to show `SlumberChunker` with `OpenRouter` models by @chonknick in https://github.com/chonkie-inc/chonkie/pull/103
* Tutorial: fix the tutorial to remove the `Invalid Notebook` error by @chonknick in https://github.com/chonkie-inc/chonkie/pull/104
* Tutorial: Update the readability of the tutorial with better markdown  by @chonknick in https://github.com/chonkie-inc/chonkie/pull/105
* Feat: update workflow by @not-lain in https://github.com/chonkie-inc/chonkie/pull/94
* Feat: Enhance the CI/CD Pipeline to have linting run in parallel  by @chonknick in https://github.com/chonkie-inc/chonkie/pull/106
* Fix: Linting errors + Add `cloud.SDPMChunker` + `cloud.LateChunker`  by @chonknick in https://github.com/chonkie-inc/chonkie/pull/107
* Feat: Add the `base_url` to the `OpenAIEmbeddings` and `**kwargs` support by @chonknick in https://github.com/chonkie-inc/chonkie/pull/108
* Add `RAGHub` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/111
* Fix: Better `AutoEmbeddings` model matching + URI support for providers by @chonknick in https://github.com/chonkie-inc/chonkie/pull/146
* Feat: Add `URI` support via `provider://organization/model` style strings in `AutoEmbeddings` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/148
* Enhance chunker module by adding NeuralChunker and SlumberChunker to the imports and __all__ list for improved functionality. by @chonknick in https://github.com/chonkie-inc/chonkie/pull/150
* Minor: fix error messages in `cloud.NeuralChunker` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/152
* Fix: Add `NeuralChunker` and `SlumberChunker` to the chonkie.cloud by @chonknick in https://github.com/chonkie-inc/chonkie/pull/151


**Full Changelog**: https://github.com/chonkie-inc/chonkie/compare/v1.0.7...v1.0.8