v1.9.0

lumina-ai-inc/chunkrv1.9.0Mar 28, 2025by github-actions[bot]

AI Summary

This release introduces configurable chunking settings and updates the caching mechanism for better performance.

Key Highlights

Chunking can now be configured using `embed_sources` in `segment_processing.{segment_type}`
Support for custom HuggingFace tokenizers via the `tokenizer` field
Updated hashmap to LRU for improved caching efficiency

New Features

Configurable chunking with custom tokenizers

Full Release Notes

## [1.9.0](https://github.com/lumina-ai-inc/chunkr/compare/v1.8.2...v1.9.0) (2025-03-28)


### Features

* Chunking can now be configured using `embed_sources` in `segment_processing.{segment_type}` configuration and allows the choice of pre-configured tokenizers or any huggingface tokenizer by setting the `tokenizer` field in `chunk_processing` ([#420](https://github.com/lumina-ai-inc/chunkr/issues/420)) ([d88ac64](https://github.com/lumina-ai-inc/chunkr/commit/d88ac646ece3935f1c7fcd028bb6c5df0b7d00d3))


### Bug Fixes

* Updated hashmap to lru for caching ([d868c76](https://github.com/lumina-ai-inc/chunkr/commit/d868c76dd16a6751e3baab43190b81e827e26395))