chunkr-ai-v0.0.44
lumina-ai-inc/chunkrchunkr-ai-v0.0.44Mar 28, 2025by github-actions[bot]
AI Summary
The AI package gains the ability to configure chunking with custom HuggingFace tokenizers.
Key Highlights
- Chunking can now be configured using `embed_sources` in `segment_processing.{segment_type}`
- Support for custom HuggingFace tokenizers via the `tokenizer` field
New Features
- Configurable chunking with custom tokenizers
Full Release Notes
## [0.0.44](https://github.com/lumina-ai-inc/chunkr/compare/chunkr-ai-v0.0.43...chunkr-ai-v0.0.44) (2025-03-28)
### Features
* Chunking can now be configured using `embed_sources` in `segment_processing.{segment_type}` configuration and allows the choice of pre-configured tokenizers or any huggingface tokenizer by setting the `tokenizer` field in `chunk_processing` ([#420](https://github.com/lumina-ai-inc/chunkr/issues/420)) ([d88ac64](https://github.com/lumina-ai-inc/chunkr/commit/d88ac646ece3935f1c7fcd028bb6c5df0b7d00d3))