v1.0.6

feyninc/chonkiev1.0.6Apr 28, 2025by chonknick

AI Summary

This release introduces advanced chunking strategies including the agentic `SlumberChunker` and neural `NeuralChunker`, alongside the new `Genie` framework for API integration.

Key Highlights

  • Introduction of `SlumberChunker`, an agentic chunker requiring the `genie` optional install.
  • Introduction of `NeuralChunker`, a fast, high-quality BERT-like chunker requiring the `neural` optional install.
  • Added `auto` language detection for `CodeChunker` using the `Magika` library.
  • Introduction of the `Genie` framework to handle multiple APIs and models, starting with `GeminiGenie`.

New Features

  • SlumberChunker
  • NeuralChunker
  • CodeChunker auto-detection
  • Genie framework
  • GeminiGenie

Full Release Notes

## ✨ Highlights

1. Welcome Chonkie's very own agentic chunker, `SlumberChunker`! Requires the `genie` optional install to work. `Genie` is Chonkie's Generative Inference Engines, that allow any generative models or API to easily plug-in with Chonkie. Currently, the `genie` optional install installs dependencies for `GeminiGenie` — and will require a `GEMINI_API_KEY` to work properly. 

```bash
pip install "chonkie[genie]"
```

```python
# Import
from chonkie import SlumberChunker

# Initialize
chunker  = SlumberChunker(verbose=True) # set verbose to True, since it takes a while~

# CHONK!
chunker(text) 
```

2. A fully neural approach to chunking, `NeuralChunker`! Requires the `neural` optional install to work. This uses a BERT-like model that's fine-tuned for chunking, making it really fast and high-quality. Second only to `SlumberChunker` in terms of chunk quality. 

```bash
pip install "chonkie[neural]"
```

```python
# import 
from chonkie import NeuralChunker

# initialize
chunker = NeuralChunker()

# CHONK!
chunks = chunker(text) 
```

3. Added `auto` language detection for `CodeChunker`! Now you can just pass in code to it without having to specify the language before hand. It will detect the language by itself. While the latency for detection is minimal (sub-millisecond), it does affect performance, so please specify the language if you care.

```python
# Import
from chonkie import CodeChunker

# Initialize the "auto" CodeChunker
chunker = CodeChunker() # No need to specify, "auto" by default

# CHONK!
chunks = chunker(code) 
```

4. Added `Genie`s! Since this version features our first generative feature, the `SlumberChunker`, we added the `Genie` to work with it as well as future generative features. `Genie`'s are chonkie's way to handle multiple APIs and model interfaces working together for chunking~ The first `Genie` to be added is the `GeminiGenie` — which works with `Gemini` models. Requires the `genie` optional install. 

```bash
pip install "chonkie[genie]"
```

```python
# Import 
from chonkie import GeminiGenie

# Init
genie = GeminiGenie(api_key=YOUR_API_KEY)

# generate
genie.generate("Hi!") 

# generate JSON
genie.generate_json("Hi", JSON_SCHEMA) 
```

## What's Changed
* Feat : update test workflow to run on prs even before merging by @not-lain in https://github.com/chonkie-inc/chonkie/pull/69
* Feat: Run CI/CD on PRs as well by @chonknick in https://github.com/chonkie-inc/chonkie/pull/70
* Feat: Add support for auto-detecting language in `CodeChunker` via `Magika` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/71
* Feat: Add initial support for `SlumberChunker` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/73
* Feat: Add `auto` language support for `CodeChunker` + Add initial support for `SlumberChunker` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/74
* Update version to 1.0.6a0 in pyproject.toml and __init__.py for upcom… by @chonknick in https://github.com/chonkie-inc/chonkie/pull/75
* Fix: Resolve pickling error in BaseTokenizer by replacing lambda by @Harsha-Karimikonda in https://github.com/chonkie-inc/chonkie/pull/76
* Fix: `SlumberChunker`'s splitting algorithm from `RecursiveChunker` to custom + Pickling issue in `BaseTokenizer` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/78
* Feat: Add initial support for the `NeuralChunker` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/77
* Feat: Add support for `NeuralChunker` + Bump version to `v1.0.6` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/79
* Update README.md by @chonknick in https://github.com/chonkie-inc/chonkie/pull/80

## New Contributors
* @not-lain made their first contribution in https://github.com/chonkie-inc/chonkie/pull/69

**Full Changelog**: https://github.com/chonkie-inc/chonkie/compare/v1.0.5...v1.0.6