v1.0.4
feyninc/chonkiev1.0.4Apr 21, 2025by chonknick
AI Summary
Added support for code-specific chunking, Jina AI embeddings, and new refineries to manage chunk overlap and embeddings.
Key Highlights
- Introduction of `CodeChunker` supporting 100+ programming languages.
- Added `JinaEmbeddings` support for `SemanticChunker` and `SDPMChunker`.
- Added `OverlapRefinery` to add overlap context to chunks.
- Added `EmbeddingsRefinery` to generate embeddings for downstream vector database loading.
New Features
- CodeChunker
- JinaEmbeddings
- OverlapRefinery
- EmbeddingsRefinery
Full Release Notes
## ✨ Highlights
1. Welcome our newest chunker to the family: `CodeChunker`! `CodeChunker` is specialized to handle code files and can gain structural understanding of the code before chunking each portion separately~ Supports 100+ programming languages! Let's check out the usage~
Firstly, install the code chunker dependencies via:
```bash
pip install "chonkie[code]"
```
and then simply run it like any other chunker~
```python
# Import
from chonkie import CodeChunker
# Init
chunker = CodeChunker(language="python")
# Get some python code
code = ...
# CHONK!
chunks = chunker(code)
```
2. Added support for `JinaAI` embeddings with `JinaEmbeddings` — allowing for use with `SemanticChunker` and `SDPMChunker`!
Install it via the following command:
```bash
pip install "chonkie[jina]"
```
and use it like this~
```python
# Import
from chonkie import JinaEmbeddings
from chonkie import SemanticChunker
# Optionally import Visaulizer for visalizations
from chonkie import Visualizer()
# Init
viz = Visualizer()
embeddings = JinaEmbeddings()
chunker = SemanticChunker(embeddings)
# Get some text
text = ...
# CHONK!
chunks = chunker(text)
# Optional: Visualize!
viz(chunks)
```
3. Added support for the `OverlapRefinery` which allows you to add overlap context to your chunks~ It's available in the default install and can be used with any chunker. Just chunk with a chunker and pass your chunks through the refinery!
```python
# import
from chonkie import RecursiveChunker, OverlapRefinery
# Init
chunker = RecursiveChunker()
refinery = OverlapRefinery("gpt2") # By default initializes with "character" tokenizer, can pass in "gpt2" to match the chunker
# Get some text
text = ...
# CHONK and Refine!
chunks = chunker(text)
chunks = refinery(chunks)
```
4. Added support for the `EmbeddingsRefinery` which allows you to run the chunks through a embedding model and have the embeddings available for downstream loading in a vector database. Similar to the `OverlapRefinery` just pass in the `chunks` from a `Chunker` into a `EmbeddingsRefinery` object loaded with the appropriate embedding model and each `Chunk` will then be loaded with `.embedding` value which can be used downstream.
## What's Changed
* Add initial support for chunking code via `CodeChunker` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/53
* Add Initial support for code chunking via `CodeChunker` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/54
* Feat: Add Jina AI Embeddings support by @Harsha-Karimikonda in https://github.com/chonkie-inc/chonkie/pull/35
* Feat: Add support for JinaEmbeddings + OverlapRefinery + EmbeddingsRefinery by @chonknick in https://github.com/chonkie-inc/chonkie/pull/57
* Fix: Paths for the Chonkie Cloud chunkers; `No module named 'chonkie.cloud.chunkers'` error by @chonknick in https://github.com/chonkie-inc/chonkie/pull/58
* Fix: Attempt fixing `chonkie.cloud` path bug + update README.md to have integrations by @chonknick in https://github.com/chonkie-inc/chonkie/pull/60
## New Contributors
* @Harsha-Karimikonda made their first contribution in https://github.com/chonkie-inc/chonkie/pull/35
**Full Changelog**: https://github.com/chonkie-inc/chonkie/compare/v1.0.3...v1.0.4