v1.0.4

feyninc/chonkiev1.0.4Apr 21, 2025by chonknick

AI Summary

Added support for code-specific chunking, Jina AI embeddings, and new refineries to manage chunk overlap and embeddings.

Key Highlights

  • Introduction of `CodeChunker` supporting 100+ programming languages.
  • Added `JinaEmbeddings` support for `SemanticChunker` and `SDPMChunker`.
  • Added `OverlapRefinery` to add overlap context to chunks.
  • Added `EmbeddingsRefinery` to generate embeddings for downstream vector database loading.

New Features

  • CodeChunker
  • JinaEmbeddings
  • OverlapRefinery
  • EmbeddingsRefinery

Full Release Notes

## ✨ Highlights

1. Welcome our newest chunker to the family: `CodeChunker`!  `CodeChunker` is specialized to handle code files and can gain structural understanding of the code before chunking each portion separately~  Supports 100+ programming languages! Let's check out the usage~

Firstly, install the code chunker dependencies via:
```bash
pip install "chonkie[code]"
```
and then simply run it like any other chunker~

```python
# Import
from chonkie import CodeChunker

# Init
chunker = CodeChunker(language="python")

# Get some python code
code = ...

# CHONK!
chunks = chunker(code) 
```

2. Added support for `JinaAI` embeddings with `JinaEmbeddings` — allowing for use with `SemanticChunker` and `SDPMChunker`! 

Install it via the following command: 
```bash
pip install "chonkie[jina]"
```

and use it like this~

```python
# Import 
from chonkie import JinaEmbeddings 
from chonkie import SemanticChunker

# Optionally import Visaulizer for visalizations
from chonkie import Visualizer()

# Init
viz = Visualizer()
embeddings = JinaEmbeddings()
chunker = SemanticChunker(embeddings) 


# Get some text
text = ...

# CHONK!
chunks = chunker(text) 

# Optional: Visualize!
viz(chunks) 
```

3. Added support for the `OverlapRefinery` which allows you to add overlap context to your chunks~ It's available in the default install and can be used with any chunker. Just chunk with a chunker and pass your chunks through the refinery!

```python
# import
from chonkie import RecursiveChunker, OverlapRefinery

# Init
chunker = RecursiveChunker()
refinery = OverlapRefinery("gpt2") # By default initializes with "character" tokenizer, can pass in "gpt2" to match the chunker

# Get some text 
text = ...

# CHONK and Refine!
chunks = chunker(text) 
chunks = refinery(chunks)
```

4. Added support for the `EmbeddingsRefinery` which allows you to run the chunks through a embedding model and have the embeddings available for downstream loading in a vector database. Similar to the `OverlapRefinery` just pass in the `chunks` from a `Chunker` into a `EmbeddingsRefinery` object loaded with the appropriate embedding model and each `Chunk` will then be loaded with `.embedding` value which can be used downstream. 

## What's Changed
* Add initial support for chunking code via `CodeChunker` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/53
* Add Initial support for code chunking via `CodeChunker` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/54
* Feat: Add Jina AI Embeddings support by @Harsha-Karimikonda in https://github.com/chonkie-inc/chonkie/pull/35
* Feat: Add support for JinaEmbeddings + OverlapRefinery + EmbeddingsRefinery by @chonknick in https://github.com/chonkie-inc/chonkie/pull/57
* Fix: Paths for the Chonkie Cloud chunkers; `No module named 'chonkie.cloud.chunkers'` error by @chonknick in https://github.com/chonkie-inc/chonkie/pull/58
* Fix: Attempt fixing `chonkie.cloud` path bug + update README.md to have integrations  by @chonknick in https://github.com/chonkie-inc/chonkie/pull/60

## New Contributors
* @Harsha-Karimikonda made their first contribution in https://github.com/chonkie-inc/chonkie/pull/35

**Full Changelog**: https://github.com/chonkie-inc/chonkie/compare/v1.0.3...v1.0.4