v1.0.3

feyninc/chonkiev1.0.3Apr 14, 2025by chonknick

AI Summary

Introduced a visualization tool for debugging chunk quality and a recipe system for multilingual and document-specific chunking.

Key Highlights

  • Introduction of `Visualizer` for terminal and HTML visualization of chunks.
  • Added `Recipes` for multilingual support (en, hi, zh, jp, ko) and document types (e.g., markdown).
  • Performance enhancements for `RecursiveChunker`, `SentenceChunker`, and `WordTokenizer`.

New Features

  • Visualizer
  • Recipes

Full Release Notes

## ✨ Highlights

* The new Chonkie `Visualizer` is here! You can now view your chunks, understand chunk quality and debug your chunker with visual feedback~ Use the `print` method to print rich text on your terminal or use the `save` method to save a highlighted `html` on your device! It's very simple to use, just pass in your chunks~

```python
from chonkie import Visualizer

viz = Visualizer()

# Print the chunks on the terminal with .print or directly call the Visualizer object too
viz.print(chunks) 

# Save the HTML file
viz.save("chonkie.html", chunks)
```

<img width="1028" alt="image" src="https://github.com/user-attachments/assets/ea38959b-01bd-4be9-90de-6442542e98a0" />


* Chonkie now adds support for `Recipes` which allow you to use multilingual chunking out of the box, as well as document specific chunking methods. Initial support starts with: `en`, `hi`, `zh`, `jp` and `ko`, while document type `markdown` is supported too. Use it via the `from_recipe` class method with any chunker that takes delimiters or `RecursiveRules`.

```python
from chonkie import RecursiveChunker

# Initialize the recursive chunker to chunk Markdown
chunker = RecursiveChunker.from_recipe("markdown", lang="en")

# Initialize the recursive chunker to chunk Hindi texts
chunker = RecursiveChunker.from_recipe(lang="hi")
```
* Additional fixes for performance enhancements in `RecursiveChunker`, `SentenceChunker`, and `WordTokenizer`

## What's Changed
* Fix: Refactor `SentenceChunker` to remove estimate + feedback by @chonknick in https://github.com/chonkie-inc/chonkie/pull/23
* Fix: `RecursiveRules.from_dict()` showing `Key error: 'levels' does not exist` because of `.pop()` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/24
* Fix: Remove `.find` for indexing in the `RecursiveChunker` for better efficiency by @Pratik960 in https://github.com/chonkie-inc/chonkie/pull/26
* Feat: Add Initial support for `Recipes`  and multilingual CHONKs! by @chonknick in https://github.com/chonkie-inc/chonkie/pull/27
* Fix: `RecursiveChunker` whitespace splitting is not reconstructable (missing spaces) + (#26, #27) by @chonknick in https://github.com/chonkie-inc/chonkie/pull/28
* Update version to 1.0.3a1 in pyproject.toml and __init__.py for Chonkie by @chonknick in https://github.com/chonkie-inc/chonkie/pull/29
* Fix: `WordTokenizer.count_tokens` should use `.tokenize` instead of `.encode` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/36
* Add Chonkie Vizard — easy chunk visualization with `Visualizer` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/39
* Feat: Add Chonkie Vizard to main! `Visualizer` class for easy chunk visualizations by @chonknick in https://github.com/chonkie-inc/chonkie/pull/40
* Fix: Add `chonkie.utils` module to package list in `pyproject.toml` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/41
* Fix: `chonkie.cloud` does not contain `chunkers` error + bump up version to `v1.0.3` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/43

## New Contributors
* @Pratik960 made their first contribution in https://github.com/chonkie-inc/chonkie/pull/26

**Full Changelog**: https://github.com/chonkie-inc/chonkie/compare/v1.0.2...v1.0.3