Starstruk Logo MarkStarstruk

v1.5.0

feyninc/chonkiev1.5.0Dec 16, 2025by chonknick

AI Summary

Introduces new tokenization capabilities and Python version support, dropping Python 3.9 support while adding Python 3.13 support.

Key Highlights

  • Dropped support for Python 3.9; added support for Python 3.13
  • Added `RowTokenizer` for chunking tables based on row values
  • Added `ByteTokenizer`
  • Parameter hints now pass through from original objects to pipelines

Breaking Changes

  • Dropped support for Python 3.9

New Features

  • RowTokenizer
  • ByteTokenizer
  • Catsu embeddings support
  • Pipeline parameter hints

Full Release Notes

## ✨ Highlights

* This version removes support for `Py3.9`, and adds support for `Py3.13`. 
* Added a `RowTokenizer` to chunk tables based on row values, thanks to @chonk-lain. 
* Parameter hints now pass through from their original object to the pipelines, thanks to @akx!
* Multiple bug fixes and patches!

<p align="center">
  <img src="https://github.com/user-attachments/assets/65fbfe3a-31d6-4459-b4bb-fe1908009e36" width="50%">
</p>

## What's Changed
* Fix broken pricing link by @shreyash-chonkie in https://github.com/chonkie-inc/chonkie/pull/389
* Feat: Add Pipeline to the cloud client + Add Docs Pages by @chonknick in https://github.com/chonkie-inc/chonkie/pull/390
* Fix: Add update flow if the pipeline already exists on the `run()` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/391
* fix(slumber): preserve spaces in chunk concatenation by @kiendoantrung in https://github.com/chonkie-inc/chonkie/pull/381
* chore: update geminie geinie default parameters by @chonk-lain in https://github.com/chonkie-inc/chonkie/pull/393
* Add support for custom Chonkie tokenizer instances by @chonknick in https://github.com/chonkie-inc/chonkie/pull/396
* Feat: Add `ByteTokenizer` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/397
* Docs: Add byte-level tokenizer to documentation by @chonknick in https://github.com/chonkie-inc/chonkie/pull/398
* Fix: Add ability to wrap AutoTokenizer in another AutoTokenizer instance by @chonknick in https://github.com/chonkie-inc/chonkie/pull/399
* Fix: Deprecate `Py3.9` and add support for `Py3.13`! by @chonknick in https://github.com/chonkie-inc/chonkie/pull/320
* feat: add row based table chunker by @chonk-lain in https://github.com/chonkie-inc/chonkie/pull/392
* Feat: Add Catsu embeddings support to Chonkie by @chonknick in https://github.com/chonkie-inc/chonkie/pull/402
* Fix: Pass-through original object type through pipeline registration functions by @akx in https://github.com/chonkie-inc/chonkie/pull/394

## New Contributors
* @akx made their first contribution in https://github.com/chonkie-inc/chonkie/pull/394

**Full Changelog**: https://github.com/chonkie-inc/chonkie/compare/v1.4.2...v1.5.0