Starstruk Logo MarkStarstruk

v1.0.9

feyninc/chonkiev1.0.9Jun 4, 2025by chonknick

AI Summary

Performance optimizations via Cython and the addition of support for PostgreSQL vector databases.

Key Highlights

  • Added Cython functions for `split` and `merge` operations for performance boosts.
  • Added `PsycopgHandshake` for initial support of `pgvector`.
  • Added `experimental.CodeChunker` for improved code chunking.

New Features

  • Cython optimization for basic ops.
  • PostgreSQL (pgvector) support.
  • Experimental code chunker.
  • Gemini embeddings support.

Full Release Notes

## What's Changed
* Fix pickling issue in BaseTokenizer by @chonknick in https://github.com/chonkie-inc/chonkie/pull/153
* Improve test coverage for OverlapRefinery by @chonknick in https://github.com/chonkie-inc/chonkie/pull/154
* Feat: Enhance the tests for `OverlapRefinery`, `Tokenizers` and `SlumberChunker` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/155
* Feat: Enhance the tests for `SPDMChunker`, `SemanticChunker`, `NeuralChunker` and `Visualizer` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/156
* Feat: Add cloud `CodeChunker` + tests by @chonknick in https://github.com/chonkie-inc/chonkie/pull/157
* Fix: `CodeChunker` fails multiprocessing due to pickling issues; Run it sequentially by @chonknick in https://github.com/chonkie-inc/chonkie/pull/158
* Add GeminiEmbeddings Support by @chonknick in https://github.com/chonkie-inc/chonkie/pull/159
* Fix: OverlapRefinery with `mode="recursive"` fails with level gets too low by @chonknick in https://github.com/chonkie-inc/chonkie/pull/161
* Fix: `OverlapRefinery` is caching too much; revert `context_size` calculation caching by @chonknick in https://github.com/chonkie-inc/chonkie/pull/162
* Feat: Create Cython functions for `split` and `merge` basic ops for chunking! by @chonknick in https://github.com/chonkie-inc/chonkie/pull/163
* Feat: Add CPython optimized methods for `split` and `merge` ops for performance boosts! by @chonknick in https://github.com/chonkie-inc/chonkie/pull/164
* Migrate Docs from `chonkie-docs` repo to `chonkie` for easier management + maintainance by @chonknick in https://github.com/chonkie-inc/chonkie/pull/149
* Migrate DOCS.md by @chonknick in https://github.com/chonkie-inc/chonkie/pull/165
* Update DOCS by @chonknick in https://github.com/chonkie-inc/chonkie/pull/166
* Update DOCS.md + Add example for `JSONPorter` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/167
* Feat: Add support for `PsycopgHandshake` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/171
* Feat: Add `PsycopgHandshake` for initial support of `pgvector` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/172
* Update semantic.py to fix  check if any of the split token counts are greater than the max chunk size by @geosmart in https://github.com/chonkie-inc/chonkie/pull/175
* Fix: Refactor `PyscopgHandshake` to `PgvectorHandshake` via `vecs`  by @chonknick in https://github.com/chonkie-inc/chonkie/pull/174
* Update DOCS.md + Fix `SemanticChunker` bug for `all` comparison by @chonknick in https://github.com/chonkie-inc/chonkie/pull/176
* Update version to 1.0.9 in pyproject.toml for the next release. by @chonknick in https://github.com/chonkie-inc/chonkie/pull/177
* Feat: Add initial support for `experimental.CodeChunker` — improved code chunking for few languages by @chonknick in https://github.com/chonkie-inc/chonkie/pull/178
* Fix: `cython` module build failing during CD — Add `cython` build info to `pyproject.toml` by @chonknick in https://github.com/chonkie-inc/chonkie/pull/180
* Fix: Use `cibuildwheel` to build and publish wheels to PyPI by @chonknick in https://github.com/chonkie-inc/chonkie/pull/182
* Fix: Update the CD script to use `uv` instead in the hopes of finally publishing the `v1.0.9`  by @chonknick in https://github.com/chonkie-inc/chonkie/pull/183

## New Contributors
* @geosmart made their first contribution in https://github.com/chonkie-inc/chonkie/pull/175

**Full Changelog**: https://github.com/chonkie-inc/chonkie/compare/v1.0.8...v1.0.9