v1.12.0

anthropics/anthropic-tokenizer-typescriptv1.12.0Jul 30, 2024by adbar

AI Summary

Performance-focused release enforcing fixed output formats, improving extraction accuracy through baseline improvements, and fixing various bugs in CLI, images, XML-TEI, and table handling.

Key Highlights

Fixed list of output formats enforced on CLI
Faster and more accurate extraction
Improved justext fallback mechanism
Baseline prevents LXML error in JSON-LD
Review of XPaths for undesirable content

Breaking Changes

Enforce fixed list of output formats, deprecate -out on CLI

New Features

Enhanced JSON-LD handling
Improved markdown formatting
XML-TEI RelaxNG replaced with DTD

Full Release Notes

Breaking change:
- enforce fixed list of output formats, deprecate `-out` on the CLI (#647)

Faster, more accurate extraction:
- review link and structure checks (#653)
- improve justext fallback (#652)
- baseline: prevent LXML error in JSON-LD (#643), do not use as backup extraction (#646)
- review XPaths for undesirable content (#645)

Bugfixes and maintenance:
- CLI fix: markdown format should trigger `include_formatting` (#649)
- images fix: use a length threshold on src attribute (#654)
- XML-TEI: replace RelaxNG by DTD, remove pickle, and update (#655)
- formatting & markdown fix: add newlines (#656)
- table fix: prevent `MemoryError` & `ValueError` during conversion to text (#658)

Documentation:
- update `crawls.rst`: `known` is an unexpected argument, by @tommytyc in #638