v1.10.0

adbar/trafilaturav1.10.0May 30, 2024by adbar

AI Summary

This release focuses on stability and code organization. It introduces stricter error handling for deprecated arguments, reorganizes the deduplication module, and adds a port of readability.js's `is_probably_readerable` function. Various extraction and CLI bugs were also fixed.

Key Highlights

Deprecated CLI and function arguments now raise errors instead of being silently ignored.
Code restructuring: `trafilatura.hashing` module moved to `trafilatura.deduplication`.
Port of `is_probably_readerable` from readability.js for better content detection.
Markdown table fixes and CLI improvements for file processing and stdin handling.

Breaking Changes

Module reorganization: `trafilatura.hashing` has been moved to `trafilatura.deduplication`.

New Features

Port of `is_probably_readerable` from readability.js
Markdown table fixes
CLI fixes for file processing options, modification time, and binary stdin reading
Fix for deflate encoding and optional zstd support

Full Release Notes

Breaking changes:
- raise errors on deprecated CLI and function arguments (#581)
- regroup classes and functions linked to deduplication (#582)
``trafilatura.hashing`` → ``trafilatura.deduplication``

Extraction:
- port of is_probably_readerable from readability.js by @zirkelc in #587
- Markdown table fixes by @naktinis in #601
- fix list spacing in TXT output (#598)
- CLI fixes: file processing options, mtime, and tests (#605)
- CLI fix: read standard input as binary (#607)

Downloads:
- fix deflate and add optional zstd to accepted encodings (#594)
- spider fix: use internal download utilities for robots.txt (#590)

Maintenance:
- add author XPaths (#567)
- update justext and lxml dependencies (#593)
- simplify code: unique function for length tests (#591)

Docs:
- fix typos by @RainRat in #603