Starstruk Logo MarkStarstruk

v0.0.9

bilibili/Index-anisorav0.0.9Jun 26, 2026by paulpierre

AI Summary

Fixes Unicode encoding errors, bypasses JS checks, and adds configurable heading styles and URL exclusion. Bumps minimum Python version to 3.8 and improves test coverage.

Key Highlights

  • Fixes UnicodeEncodeError on non-ASCII content.
  • Configurable heading styles (ATX, ATX_CLOSED, UNDERLINE, SETEXT).
  • URL filtering with `exclude_paths` parameter.
  • Comprehensive test suite with 95% coverage.

Breaking Changes

  • Minimum Python version bumped to 3.8.

New Features

  • Configurable heading style for markdownify.
  • URL exclusion parameter (`exclude_paths`) for various functions.

Full Release Notes

## What's New

This release resolves **6 open issues** and introduces several community-requested features.

### Fixed
- **#7** — `UnicodeEncodeError` on non-ASCII content. All file writes now use `encoding='utf-8'`.
- **#8** — Bypass JavaScript checks. A browser-like `User-Agent` header is now sent on all `requests.get` calls.
- **#10** — `target_content` CSS selector handling. Null `href` links are now skipped in `get_target_links`.
- **#11** — `UnboundLocalError` in `get_target_content` when no tags are found. `main_content` is now initialized to `None` and checked before `str()` conversion.

### Added
- **#9** — `exclude_paths` parameter for URL filtering. Available in `crawl()`, `get_target_links()`, `worker()`, `md_crawl()`, and the CLI (`-x` / `--exclude-paths`).
- **#20** — Configurable `heading_style` for markdownify (`ATX`, `ATX_CLOSED`, `UNDERLINE`, `SETEXT`). CLI flag: `-s` / `--heading-style`.
- **#17** — Proper `dependencies` in `pyproject.toml` (`beautifulsoup4`, `markdownify`, `requests`) enabling `uvx` support out of the box.
- Comprehensive test suite — 60 tests with 95% coverage across `__init__.py` (96%) and `cli.py` (88%).
- `dev` optional dependencies (`pytest`, `pytest-cov`) for contributors.

### Changed
- Minimum Python version bumped to 3.8.
- `pyproject.toml` now uses `[project.scripts]` instead of `[tool.poetry.scripts]`.

**Full changelog:** https://github.com/paulpierre/markdown-crawler/blob/main/CHANGELOG.md