Starstruk Logo MarkStarstruk

v2.14.0

spider-rs/spiderv2.14.0Nov 30, 2024by j-mendez

AI Summary

Adds async streaming for transforms, optional string-interning, and fixes various crawling and URL handling issues.

Key Highlights

  • Add `transform_content_send` for async streaming.
  • Add optional string-interning.
  • Fix crawl, establish domain removal.
  • Add streaming markdown/commonmark and text transforming.

New Features

  • Async streaming transform
  • Optional string-interning
  • Streaming markdown/text transform

Full Release Notes

### Release Notes

#### Features
- **feat(transform):** add `transform_content_send` for async streaming.

#### Improvements
- **chore(interning):** add optional string-interning.
- **chore(website):** fix crawl, establish domain removal [#233].
- **chore(transform):** add streaming markdown/commonmark transforming.
- **chore(transform):** add streaming text transforming.
- **chore(chrome):** add request interception analytics ignore.

#### Bug Fixes
- **chore(page):** fix URL encode handling mismatch.
- **chore(transform):** fix repeated text streaming.
- **chore(page):** fix page link return with full URLs.
- **chore(website):** fix crawl delay handling.
- **perf(website):** reduce extra context switching on crawls.

Thank you for the help @Revertron!

**Full Changelog**: https://github.com/spider-rs/spider/compare/v2.13.78...v2.14.0