Starstruk Logo MarkStarstruk

v2.24.15

spider-rs/spiderv2.24.15Jan 4, 2025by j-mendez

AI Summary

Adds a validation callback for pages and a `basic` feature flag to disable io-uring on Linux while keeping default features. Also includes fixes for redirects and URL parsing.

Key Highlights

  • Added `on_should_crawl_callback` to validate pages before crawling.
  • Added `basic` feature flag to disable io-uring on Linux.
  • Fixed crawling initial redirects and URL parsing issues.
  • Added `page_error_status_details` and `blocked_crawl` features.

New Features

  • Validation callback for pages
  • Blocked crawl feature
  • Basic feature flag
  • Page error status details

Full Release Notes

# Whats Changed

Add a callback to perform validation using [spider::page::Page](https://docs.rs/spider/latest/spider/page/struct.Page.html).
You can now use the `basic` feature flag to easily disable io-uring on linux and still get the default features with `"default-features = false"`.

1. feat(website): add on_should_crawl_callback [#241]
1. feat(page): add blocked_crawl [#242]
1. chore(disk): fix cfg aho_corasick
1. chore(fs): remove tentril crate
1. chore(page): fix crawling initial redirects
1. chore(chrome): fix compile fs flag
1. feat(cargo): add basic feature flag
1. chore(connect): fix compile missing libc
1. feat(page): add page_error_status_details
1. perf(page): remove parsing url directly

**Full Changelog**: https://github.com/spider-rs/spider/compare/v2.23.7...v2.24.15