Starstruk Logo MarkStarstruk

v1.98.8

spider-rs/spiderv1.98.8Jul 4, 2024by j-mendez

AI Summary

Improves robots.txt parsing reliability for root URLs and adds serialization support for wait states.

Key Highlights

  • Fixed robots.txt detection for root URLs
  • Safe handling of robots fetch
  • Added `serde` `WaitFor` type

New Features

  • New `WaitFor` type for Serde serialization

Full Release Notes

# Whats Changed

1. Fix robots.txt detection for root urls and add safe handling of robots fetch.
1. Add serde WaitFor

**Full Changelog**: https://github.com/spider-rs/spider/compare/v1.98.4...v1.98.8