v2.26.27

spider-rs/spiderv2.26.27Jan 18, 2025by j-mendez

AI Summary

Added automatic sitemap discovery on errors, enabled memory-based caching for Chrome hybrid mode, and fixed various issues related to queue draining and worker initialization.

Key Highlights

Auto find sitemap URL
Added memory cache flag
Fixed queue draining
Fixed crawl depth
Fixed worker init
Fixed status codes

New Features

Sitemap auto-discovery
Memory caching

Full Release Notes

# Whats Changed

1. add auto find sitemap url on 404 or network error.
2. fix chrome_cache_hybrid compile.
3. add `cache_chrome_hybrid_mem` flag to use memory instead of disk.
4. fix q draining across website methods
5. fix crawl depth handling
6. fix worker init background connect
7. add proper status code from errors

**Full Changelog**: https://github.com/spider-rs/spider/compare/v2.26.1...v2.26.27