v1.95.27

spider-rs/spiderv1.95.27May 28, 2024by j-mendez

AI Summary

Addresses major efficiency bottlenecks in crawl delays by capping robot delays at 60 seconds and fixing crawl limit exits.

Key Highlights

Fixed crawl delays respecting limits (max 60s)
Fixed crawl limit immediate exit
Added zstd support for reqwest
Bumped chromiumoxide and tiktoken-rs versions

Full Release Notes

## Whats Changed

This release provides a major fix for crawls being delayed by respect robots or crawl delays. If you set a limit or budget for the crawl and a robots.txt contains a delay of 10s this would be a bottleneck for the entire crawl when limits applied since the we would have to wait for each link to process prior to exiting. The robots delay is now maxed at 60s for efficiency.

* chore(cli): fix limit respecting
* chore(robots): fix respect robots [#184]
* bump chromiumoxide@0.6.0 
* bump tiktoken-rs@0.5.9
* bump hashbrown@0.14.5
* add zstd support reqwest
* unpin smallvec
* chore(website): fix crawl limit immediate exit
* chore(robots): add max delay respect

**Full Changelog**: https://github.com/spider-rs/spider/compare/v1.95.6...v1.95.27