v2.33.11
spider-rs/spiderv2.33.11Mar 14, 2025by j-mendez
AI Summary
Adds `Website::with_crawl_timeout` to enforce a maximum crawl duration, which is useful when robots.txt changes affect expected crawl durations.
Key Highlights
- New `with_crawl_timeout` builder method
- Control over crawl duration with robots.txt
- Example code demonstrating timeout usage
New Features
- Crawl timeout control
Full Release Notes
# Whats Changed
Add `Website::with_crawl_timeout` builder method to add a max timeout for the crawl.
This is useful when robots.txt can change the expected crawl durations.
Example:
```rust
use std::time::Duration;
use spider::tokio;
use spider::website::Website;
use tokio::io::AsyncWriteExt;
#[tokio::main]
async fn main() {
let mut website: Website = Website::new("https://spider.cloud").with_crawl_timeout(Some(Duration::from_millis(10))).build().unwrap();
let mut rx2: tokio::sync::broadcast::Receiver<spider::page::Page> =
website.subscribe(0).unwrap();
let mut stdout = tokio::io::stdout();
let join_handle = tokio::spawn(async move {
while let Ok(res) = rx2.recv().await {
let _ = stdout
.write_all(format!("- {}\n", res.get_url()).as_bytes())
.await;
}
stdout
});
let start = std::time::Instant::now();
website.crawl().await;
website.unsubscribe();
let duration = start.elapsed();
let mut stdout = join_handle.await.unwrap();
let _ = stdout
.write_all(
format!(
"Time elapsed in website.crawl() is: {:?} for total pages: {:?}",
duration,
website.get_size().await
)
.as_bytes(),
)
.await;
}
```
**Full Changelog**: https://github.com/spider-rs/spider/compare/v2.33.1...v2.33.11