v1.85.4

spider-rs/spiderv1.85.4Mar 11, 2024by j-mendez

AI Summary

Adds a mid-crawl queue management system that allows users to dynamically update or add links to the crawl context during execution.

Key Highlights

  • Added website.queue for mid-crawl link updates
  • Fixed semaphore limiting for scraping
  • Added example for dynamic URL manipulation

New Features

  • Mid-crawl queue management

Full Release Notes

# Whats Changed

You can now update the crawl links outside of the context by using `website.queue` to get a sender.

* feat(q): add mid crawl queue
* chore(chrome): fix semaphore limiting scrape

```rs
use spider::tokio;
use spider::url::Url;
use spider::website::Website;

#[tokio::main]
async fn main() {
    let mut website: Website = Website::new("https://rsseau.fr");
    let mut rx2 = website.subscribe(16).unwrap();
    let mut g = website.subscribe_guard().unwrap();
    let q = website.queue(100).unwrap();

    tokio::spawn(async move {
        while let Ok(res) = rx2.recv().await {
            let u = res.get_url();
            println!("{:?}", u);
            let mut url = Url::parse(u).expect("Failed to parse URL");

            let mut segments: Vec<_> = url
                .path_segments()
                .map(|c| c.collect::<Vec<_>>())
                .unwrap_or_else(Vec::new);

            if segments.len() > 0 && segments[0] == "en" {
                segments[0] = "fr";
                let new_path = segments.join("/");
                url.set_path(&new_path);
                // get a new url here or perform an action and queue links
                // pre-fetch all fr locales
                let _ = q.send(url.into());
            }
            g.inc();
        }
    });

    let start = std::time::Instant::now();
    website.crawl().await;
    let duration = start.elapsed();

    println!(
        "Time elapsed in website.crawl() is: {:?} for total pages: {:?}",
        duration,
        website.get_links().len()
    )
}
```

Thanks @oiwn

**Full Changelog**: https://github.com/spider-rs/spider/compare/v1.84.11...v1.85.4