v1.99.10

spider-rs/spiderv1.99.10Jul 19, 2024by j-mendez

AI Summary

Major performance increase for CLI scraping is introduced alongside a new utility crate designed for manual content extraction using CSS selectors.

Key Highlights

Major performance increase for CLI scraping and downloading
New `spider_utils` crate for manual content extraction
CSS generic element scraping support

New Features

New `spider_utils` crate with `build_selectors` and `css_query_select_map_streamed`

Full Release Notes

# Whats Changed

1. Major performance increase for the CLI scraping and downloading. We now use subscriptions and concurrently stream process files storing and stdout.
2. You can now use the `spider_utils` crate to extract content manually as needed using CSS generic CSS element scraping.

```rust
use spider::{
    hashbrown::{HashMap, HashSet},
    packages::scraper::Selector,
};
use spider_utils::{build_selectors, css_query_select_map_streamed};

async fn css_query_selector_extract() {
    let map = HashMap::from([(
        "list",
        HashSet::from([".list", ".sub-list"]),
    )]);
    let data = css_query_select_map_streamed(
        r#"<html>
            <body>
                <ul class="list"><li>First</li></ul>
                <ul class="sub-list"><li>Second</li></ul>
            </body>
        </html>"#,
        &build_selectors(map),
    )
    .await;
    println!("{:?}", data);
    // {"list": ["First", "Second"]}
}
```

thanks for the help @gjtorikian


**Full Changelog**: https://github.com/spider-rs/spider/compare/v1.99.5...v1.99.10