v2.6.2

spider-rs/spiderv2.6.2Sep 21, 2024by j-mendez

AI Summary

Open-sourced transformation utilities for high-performance content output conversion (Markdown, Text).

Key Highlights

  • Open-sourced spider_transformations crate
  • Support for Markdown and Text output formats
  • Feature flag integration with spider_utils

New Features

  • HTML to Markdown transformation
  • HTML to Text transformation
  • TransformConfig for formatting options

Full Release Notes

# Whats Changed

We Open Sourced our transformation utils for [Spider cloud](https://spider.cloud) that provides high performance output to markdown, text, and other formats.

You can install `spider_transformations` on it's own or use the feature flag `transformations` when installing `spider_utils`.

```rs
use spider::tokio;
use spider::website::Website;
use spider_utils::spider_transformations::transformation::content::{
    transform_content, ReturnFormat, TransformConfig,
};
use tokio::io::AsyncWriteExt;

#[tokio::main]
async fn main() {
    let mut website: Website = Website::new("https://rsseau.fr");
    let mut rx2: tokio::sync::broadcast::Receiver<spider::page::Page> =
        website.subscribe(0).unwrap();
    let mut stdout = tokio::io::stdout();

    let mut conf = TransformConfig::default();
    conf.return_format = ReturnFormat::Markdown;

    let join_handle = tokio::spawn(async move {
        while let Ok(res) = rx2.recv().await {
            let markup = transform_content(&res, &conf, &None, &None);

            let _ = stdout
                .write_all(format!("- {}\n {}\n", res.get_url(), markup).as_bytes())
                .await;
        }
        stdout
    });

    let start = std::time::Instant::now();
    website.crawl().await;
    website.unsubscribe();
    let duration = start.elapsed();
    let mut stdout = join_handle.await.unwrap();

    let _ = stdout
        .write_all(
            format!(
                "Time elapsed in website.crawl() is: {:?} for total pages: {:?}",
                duration,
                website.get_links().len()
            )
            .as_bytes(),
        )
        .await;
}
```

**Full Changelog**: https://github.com/spider-rs/spider/compare/v2.5.2...v2.6.2