v2.6.2
spider-rs/spiderv2.6.2Sep 21, 2024by j-mendez
AI Summary
Open-sourced transformation utilities for high-performance content output conversion (Markdown, Text).
Key Highlights
- Open-sourced spider_transformations crate
- Support for Markdown and Text output formats
- Feature flag integration with spider_utils
New Features
- HTML to Markdown transformation
- HTML to Text transformation
- TransformConfig for formatting options
Full Release Notes
# Whats Changed
We Open Sourced our transformation utils for [Spider cloud](https://spider.cloud) that provides high performance output to markdown, text, and other formats.
You can install `spider_transformations` on it's own or use the feature flag `transformations` when installing `spider_utils`.
```rs
use spider::tokio;
use spider::website::Website;
use spider_utils::spider_transformations::transformation::content::{
transform_content, ReturnFormat, TransformConfig,
};
use tokio::io::AsyncWriteExt;
#[tokio::main]
async fn main() {
let mut website: Website = Website::new("https://rsseau.fr");
let mut rx2: tokio::sync::broadcast::Receiver<spider::page::Page> =
website.subscribe(0).unwrap();
let mut stdout = tokio::io::stdout();
let mut conf = TransformConfig::default();
conf.return_format = ReturnFormat::Markdown;
let join_handle = tokio::spawn(async move {
while let Ok(res) = rx2.recv().await {
let markup = transform_content(&res, &conf, &None, &None);
let _ = stdout
.write_all(format!("- {}\n {}\n", res.get_url(), markup).as_bytes())
.await;
}
stdout
});
let start = std::time::Instant::now();
website.crawl().await;
website.unsubscribe();
let duration = start.elapsed();
let mut stdout = join_handle.await.unwrap();
let _ = stdout
.write_all(
format!(
"Time elapsed in website.crawl() is: {:?} for total pages: {:?}",
duration,
website.get_links().len()
)
.as_bytes(),
)
.await;
}
```
**Full Changelog**: https://github.com/spider-rs/spider/compare/v2.5.2...v2.6.2