v1.86.16

spider-rs/spiderv1.86.16Mar 19, 2024by j-mendez

AI Summary

Introduces dynamic browser interaction capabilities using custom scripts powered by OpenAI and adds support for crawling Cold Fusion files.

Key Highlights

  • Added OpenAI Chrome driver for custom scripts
  • Added Cold Fusion file crawling support
  • Added screenshot configuration options
  • Added network idle waiting logic

New Features

  • OpenAI Chrome driver
  • Cold Fusion support
  • Screenshot configuration

Full Release Notes

## What's Changed

You can now dynamically drive the browser with custom scripts using OpenAI.
Make sure to set the `OPENAI_API_KEY` env variable or pass it in to the program.

* Openai/chrome driver by @j-mendez in https://github.com/spider-rs/spider/pull/174
* chore(page): add cold fusion file crawling support

```rust
extern crate spider;
use spider::configuration::{GPTConfigs, WaitForIdleNetwork};
use spider::tokio;
use spider::website::Website;

#[tokio::main]
async fn main() {
    let _ = tokio::fs::create_dir_all("./storage/").await;

    let screenshot_params =
        spider::configuration::ScreenshotParams::new(Default::default(), Some(true), Some(true));
    let screenshot_config =
        spider::configuration::ScreenShotConfig::new(screenshot_params, true, true, None);

    let mut website: Website = Website::new("https://google.com")
        .with_chrome_intercept(true, true)
        .with_wait_for_idle_network(Some(WaitForIdleNetwork::new(Some(Duration::from_secs(30)))))
        .with_screenshot(Some(screenshot_config))
        .with_limit(1)
        .with_openai(Some(GPTConfigs::new(
            "gpt-4-1106-preview",
            "Search for Movies",
            500,
        )))
        .build()
        .unwrap();
    let mut rx2 = website.subscribe(16).unwrap();

    tokio::spawn(async move {
        while let Ok(page) = rx2.recv().await {
            println!("{}\n{}", page.get_url(), page.get_html());
        }
    });

    website.crawl().await;
}
```

The output of the custom script from the AI:

![Custom Script output](https://github.com/spider-rs/spider/assets/8095978/566a9904-8888-4e02-9807-d4998221e94c)

The screenshot of the page output:

![Output of the page](https://github.com/spider-rs/spider/assets/8095978/39eb0638-2386-4721-ad0f-50f179d3a5f2)

**Full Changelog**: https://github.com/spider-rs/spider/compare/v1.85.4...v1.86.16