spider Releases

This release focuses on the Tau project, moving the CLI into the main repository, fixing deployment issues, enhancing the spore-drive interface, and addressing Windows compilation and system service problems.

Jan 14, 2025

Performance improvements by skipping URL parsing per page, new functionality to collect links for a new domain target, and fixes for root domain parsing.

Jan 11, 2025

Adds a validation callback for pages and a `basic` feature flag to disable io-uring on Linux while keeping default features. Also includes fixes for redirects and URL parsing.

Jan 4, 2025

Linux now uses io_uring for the DNS connection phase to improve performance, with a feature flag to disable it for older kernels.

Dec 31, 2024

Introduces SQLite for improved memory handling with disk features, allowing for billions of resources. Also fixes concurrency issues.

Dec 24, 2024

Fixes HTTP crawling past the first page and safe handling of absolute URLs.

Dec 18, 2024

Adds balance and remote_addr feature flags, re-usable chrome client, and fixes Chrome navigation issues.

Dec 10, 2024

Adds GitHub action targets, port configuration, and NO_COLOR support for the bore project.

Dec 5, 2024

Fixes Chrome link returns, adds hydration script ignores for next.js/astro, adds custom domain interception, and analytics/ads blocking.

Dec 5, 2024

Chrome crawls now report total bytes used over the network, and ignore list for unwanted requests is improved.

Dec 4, 2024

Major performance increase for Chrome crawling by blocking unwanted XHR requests and scripts.

Dec 4, 2024

Adds async streaming for transforms, optional string-interning, and fixes various crawling and URL handling issues.

Nov 30, 2024

Addresses critical stability issues regarding gateway retries and limit handling to prevent infinite loops and process breaks.

Nov 27, 2024

A stability-focused release fixing various issues that could cause the process to hang or consume excessive CPU resources.

Nov 7, 2024

Focuses on improving smart mode parsing performance and fixing rendering issues with inline JavaScript and HTML encoding.

Nov 5, 2024

Major performance boost by processing crawling tasks concurrently, enabling better handling of Next.js SSG sites.

Oct 31, 2024

Fixes crawling bugs and adds support for specific JS frameworks and OpenAI configuration options.

Oct 23, 2024

Enhances transformation capabilities and fixes stealth mode/browser automation issues.

Oct 22, 2024

Adds new data extraction methods (XPath) and output formats (XML) for web scraping transformations.

Oct 9, 2024

Improves Chrome integration, request interception, and stealth capabilities.

Oct 5, 2024