spider Releases

Pingora release adding extensive proxy configuration options, HTTP/2 fixes, and cache improvements.

May 9, 2025

The page streaming rewriter now handles built-in metadata extraction by default, accessible via `page.metadata` or `page.get_metadata()`. Additionally, this release fixes concurrent context creation in Chrome and bumps the CDP revision.

May 7, 2025

This release fixes problematic error returns, moves the HTTP package, and adds support for custom ACME providers. It also improves DNS handling for ACME types and updates major dependencies.

May 1, 2025

Fixes HTTP and smart mode request Host headers to prevent issues with redirects and fixes OpenAI automation usage. Also introduces antibot detection and improves sitemap handling.

Apr 18, 2025

Fixes the Dockerfile, updates Linux installation instructions, and adds configurable control and tunnel addresses for the server.

Apr 14, 2025

Fixes XML parsing for initial links and adds missing Chrome headers. Introduces support for real browser loading in Chrome.

Mar 29, 2025

Adds support for the `rquest` client library with a feature flag, including emulation capabilities. Improves Chrome request timeout handling.

Mar 26, 2025

Introduces event tracking for headless browsing to map requests and responses with timing and byte transfer data. Adds stable media extraction handling.

Mar 19, 2025

Adds `Website::with_crawl_timeout` to enforce a maximum crawl duration, which is useful when robots.txt changes affect expected crawl durations.

Mar 14, 2025

Removes the `jemalloc` flag (which should be set at the top level of main) and adds asset support for Chrome media requests.

Mar 9, 2025

Adds thread-safe crawling and bootstrapping setup methods, specifically `crawl_chrome_send` and `crawl_raw_send`.

Mar 8, 2025

Enhanced Chrome performance by optimizing page deserialization and removing unused Bytes wrappers, alongside adding Chrome to block lists.

Mar 7, 2025

This release focuses on the Tau project, addressing ACME certificate issues, updating UFW rules for spore-drive, fixing Windows executable path problems, and removing the Spin dependency.

Mar 6, 2025

Fixed a sequence issue with Chrome's wait_for events and added functionality to cancel navigation networks.

Mar 5, 2025

Enhanced Chrome with document reload protection and improved firewall protection, while optimizing performance by removing unnecessary indirections and fixing initialization issues.

Feb 23, 2025

Released Open-Sora v1.3, focusing on upgraded video compression, enhanced ST-DiT architecture, and improved training data quality.

Feb 21, 2025

Introduced the firewall feature for malice protection and enhanced Chrome rendering with smart mode and lazy loading.

Feb 16, 2025

Improved CLI usability with a help trigger and enhanced connection handling with WebSocket retries and HTTP fallback.

Feb 13, 2025

Introduced web page normalization to prevent duplicate content and crawl traps, and enabled crawling of websites on ports other than the default 80 and 443.

Jan 25, 2025

Added automatic sitemap discovery on errors, enabled memory-based caching for Chrome hybrid mode, and fixed various issues related to queue draining and worker initialization.

Jan 18, 2025