v2.52.2

## Changes since vv1.7.5

- fix: concurrency/memory/panic audit hardening (v2.52.7) (#401)
- chore(release): bump workspace to v2.52.6
- browser: per-crawl control map for optional chrome enhancements (#400)
- chore(release): bump workspace to v2.52.5
- browser: genericize internal gateway topology/metrics in doc comments
- browser: harden in-request gateway re-acquire (jitter, opt-out, timeouts)
- security: fix cargo-audit failures (quick-xml DoS + lopdf)
- chore(deps): bump actions/checkout from 4 to 7 (#399)
- security: bump quinn-proto to 0.11.15 (RUSTSEC-2026-0185)
- browser: connection reliability and failover improvements
- chore(release): v2.52.4 — UTM-tag spider.cloud README links
- chore(release): v2.52.3 — markdown scrape default, api cloud mode, branded login page
- feat(cli): markdown scrape by default + branded sign-in page
- chore(release): bump workspace to v2.52.2
- feat(cli): browser OAuth login + MCP OAuth client example
- chore(release): bump workspace to v2.52.1
- fix(website): bound sitemap crawl task spawning with semaphore (#397)
- feat(chrome_intercept): with_remote_local_policy (spider 2.52.0) (#396)
- Fix 500ms timeout for proxy error statuses 598/599 (#395)
- chore(release): bump workspace to v2.51.208
- fix(website): crawl_raw uses raw sitemap chain on chrome builds (#393)
- feat: add on_link_blocked_callback for robots.txt blocked URLs (#394)
- perf(page,website): drop redundant heap allocs on link-validation paths (v2.51.207)
- fix(chrome): close leaked tabs in smart-upgrade + spawn-pages paths (v2.51.206)
- fix(agent,mcp): bound page tabs, MCP sessions, and agent KV stores (v2.51.205)
- docs: tighten README and rework intro
- feat(agent,scripting): client reuse + usage tracking (v2.51.204)
- perf(agent,scripting): per-worker interpreter cache + permit timeout (v2.51.203)
- chore(release): v2.51.202 — patch bump
- chore(release): v2.51.201 — formalize AdaptiveSemaphore safety contract
- feat(website): AdaptiveSemaphore + with_adaptive_concurrency hook for runtime concurrency
- feat(agent): pure-Rust embedded Python + JS scripting (v2.51.200)
- feat(website): RemoteFetcher hook for delegated per-URL fetch (v2.51.198)
- chore(release): sync internal cross-crate deps to v2.51.197
- chore(release): v2.51.197 — chrome wait_for cap removal in non-success path
- fix(chrome): honour caller request_timeout in non-run_events wait branch
- chore(release): v2.51.196 — per-website decentralized worker URL overrides
- chore(release): sync internal cross-crate deps to v2.51.195
- chore(release): v2.51.195 — wreq network_interface plumbing
- feat(wreq): wire network_interface through wreq configure_base_client
- chore(release): sync internal cross-crate deps to v2.51.194
- chore(release): v2.51.194 — wreq emulation header survival fix
- fix(wreq): skip default_headers(empty) so emulation headers survive
- chore(release): sync internal cross-crate deps to v2.51.193
- chore(release): v2.51.193 — chrome_store_page tab keep-alive
- fix(website): keep chrome tab alive for chrome_store_page subscribers
- chore(examples): track chrome_proxy_dns_shortcircuit source
- chore(release): v2.51.192 — chrome hedge stack-overflow fix
- fix(page): heap-pin chrome DNS hedge future to stop crawl stack overflow
- chore(release): update Cargo.lock for v2.51.191
- chore(release): v2.51.191 — SSRF redirect guard + fmt
- fix(redirect): block SSRF redirect targets in all follow policies
- feat(page): expand IGNORE_EXTENSIONS + spool-safe chrome tunnel DNS confirm — v2.51.190
- perf(page): cross-feature NXDOMAIN cache + DNS-probe & CDP-subscribe skip — v2.51.189
- chore(examples): unregister chrome_proxy_dns_shortcircuit (untracked source)
- test(example): chrome_nxdomain_test with CHROME_WS_URL plumbing
- chore(release): sync internal cross-crate deps to v2.51.188
- feat(chrome): CDP loadingFailed hedge trigger + v2.51.188
- chore(release): sync internal cross-crate deps to v2.51.187
- feat(chrome): DNS hedge alongside chrome navigation + v2.51.187
- chore(release): sync internal cross-crate deps to v2.51.186
- chore(release): cargo fmt + sync sister crates to v2.51.186
- feat(chrome): two-signal NXDOMAIN cache + v2.51.186 (supersedes v2.51.185 pre-flight)
- feat(chrome): opt-in pre-flight local DNS shortcircuit + v2.51.185
- feat(solvers): detect embedded Cloudflare Turnstile widgets + v2.51.184
- docs(readme): rewrite for clarity, lead with streaming + benchmarks
- chore(release): sync internal cross-crate deps to v2.51.183
- release(spider): v2.51.183 — chain-walk classifier for cache-wrapped errors
- chore(release): sync internal cross-crate deps to v2.51.182
- release(spider): v2.51.182 + regression test for cache-gate fix
- fix(client): gate http-cache-reqwest middleware on configuration.cache
- chore(release): sync internal cross-crate deps to v2.51.181
- perf(page,utils): swap tokio yield_now for consume_budget in CPU-bound chunk loops + v2.51.181
- chore(release): sync internal cross-crate deps to v2.51.180
- fix(website): smart-mode chrome fallback for transport-level HTTP failures + v2.51.180
- chore(release): sync internal cross-crate deps to v2.51.179
- chore: silence pre-existing clippy dead-code warnings + v2.51.179
- chore(release): bump remaining workspace crates to v2.51.178
- chore(release): sync internal cross-crate deps to v2.51.178
- fix(page): wire two-signal NXDOMAIN upgrade into Page::new_page_streaming + v2.51.178
- fix(page): don't cache TimedOut + bump confirm_tunnel timeouts to 1500ms + v2.51.176
- feat(page): chrome two-signal NXDOMAIN upgrade for empty-200 → 504 reclass + v2.51.175
- feat(utils): wire two-signal NXDOMAIN upgrade into all fetch_page_html error paths + v2.51.174
- chore(release): sync internal cross-crate deps to v2.51.173
- feat(page): SOCKS proxy CONNECT failure detection + v2.51.173
- chore(page): fix clippy::doc_lazy_continuation in two-signal docstring
- chore(examples): add dns_nxdomain_test + chrome_remote_logrocket repros
- test(page): gate non-cache helper tests off cache_request feature
- chore(release): sync internal cross-crate deps to v2.51.172
- feat(page): two-signal upgrade for cache_request 526 + v2.51.172
- chore(release): sync internal cross-crate deps to v2.51.171
- perf(page): swap HostDnsCache RwLock for DashMap (no user-facing locks) + v2.51.171
- chore(release): sync internal cross-crate deps to v2.51.170
- perf(page): bounded process-wide host DNS cache for tunnel-confirmation lookups + v2.51.170
- chore(release): sync internal cross-crate deps to v2.51.169
- feat(page): chrome path two-signal proxy tunnel classification + v2.51.169
- chore(release): sync internal cross-crate deps to v2.51.168
- feat(page): two-signal proxy tunnel classification via local DNS confirmation + v2.51.168
- chore(release): sync internal cross-crate deps to v2.51.167
- revert(page): tunnel/proxy CONNECT failures stay retryable + v2.51.167
- chore(release): sync internal cross-crate deps to v2.51.166
- fix(page): chrome ERR_TUNNEL_CONNECTION_FAILED / ERR_PROXY_CONNECTION_FAILED → 526 + v2.51.166
- chore(release): sync internal cross-crate deps to v2.51.165
- fix(page): classify proxy CONNECT tunnel failures as 526 (non-retryable) + v2.51.165
- chore(release): sync internal cross-crate deps to v2.51.164
- chore(release): bump workspace to v2.51.164
- fix(config): http first-byte watchdog fires for single-proxy callers
- chore(release): sync internal cross-crate deps to v2.51.163
- fix(parallel_backends): short-circuit grace-period on permanent target failures + v2.51.163
- chore(release): sync internal cross-crate deps to v2.51.162
- fix(page): walk source chain in is_dns_error to catch macOS GAI NXDOMAIN + v2.51.162
- chore(release): sync internal cross-crate deps to v2.51.161
- fix(chrome): preserve HTTP final_redirect_destination across smart upgrade + v2.51.161
- chore(release): sync internal cross-crate deps to v2.51.160
- fix(website): apply redirect-aware selectors before smart establish link extraction + v2.51.160
- chore(release): sync internal cross-crate deps to v2.51.159
- fix(page): classify cache-wrapped transport errors as 526 + reorder DNS check + v2.51.159
- chore(release): sync internal cross-crate deps to v2.51.158
- fix(website): gate first-party allow builders on chrome feature + v2.51.158
- feat(intercept): expose chromey 2.49 allow_first_party_* flags via Website builder
- style: clippy fixes (manual_pattern_char_comparison, question_mark)
- chore(ci): bump softprops/action-gh-release to v3
- chore(release): sync internal cross-crate deps to v2.51.157
- chore(release): bump workspace to v2.51.157
- style: cargo fmt on new proxy_strategy + utils + website hunks
- feat(proxy_strategy): trait + Website hook + lazy secondary client
- feat(configuration): ProxyKind enum + proxies_by_kind sidecar
- feat(utils): media_asset classifier + LazyArc<T> primitive
- chore(utils): solver-gate diagnostic logging in fetch_page_html_chrome_base
- chore(release): sync internal cross-crate deps to v2.51.156
- fix(chrome,validation): smart-mouse jitter loop + PerimeterX title parity
- feat(chrome): route el.click() through smart-mouse path — v2.51.156
- fix(chrome): route point-based clicks/drags through smart-mouse paths
- chore(release): sync internal cross-crate deps to v2.51.155
- fix(utils): handle_base must not byte-slice on UTF-8 boundaries — v2.51.155
- ci(bench): install spider_worker before decentralized bench
- fix(ci): bump hickory 0.26.0->0.26.1, ignore unfixable 0.25.x advisories, fix worker
- chore(release): sync internal cross-crate deps to v2.51.154
- fix(chrome): re-honor quiet_ms in wait-for-DOM v2 to handle bursty SPAs + v2.51.154
- chore(release): sync internal cross-crate deps to v2.51.153
- fix(utils): never cache failed responses + v2.51.153
- chore(release): sync internal cross-crate deps to v2.51.152
- fix(http): balance vitals counter on first-byte timeout early-return + v2.51.152
- chore(release): sync internal cross-crate deps to v2.51.151
- feat(http): wire first-byte watchdog into Page::new_page_streaming + crawl_smart auto-arm + v2.51.151
- style: cargo fmt + targeted clippy fixes (drop redundant &'static, single-line log)
- chore(release): sync internal cross-crate deps to v2.51.150
- feat(http): auto-arm first-byte watchdog under balance + 2 HTTP proxies + v2.51.150
- chore(release): sync internal cross-crate deps to v2.51.149
- feat(http): first-byte watchdog foundation + v2.51.149
- chore(release): sync internal cross-crate deps to v2.51.148
- chore(release): bump workspace to v2.51.148
- fix(chrome,hedge): track hedge's own connected_url + browser_dead
- chore(release): sync internal cross-crate deps to v2.51.147
- chore(release): bump workspace to v2.51.147
- test(chrome): cooldown deadline math + last-write-wins on mark_url_bad
- chore(release): sync internal cross-crate deps to v2.51.146
- feat(chrome): first-byte watchdog jitter + measurement test + v2.51.146
- chore(release): sync internal cross-crate deps to v2.51.145
- feat(chrome): mark-bad endpoint rotation + auto-populated chrome_endpoint_url + v2.51.145
- feat(chrome): first-byte watchdog + browser_dead signal for backend rotation
- perf(chrome): lazy-cache multi-endpoint failover across setup_browser_configuration
- chore(release): sync internal cross-crate deps to v2.51.144
- fix(detect): trim leading whitespace before binary/asset sniffing + v2.51.144
- chore(release): sync internal cross-crate deps to v2.51.143
- chore(release): lockfile dep refresh + v2.51.143
- chore(release): commit + push internal dep sync before publishing
- chore(release): sync internal cross-crate deps to v2.51.142
- chore(release): cargo fmt + lockfile refresh + v2.51.142
- chore(release): sync internal cross-crate deps to v2.51.141
- fix(chrome): honor request_timeout as a hard wall-clock cap + v2.51.141
- feat(chrome_remote_cache): main-doc-only dump knob + v2.51.140
- chore(release): sync internal cross-crate deps to v2.51.139
- fix(utils): reclassify PerimeterX block-page signals + v2.51.139
- chore(release): v2.51.138 — sync workspace versions for wait_for_idle_dom signal-stack
- perf(chrome): signal-stack wait_for_idle_dom — RAF-cadence, no quiet timer
- perf(page): metadata-first spool short-circuit + helpers + v2.51.137
- fix(page): reclassify empty 200 responses to 504 + v2.51.136
- chore(release): sync cross-crate deps to v2.51.135
- fix(chrome): strip JSON quotes from CDP-captured response headers + v2.51.135
- chore(release): sync cross-crate deps to v2.51.134
- perf(chrome): skip streaming extraction on assets + v2.51.134
- chore(release): sync cross-crate deps to v2.51.133
- fix(smart): preserve legacy link-set semantics on chrome upgrade + v2.51.133
- chore(release): sync cross-crate deps to v2.51.132
- perf(smart): stream link extraction during chrome upgrade + v2.51.132
- chore(release): sync cross-crate deps to v2.51.131
- perf(chrome): fold link extraction into chunk pump (single-pass) + v2.51.131
- perf(chrome): skip redundant fill-fetch when HTML extraction already ran + v2.51.130
- perf(utils,page): pre-compute is_valid_utf8 during streaming + v2.51.129
- fix(retry): drop h2 INTERNAL_ERROR from retryable reasons + v2.51.128
- perf(chrome): switch SSL/cert detector trio to single-pass Aho-Corasick + v2.51.127
- fix(retry): widen permanent-failure classification for HTTP/2 + 5xx + Chrome URL family + v2.51.126
- fix(retry): classify SSL handshake failures as 526 across HTTP + Chrome + v2.51.125
- chore(release): sync cross-crate deps to v2.51.124
- chore(release): bump workspace to v2.51.124
- chore(release): sync cross-crate deps to v2.51.123
- fix(chrome): single-URL with_chrome_connections takes default path + v2.51.123
- chore(release): sync cross-crate deps to v2.51.122
- chore(release): bump workspace to v2.51.122
- feat(page): classify address-unreachable + connection-refused as permanent
- perf(chrome): batch Fetch.disable cleanup + LLM JS wrapper robustness + v2.51.121
- feat(remote_cache): runtime config for skip_browser HTTP dumps
- chore(release): bump workspace to v2.51.119
- perf(parallel_backends): skip HTML-shape probes on assets in html_quality_score
- test(smart): live smoke test covering v2.51.117 source_str→target_url refactor
- chore(release): bump workspace to v2.51.117
- perf(utils): drop redundant HTML UTF-8 decode in fetch_page_html_chrome_base
- fix(website): emit synthetic Page to subscribers on chrome tab-creation timeout; bump to v2.51.116
- chore(release): bump workspace to v2.51.115
- fix(chrome_remote_cache): bypass proxies on remote cache HTTP client
- feat(chrome_remote_cache): read-only mode + v2.51.114
- chore(release): bump workspace to v2.51.113
- chore(release): bump workspace to v2.51.112
- perf(uring_fs): round-robin worker pool for parallel ring submission + v2.51.111
- chore(release): bump workspace to v2.51.110
- chore(release): bump workspace to v2.51.109
- chore(release): bump workspace to v2.51.108
- fix(website): use page.size() for initial_html_length on balanced spools + v2.51.107
- fix(website): retry links on semaphore close + spool-aware JS challenge detect + v2.51.106
- perf(chrome): lock-free OnceLock get/set init for tab-closer + v2.51.105
- feat(chrome,hedge,parallel_backends): concurrent tab-closer + explicit context dispose + v2.51.104
- fix(parallel_backends,chrome): close CDP tabs + background tab-closer watcher + v2.51.103
- chore(tests): move logrocket network probe out of tree
- chore(release): sync internal deps to v2.51.102
- feat(tests): live network map + markdown integrity check for logrocket + v2.51.102
- chore(release): sync internal deps to v2.51.101
- feat(website): expose with_max_main_frame_navigations on Website + v2.51.101
- chore(release): sync internal deps to v2.51.100
- feat(chrome): navigation-loop guard for JS / meta-refresh loops + v2.51.100
- chore(release): sync internal deps to v2.51.99
- feat(chrome): enforce redirect_limit on chromey path (opt-in) + v2.51.99
- chore(release): v2.51.98 — bump spider_network_blocker 0.0.81 → 0.0.82
- chore(release): v2.51.97 — bump async-openai 0.33→0.35, tiktoken-rs 0.9→0.11
- chore(release): v2.51.96
- chore(release): v2.51.95
- fix(page): webdriver+chrome+basic build under Page Drop impl
- chore(release): v2.51.94
- chore(release): sync internal deps to v2.51.93
- chore(release): v2.51.93
- fix: classify permanent DNS failures as 525 across http + chrome
- chore(release): v2.51.92
- chore(templates): add apache forbidden updated template
- fix(website): sitemaps push now compiles under default features
- perf,fix: eliminate wasteful clones and fix 5 test failures
- perf(website): consume CaseInsensitiveString via into_string()
- perf(utils,website,chrome): drop redundant to_string() and clones
- chore: bump workspace to 2.51.86 after cargo update
- perf(page,utils): eliminate wasteful clones in metadata and cache paths
- perf(page): borrow external_domains_caseless in link-stream rewriters
- chore: sync spider_agent internal deps to 2.51.83
- fix(chrome): update remaining Page::new call sites missed in 2.51.82
- chore: sync internal spider_agent deps to 2.51.82
- perf(chrome): bundle Chrome fetch params into borrowed struct for inlining
- fix(balance,chrome): cap failure-fallback wall time to request_timeout
- fix(crawl): missing .await on channel_send_page lost pages to subscribers
- feat(balance): per-Website lazy spool tmpdir with drop cleanup
- feat(balance,chrome): wire direct-to-disk spool into fetch_page_html_chrome_base
- perf(balance): DoS-cap the streaming spool writer + audit with_capacity sites
- feat(balance): signature-equivalent spool infrastructure; defer fetch wire
- feat(balance,chrome): stream-to-disk infrastructure for chrome spooling
- feat(chrome,balance): adaptive chrome HTML streaming via chromey 2.46.40
- perf(balance): streaming spool writer with inline vitals + full disk streaming
- chore: bump all crates to v2.51.72
- fix(page): drop unwrap in balance bytes tracking, bump crates to v2.51.71
- chore: bump sibling crates to v2.51.70 to match spider release
- feat(retry): add per-attempt hedge override on RetryDirective, fix E0716 in retry loop, bump v2.51.70
- chore: sync internal crate dependency versions to 2.51.69
- perf: pre-allocate Vec/HashMap/HashSet with known capacity, bump all crates to v2.51.69
- feat(retry): add configurable RetryStrategy trait for dynamic per-attempt profile switching, bump all crates to v2.51.68
- feat(page): add Page::size() for zero-cost content byte length, bump all crates to v2.51.67
- chore: sync internal crate dependency versions to 2.51.66
- fix(page): refuse to spool Chrome empty-HTML shells, bump all crates to v2.51.66
- chore: sync internal crate dependency versions to 2.51.65
- fix(warc): emit HTTP response headers so WARC files replay in replayweb.page (#384), bump all crates to v2.51.65
- feat(chrome): restore Chrome hedge path (HedgeBrowser + race logic), bump all crates to v2.51.64
- chore: bump all crates to v2.51.63
- feat(chrome): add disable_javascript config to suppress JS execution via CDP, bump all crates to v2.51.62
- chore: bump all crates to v2.51.61
- chore: cargo fmt, cargo update, bump all crates to v2.51.60
- test(chrome): add concurrency e2e tests for Chrome crawling
- fix(chrome): send shutdown signal immediately after page close, fix e2e test URL
- fix(chrome): prevent deadlock in goto_with_html_once when chrome_intercept enabled
- chore: bump all crates to v2.51.59
- chore(audit): suppress rustls-webpki and rand advisories
- chore: bump all crates to v2.51.58, spider_remote_cache to v0.3
- fix(cache): eagerly init worker, remove redundant set_client + v2.51.56
- fix(cache): fix chrome_remote_cache compilation + v2.51.56
- fix(cache): fix chrome_remote_cache compilation — cfg-gate tracing, bump chromey
- perf(cache): use spider_remote_cache for batched concurrent uploads + v2.51.55
- feat(chrome): WaitForLoad/WaitForNetworkIdle/WaitForNetworkAlmostIdle use CDP events + v2.51.54
- feat(chrome): add WaitForLoad, WaitForNetworkIdle, WaitForNetworkAlmostIdle actions + v2.51.53
- bump all crates to v2.51.52
- perf(chrome): concurrent page_wait — two-phase network-then-DOM + v2.51.51
- fix(balance): eliminate byte-counter leak causing Chrome crawl deadlock + v2.51.50
- feat(spider): add wait_guard feature — adaptive wait-for timeout for chronic bad domains + v2.51.49
- chromey v2.46.21, bump all crates to v2.51.48
- bump all crates to v2.51.47
- bump all crates to v2.51.46
- fix(chrome): add backoff to single-endpoint retry loop, cap unbounded network waits + v2.51.45
- bump all crates to v2.51.44
- perf(content): SIMD-accelerate estimate_text_length, estimate_tag_bytes, estimate_base64_bytes + v2.51.43
- chromey v2.46.13 -> v2.46.15, bump all crates to v2.51.42
- bump all crates to v2.51.41
- chromey v2.46.12 -> v2.46.13
- fix(hedge+chrome): remove chrome hedge path, fix backend semaphore deadlock, skip warmup hedges + v2.51.40
- perf(clean_html): zero-cost catch_unwind via AssertUnwindSafe, drop html clone + v2.51.39
- fix(stability): catch lol_html rewrite_str UTF-8 panic in clean_html_* + v2.51.38
- fix(balance): close byte-tracking memory leak in spool budget counter + v2.51.37
- perf(balance): zero-disk-IO link extraction for spooled pages, spool-aware binary detection + v2.51.36
- perf(simd): trie-dispatch content type classifier, phf hop-by-hop header filter + v2.51.35
- perf(simd): zero-alloc asset extension check, memchr robotparser, branch-free networking_capable + v2.51.34
- fix(concurrency): eliminate blocking paths — Notify-based ChannelGuard, TabCloseGuard timeout, eager abort_all + v2.51.33
- chore: auto-sync internal dep versions in release.sh
- chore: fix all internal dep versions, bump to v2.51.32
- chore: fix spider_agent dep versions, bump to v2.51.31
- fix(chrome): close leaked tabs preventing hedge deadlocks, remove forced retry + v2.51.30
- chore: regenerate Cargo.lock for v2.51.29
- refactor(parallel_backends): remove LightPanda references, rename to Cdp + v2.51.29
- chore: regenerate Cargo.lock for v2.51.28
- fix(parallel_backends): eliminate redundant grace period + double stat recording + v2.51.28
- chore: regenerate Cargo.lock for v2.51.27
- fix(parallel_backends): restore RNG primary/backend launch swap (0-1ms) + v2.51.27
- chore: regenerate Cargo.lock for v2.51.26
- perf(parallel_backends): true primary-vs-backend racing + remove primary jitter + v2.51.26
- chore: regenerate Cargo.lock for v2.51.25
- perf(uring_fs): unified read_file_chunked with auto uring/tokio fallback + v2.51.25
- chore: regenerate Cargo.lock for v2.51.24
- fix(balance): uring_fs fallback to tokio::fs on read failure + v2.51.24
- chore: regenerate Cargo.lock for v2.51.23
- fix(sync): prevent subscription_guard deadlock when receivers dropped + v2.51.23
- chore: regenerate Cargo.lock for v2.51.22
- fix(perf): restore uring_fs fast path for spool reads + parallel DNS pre-resolve + v2.51.22
- chore: regenerate Cargo.lock for v2.51.21
- perf(balance): true streaming disk I/O for lol_html link extraction + clippy fixes + v2.51.21
- chore: regenerate Cargo.lock for v2.51.20
- fix(sync): eliminate ChannelGuard livelock on repeated lock() calls + v2.51.20
- chore: regenerate Cargo.lock for v2.51.19
- fix(balance): eliminate sync blocking disk I/O from async link extraction + v2.51.19
- chore: cargo fmt, clippy fixes, bump all crates to v2.51.18
- chore: cargo fmt, clippy fixes, bump all crates to v2.51.18
- perf: eliminate UTF-8 round-trips in Chrome content injection pipeline
- perf(simd): SIMD UTF-8 validation, zero-copy link streaming, and byte-level optimizations
- chore: regenerate Cargo.lock for v2.51.17
- test(e2e): Chrome hedge parallel_backends E2E tests + v2.51.17
- test(e2e): add Chrome + hedge + parallel_backends E2E tests
- chore: regenerate Cargo.lock for v2.51.16
- fix(hedge+parallel_backends): collect backend results in hedge Chrome path + atomic EMA updates + v2.51.16
- perf(balance): pressure-aware spool + lock-free vitals for scaling + v2.51.15
- fix(parallel_backends): add backend_timeout_ms to test constructors
- fix(parallel_backends): hard outer timeout prevents stalled backends from blocking Chrome + v2.51.12
- feat(chrome): multi-endpoint connection failover + v2.51.11
- perf(hedge): zero-latency signal-based Chrome hedge escalation + v2.51.10
- feat(hedge): Chrome hedge via new WS connection on stall + OOM-safe pre-alloc caps + v2.51.9
- perf(async): eliminate all spawn_blocking from spool I/O + async DNS via hickory-resolver + v2.51.8
- perf(balance): tokio mpsc cleanup — zero-CPU idle, instant wake + v2.51.7
- perf(balance): raise per-page spool threshold to 3 MiB, memory-first design + v2.51.6
- perf(balance): batch-drain cleanup task, reduce idle polling latency + v2.51.5
- fix(balance): remove thread::sleep from flush_cleanup, use yield_now + deadline + v2.51.4
- perf(balance): tokio-native spool cleanup task + v2.51.3
- perf(balance): dedicated cleanup thread for spool file deletion + v2.51.2
- fix(balance): binary_file flag in transformations, clippy fixes + v2.51.1
- feat(page): add binary_file flag for disk-safe binary detection + v2.51.0
- fix(balance): make shed_page_html fully async + v2.50.9
- refactor(balance): transform_content_send uses byte pipeline by default, clean up dead code + v2.50.8
- feat(balance): public async content API — get_html_async, stream_html_bytes_async + v2.50.7
- perf(balance): fully async spool I/O, simplified decision logic + v2.50.6
- perf(balance): async disk I/O for crawl paths, eliminate runtime blocking + v2.50.5
- perf(balance): Arc-shared spool files, cached decisions, EMA adaptive thresholds + v2.50.4
- test(balance): add subscription e2e test for disk streaming + v2.50.3
- fix(balance): gate spool code behind not(decentralized) + v2.50.2
- chore(balance): add unconditional is_html_on_disk() API + v2.50.1
- feat(balance): disk-backed HTML spool for memory-balanced crawling + v2.50.0
- perf(parallel_backends): drop backend JoinSet immediately after race to free losing pages + v2.49.1
- feat(adblock): add adblock_easylist feature, bump all crates to 2.49.0
- perf(spider): automatic proxy-host DNS caching with adaptive refresh
- perf: skip HTML link extraction for single-page crawls + v2.48.52
- chore: remove lightpanda feature flag from parallel_backends + v2.48.51
- fix: filter honeypot links (aria-hidden) in all CSS selectors + v2.48.50
- perf: zero-alloc SIMD case-folding hash via case_insensitive_string 0.2.11 + v2.48.48
- perf: single-pass domain extraction, faster parent_host_match + v2.48.47
- perf: zero-copy HTML in all link extraction paths + right-sized buffers + v2.48.46
- perf: zero-copy XML stripping, right-sized response buffers + v2.48.45
- perf: eliminate String allocation per link in push_link hot path + v2.48.44
- fix: memory safeguards — bounded extra_links, eliminate page clones in channel sends + v2.48.43
- fix: add memory pressure safeguards to parallel_backends + v2.48.42
- fix: DNS errors (525) no longer trigger should_retry in get_error_status_base + v2.48.41
- fix: move crawl delay to only apply before HTTP requests, not every loop iteration (#383) + v2.48.40
- chore: bump v2.48.39, downgrade CDP timeout log to debug
- fix: remove stray cache_namespace arg from run_remote_multimodal_extraction call + v2.48.38
- fix: add missing cache_namespace param in smart feature call sites + v2.48.37
- feat: configurable cache namespace for partitioning cache keys + v2.48.36
- fix: add missing content_map field to decentralized Page struct + v2.48.35
- feat: multi return_format support for spider cloud + v2.48.34
- feat: SpiderCloudReturnFormat enum for type-safe return_format + v2.48.33
- docs: improve README examples with limit param and structured output + v2.48.32
- feat: add page.get_content() + spider_cloud_markdown example + v2.48.32
- fix: remove Option from subscribe, match dep versions, fix all-features compile + v2.48.31
- fix: remove Option from subscribe, always compile + v2.48.30
- docs: add CLAUDE.md development guide with Spider Cloud configs, examples, and testing
- fix: enable doc_cfg for docs.rs builds, clean up README + v2.48.29
- simplify README, placeholder API key safeguard + v2.48.25
- fix: ungate `use url::Url` for decentralized feature + v2.48.24
- perf: pre-compile lol_html CSS selectors via OnceLock + v2.48.23
- perf: zero-alloc per-link From<&str>, cache drain mem::take + v2.48.22
- perf: arena/buffer reuse — zero-alloc CaseInsensitiveString, EMA HashSets, XML buf pool + v2.48.21
- perf: add mimalloc allocator option, allocator benchmarks, README guidance
- perf: sync CPU/memory state in get_semaphore, pre-allocate link HashSets + v2.48.20
- perf: fix last fake-async rewriter in utils/mod.rs, add cooperative yield + v2.48.19
- perf: eliminate per-link clone in insert_link — accept &ref, clone only on insert + v2.48.18
- perf: eliminate fake-async in crawl loops, sitemap parsing, CSS selectors + v2.48.17
- perf: lock-free cache, sync memory state, cooperative rewriter yields + v2.48.16
- fix: smart mode upgrade heuristics — weighted scoring, tracker filtering, anti-bot scan + v2.48.15
- feat: CLI authenticate command for Spider Cloud API key + v2.48.14
- fix: docs.rs build failure — explicit feature list + v2.48.12
- fix: docs.rs build failure — replace all-features with explicit feature list
- fix: retry logic hardening — proxy-aware 401, full jitter backoff, stale page breaks + v2.48.11
- test: comprehensive error handling and retry logic validation
- fix: MCP scrape use with_limit(1) to prevent crawling (#380) + v2.48.10
- fix: parallel backends CDP/WebDriver run without lightpanda/servo features + v2.48.9
- fix: page.html private field + unused_mut warning + v2.48.8 release
- fix: suppress dead_code warnings in spider_mcp state
- chore: update Cargo.lock for v2.48.7
- chore: bump all crates to v2.48.7
- feat: per-backend proxy support for parallel backends
- feat: parallel backends memory optimizations + v2.48.6 release
- fix: release.sh cd always returns to root on publish failure
- feat: spider_mcp Spider Cloud support via SPIDER_API_KEY
- chore: align internal dep versions for v2.48.5 release
- fix: run remote cache before parallel backends in Chrome path
- docs: add spider_mcp to README
- chore: update Cargo.lock for v2.48.4
- feat: add spider_mcp crate — MCP server for Spider web crawler
- chore: update Cargo.lock for v2.48.3
- fix: parallel backends probe-on-first-request + connect timeout
- fix: restore full setup_chrome_events for LightPanda
- feat: full Chrome config parity for LightPanda backend
- chore: update Cargo.lock for v2.48.2
- chore: clippy fixes + bump crates to v2.48.2
- perf: pass full handler config to LightPanda via connect_with_config
- chore: update Cargo.lock for v2.48.1
- chore: bump crates to v2.48.1
- feat: randomise launch order between primary and backends
- feat: add parallel crawl backends — race LightPanda/Servo alongside primary (v2.48.0)
- style: fix formatting in website.rs cfg attribute
- feat(cli): add --return-format flag for content transformation (v2.47.93)
- fix: gate unused variables behind cache feature cfg
- chore: update Cargo.lock for v2.47.92
- feat(cli): add --warc flag for WARC output (v2.47.92)
- style: fix formatting in warc_chrome example
- docs(examples): add WARC output examples for HTTP and Chrome modes
- chore: bump all crates to v2.47.91
- feat(crawl): add lock-free WARC 1.1 output for web archiving (v2.47.91)
- feat(crawl): add auto-throttle and ETag conditional caching (v2.47.90)
- fix(crawl): fix compile error in crawl_concurrent_raw - clone antibot patterns before spawn
- feat(crawl): user-configurable antibot detection patterns via CustomAntibotPatterns
- feat(crawl): expand antibot detection — 7 new providers, server header scanning, 16 new patterns
- fix(crawl): restore 6 antibot body patterns accidentally dropped in v2.47.85
- perf(crawl): optimize antibot detection — use LeftmostFirst + find() instead of iterator
- chore: bump all crates to v2.47.84
- fix(crawl): expand anti-bot detection patterns for Cloudflare, Akamai, PerimeterX, DDoS-Guard
- chore: bump all crates to v2.47.83
- fix: resolve cargo fmt and clippy errors across workspace
- fix(crawl): retry antibot 403s instead of treating as hard blocks
- chore: bump all crates to v2.47.81
- fix(spider): reclassify Chrome error pages as 599, trigger retry
- fix(crawl): detect Chrome error pages (ERR_TUNNEL_CONNECTION_FAILED etc.) as invalid content, bump v2.47.80
- fix(audit): ignore RUSTSEC-2026-0049 rustls-webpki CRL advisory
- fix(crawl): add subscription_guard to all single-page fast paths, bump v2.47.79
- chore(deps): bump all crates to v2.47.78
- style: cargo fmt spider_worker
- fix(deps): upgrade warp 0.3→0.4 to resolve RUSTSEC-2026-0049
- style: cargo fmt
- perf(cache): optimize SmartCache, SelectorCache, and cache read/write paths, bump v2.47.77
- chore(deps): bump chromey v2.42.5, update lockfile, bump v2.47.76
- style: cargo fmt
- docs(spider): add doc(cfg) feature badges for traits, enable docs.rs all-features, bump v2.47.75
- fix(chrome): skip redirect-caused net::ERR_ABORTED instead of aborting navigation
- feat(spider): add PageData, Crawler trait abstractions, bump v2.47.73
- perf(dns_cache): zero-alloc cache hits via Arc<[SocketAddr]> + index iterator, bump v2.47.72
- perf(spider): skip robots.txt for single-page, enable TCP keepalive always, wire DNS cache into reqwest resolver, bump v2.47.71
- chore(deps): bump strum 0.28, phf 0.13, gemini-rust 1.7, chromey 2, llm_models_spider 0.1, bump v2.47.70
- perf(cache): batch LRU eviction to reduce write-lock hold time, fix aws-lc-sys audit, bump v2.47.69
- fix(spider): add decentralized stubs for set_url_parsed_direct and links_full, bump v2.47.68
- perf(spider): remove unnecessary Box allocations and redundant string conversions, bump v2.47.67
- feat(cache): dump chrome responses to remote cache server
- fix(spider): fall back to HTTP crawl when Chrome is unavailable (#373), bump v2.47.65
- feat(spider): proxy support for LLM HTTP requests (#378), bump v2.47.64
- test(spider): timing tests for cache_mem + chrome_remote_cache skip_browser, bump v2.47.63
- perf(spider): cache_mem auto skip_browser for chrome_remote_cache too, bump v2.47.62
- perf(spider): cache_mem auto-enables skip_browser for Chrome mode, bump v2.47.61
- fix(spider): deterministic CDP event listener shutdown via watch channel, bump v2.47.60
- feat(spider): Chrome mode remote_addr from CDP Network.responseReceived, bump v2.47.59
- fix(spider): add 3s timeout to remote cache fallback in chrome_remote_cache-only path, bump v2.47.58
- fix(spider): remote cache fallback for Chrome mode when cache_mem enabled, bump v2.47.57
- perf(spider): trie Box<str> keys + manual byte-walk + memchr dot scan, bump v2.47.55
- perf(spider): SIMD-accelerated byte scanning via memchr, u64 word comparisons, unrolled FNV hash, bump v2.47.54
- perf(spider): bloom filter bitmask addressing + inline early-exit, interner zero-clone extend, bump v2.47.53
- perf(spider): eliminate hot-path allocations — push_link, attr removal, normalize_html, bump v2.47.52
- perf(spider): NUMA thread pinning + zerocopy wire parsing, bump v2.47.51
- perf(spider): bytes::Bytes zero-copy page passing, mmap+hugepages bloom filter dedup, bump v2.47.50
- perf(spider): data structure optimizations — O(1) frontier pop, zero-alloc header matching, eliminate HashSet clones, bump v2.47.49
- feat(spider_agent): complete RemoteMultimodalConfigs builder pattern (#376), bump v2.47.48
- feat(spider_cli): add --chrome-connection-url, --cookie, --stealth CLI flags, bump v2.47.47
- perf(spider): Chrome request reliability — min 2 retries, backoff, fresh tabs, browser health signal, bump v2.47.46
- fix(spider): Chrome status code defaults to 599 instead of 200 on no response, bump v2.47.45
- feat(spider): Spider Browser Cloud integration via wss://browser.spider.cloud, bump v2.47.44
- fix(spider): fix clippy redundant_guards warning in get_html_cow, bump v2.47.43
- perf(spider): memory optimizations — Cow HTML, bounded caches, tuned buffers, bump v2.47.43
- perf(spider): eliminate redundant allocations in crawl hot paths, bump v2.47.42
- feat(spider): adaptive hedge tracking for HTTP paths, bump v2.47.41
- feat(spider): variance-aware adaptive hedge delay with auto-tuning signals, bump v2.47.40
- fix(spider): fix Chrome hedge never firing on slow sites, use raw EMA for adaptive delay, bump v2.47.39
- fix(spider): add backoff for 599/598 proxy errors in get_timeout(), bump v2.47.38
- feat(spider): add Chrome hedged requests with adaptive delay tracking, bump v2.47.37
- fix: configurable automation timeout, uring_fs test deadlock, unused import (#374), bump v2.47.36
- fix: clippy unnecessary casts and hex literal grouping in splice, bump v2.47.35
- chore: bump v2.47.34
- feat(spider): add tcp_fastopen, splice, io_uring send/recv, bump v2.47.33
- revert(spider): remove needs_retry() from initial_html_length check, bump v2.47.32
- fix(spider): set initial_html_length to 0 when page needs retry, bump v2.47.31
- fix(spider): set initial_html_length to 0 for empty pages, bump v2.47.30
- fix: retry on empty content (200 + no body) and truncated responses
- fix: remove useless Into::into after double-box removal (clippy)
- feat: enable dns_cache, rate_limit, request_coalesce in __basic defaults
- feat: 10 performance & reliability enhancements, bump to v2.47.28
- fix: ensure 599/598 errors always set should_retry for website-level retry logic
- fix: inline let binding to satisfy clippy let_and_return lint
- fix: remove retry suppression for request errors, add status-based retry classification
- feat: async file I/O for TempStorage + ExperienceMemory, bump to v2.47.25
- chore: bump all crates to v2.47.24
- style: cargo fmt
- fix: remove unnecessary i32 cast on libc::EINPROGRESS
- feat: io_uring TCP connect + lightweight background runtime, bump to v2.47.23
- fix: replace tokio-uring with raw io-uring crate + kernel probe, bump to v2.47.22
- fix: eliminate panic sites and unsafe indexing, bump to v2.47.21
- fix: detect and flag truncated responses from upstream Content-Length mismatch
- chore: bump all crates to 2.47.18
- fix: enforce hard wall-clock timeout on all crawl entry points
- feat: Auto API mode — api.openai.com uses Responses API, others use Completions, bump to v2.47.17
- feat: OpenAI Responses API support (opt-in), bump to v2.47.16
- feat: expose underlying HTTP client and configurable timeout on agent, bump to v2.47.15
- fix: auto-append /chat/completions for OpenAI-compatible URLs, add proxy support to agent, bump to v2.47.14
- fix: DNS resolve errors (525) skip all retry loops, bump to v2.47.13
- chore(crawler): add v2.47.12
- fix: 599 status with real content no longer misclassified as ConnectError, bump to v2.47.11
- fix: include webdriver feature Box<[u8]> conversion, bump to v2.47.10
- perf: reduce Page html from Box<Vec<u8>> to Box<[u8]>, bump to v2.47.9
- perf: unbox redirect_limit/cookie_str/request_timeout, remove charset alloc, bump to v2.47.8
- perf: Arc for external_domains_caseless, skip charset lowercase alloc, bump to v2.47.7
- perf: eliminate wasteful allocations on hot paths, bump to v2.47.6
- fix: lazy anti-bot body check in smart mode, exhaustive pattern tests, bump to v2.47.5
- feat: detect Alibaba TMD anti-bot, force Chrome upgrade in smart mode, bump to v2.47.4
- fix: validate 598/599 as genuine errors before ConnectError, bump to v2.47.3
- fix: 598/599 status codes return ConnectError, bump all crates to v2.47.2
- feat(chrome): add network idle timeout to prevent tarpit during CDP navigation
- feat: add per-chunk idle timeout to prevent tarpit/slow-drip antibot attacks
- chore: bump spider_agent to v2.46.3 (v2.46.2 already published)
- chore: bump all crates to v2.46.2
- feat(agent): retry on different model for 502/503/429/5xx errors
- feat(agent): add Anthropic Messages API thinking/extended thinking support
- style: fix rustfmt formatting in reqwest 0.13 upgrade
- feat: upgrade to reqwest 0.13 ecosystem
- feat(agent): eliminate panics, silent failures, and unbounded growth
- feat(agent): action handler hardening — duration caps, JS escape, timeout guards
- feat(agent): eval timeouts, spawn concurrency limiter, PageWaitStrategy, panic hardening
- feat(agent): action feedback, partial result preservation, and JS error capture
- style: fix rustfmt formatting in smart_vs_chrome tests
- chore(release): 2.45.25
- fix: use final redirect URL as base for relative link resolution (#364)
- chore: ignore transitive audit advisories with no direct fix
- chore: ignore RUSTSEC-2025-0134 (rustls-pemfile via warp)
- chore: ignore RUSTSEC-2026-0002 (lru 0.13.0 via wreq)
- docs: backfill CHANGELOG v2.45.1-v2.45.22 and add .editorconfig
- chore: remove cargo dependabot, keep github-actions only
- chore: fix clippy warnings and formatting across workspace
- chore(deps): bump flexbuffers 2->25, async-openai 0.32->0.33, clap 4.5.59->4.5.60, actions
- docs: rewrite README, add issue/PR templates, security policy, CI improvements
- docs: add Contributor Covenant v2.1 Code of Conduct
- docs: rewrite CONTRIBUTING.md and add zero-config quick start to README
- fix(cache): proper HTTP staleness for Chrome-cached pages
- fix(cache): Period policy bypasses HTTP is_stale for Chrome-rendered pages
- chore: bump all workspace crates to v2.45.22
- fix(cache): enable cache_chrome_hybrid_mem feature for Chrome cache writes
- chore: bump all crates to 2.45.21, chromey 2.38.4
- chore(agent): fix model router import
- docs(spider_agent): add SKILLS.md for engine skill integration reference
- chore(chrome): 2.45.19 root sandbox default launch
- fix: add no_sandbox() to headless BrowserConfig (#354)
- test: comprehensive crawler-test.com integration suite (302 tests, 408 URLs)
- feat: per-round model pool routing for cost-optimized automation + bump 2.45.18
- perf: skip model scoring for pools ≤ 2 models + bump 2.45.17
- test: comprehensive multi-LLM router reliability tests + bump 2.45.16
- release(cli): 2.45.15 wait_for
- feat(cli): add wait-for capabilities to spider_cli (#352)
- fix: Chrome mode honors wait_for config for networkIdle before HTML extraction
- fix: smart mode lifecycle waiting — match Chrome coverage without re-fetch
- fix: auto-retry www. URLs on SSL protocol errors — strip www prefix
- fix: reject empty HTML from all cache and seeded resource paths
- fix: invalidate empty HTML shell responses on cache read path
- feat: deferred Chrome — cache-only crawl phase before browser launch
- feat: cache-first fast path — skip browser/HTTP when cache has data
- fix: add Website::with_hedge() forwarding method + bump to v2.45.7
- feat: add work-stealing (hedged requests) for slow crawl requests
- fix: skip HTML-specific heuristics in cache empty check for non-HTML content
- chore: bump all workspace crates to v2.45.4
- fix: skip caching empty/near-empty HTML responses
- chore: bump all workspace crates to v2.45.3
- chore: bump spider to v2.45.3
- fix(uring_fs): fix compile errors when io_uring feature is enabled
- perf(io_uring): add StreamingWriter for streaming file I/O, bump v2.45.2
- feat(io_uring): expand io_uring integration for file I/O, bump v2.45.1
- refactor(agent): extract types and HTML cleaning into spider_agent_types and spider_agent_html crates
- chore(deps): major version bumps, bump v2.45.0
- chore: bump v2.44.42
- perf(website): use take() instead of clone() for subdomain base URL
- fix(website): add tests for subdomain relative URL resolution, bump v2.44.41
- fix(website): use page's own URL for relative link resolution on subdomains
- chore: clarify network event collection comment (#350)
- feat(agent): integrate llm_models_spider v0.1.9 with smart model selection
- docs: add missing feature flags, Spider Cloud example, and integration docs
- refactor(agent): move all skill content to spider_skills crate
- feat(agent): add L15-L48 skills, NEST/CIRCLE engine loops, haiku benchmark
- fix: add should_use_chrome_ai to stub RemoteMultimodalConfigs
- feat(agent): add L8-L14 skills, WAM engine loop, and Chrome AI improvements
- chore(agent): add improved steps
- test(website): fix rotation cases
- feat(agent): optimize text-only rounds and bump workspace crates
- chore: fix lint warnings and feature-combo build regressions
- fix: improve agent concurrency and ensure all-feature builds
- feat(cache): optimize automation caching and bump crates to 2.44.33
- Add crawl_smart cache parity test and bump crates to 2.44.32
- Improve cache skip-browser flows and add remote cache e2e example
- feat: add spider cloud end-to-end examples and bump crates to 2.44.30
- Update Cargo.lock for 2.44.29
- Improve remote multimodal automation reliability and release 2.44.29
- Expose optional automation reasoning in metadata and bump 2.44.28
- release: bump workspace crates to 2.44.27
- feat(spider_cli): add runtime http/headless mode controls and release 2.44.26
- chore(client): add fallback agent declaring
- docs: update dual-model example with per-endpoint URL and API key configuration
- feat: dual-model routing examples and comprehensive tests
- feat: extraction-only mode optimization for single-round data extraction
- docs: add Spider Cloud reliability integration section to README
- fix: feature flag compilation issues across multiple feature combos
- test(ignore): add test_crawl_subdomains to ignore
- fix: feature flag compilation issues across wreq, agent_full, and cache combos
- chore: bump to v2.44.20 for release
- fix: broken chrome feature — missing `relevant` field and cfg gate on detect_cf_turnstyle
- chore: bump to v2.44.19 for release
- feat: integrate spider_skills crate for skill types and web challenge definitions
- feat(agent): add URL-level relevance pre-filter for crawling, bump to v2.44.18
- perf: fast-path trie protocol detection, remove redundant char boundary check
- perf: optimize trie, robot parser, and URL preparation hot paths, bump to v2.44.17
- feat(agent): add relevance gate for remote multimodal crawling
- perf: optimize core pipeline hot paths, add benchmarks, bump to v2.44.15
- chore: bump versions to 2.44.14
- fix(agent): use CDP mouse methods for all click actions to maintain position tracking
- docs: simplify root README intro
- docs: add spider_cloud to root README, improve API response handling
- chore: bump versions to 2.44.13
- feat(spider): add spider_cloud integration and S3 skills loading
- chore: bump versions to 2.44.12
- feat(spider_agent): add dual-model routing and long-term experience memory
- fix(spider): fix broken glob feature compilation and logic errors
- chore: bump versions to 2.44.10
- feat(spider_agent): improve skill triggers and board reading for web challenges
- feat(spider_agent): improve skill strategies for real browser interaction
- feat: sync skill content with improved efficiency directives
- chore: bump versions to 2.44.9
- feat(spider_agent): add Claude-optimized automation features
- chore: bump versions to 2.44.8
- feat(spider_agent): add concurrent page spawning with OpenPage action
- fix(spider_agent): update system_prompt_compiled test for locked prompt
- chore: bump versions to 2.44.7
- feat(spider_agent): integrate llm_models_spider for auto-updated vision detection
- feat(spider_agent): add inline vision model detection
- chore: bump versions to 2.44.6
- feat(spider): enable HTTP extraction without agent_chrome feature
- feat(agent): enhance CAPTCHA handling and lock system prompt
- feat(agent): improve web automation for CAPTCHAs and visual challenges
- docs(readme): add spider_agent section and link
- chore(examples): add remote_multimodal_multi example
- feat(agent): consolidate automation into spider_agent with seamless feature integration
- chore(example): add extraction example multi
- chore(website): fix scrape subscription hang
- chore(examples): add remote_multimodal
- chore: bump all crates to v2.44.3 for version parity
- fix(spider): keep spider_agent optional, add agent_webdriver feature
- feat(spider): make spider_agent a required dependency
- feat: granular usage tracking in AutomationUsage
- chore(spider): bump to v2.43.23, update spider_agent to 0.6
- feat(spider_agent): add usage limits, custom tools, and granular tracking
- feat(automation): add api_calls tracking and HTTP extraction support
- chore(spider): bump to v2.43.21, update spider_agent to 0.5
- feat(spider_agent): add complete WebAutomation parity and PromptUrlGate
- docs: expand v2 changelog with comprehensive feature documentation
- docs: update changelog for v2.43.20 and spider_agent releases
- chore(spider): update spider_agent dependency to 0.4
- feat(spider_agent): add performance optimizations for automation
- feat(spider_agent): add comprehensive automation module
- chore(spider_agent): bump version to 0.2.0
- feat(spider_agent): enhance memory with URL/action/extraction history
- feat(spider_agent): add webdriver support via thirtyfour
- feat(spider_agent): add chrome browser and temp storage support
- feat(spider_agent): add usage tracking and multimodal example
- fix(spider): fix doctest and update chromey for adblock compatibility
- feat(spider_agent): add standalone multimodal agent crate
- fix(search): use reqwest::Client directly for cache feature compatibility
- chore(docs): fix rustdoc broken intra-doc links
- feat(search): add search_and_extract and research methods
- feat(search): add web search integration with multiple providers
- perf(automation): byte-size-based smart cleaning for best performance
- feat(cache): add skip browser mode + smart HTML cleaning
- fix(automation): correct lol_html API usage and add intelligent screenshot detection
- feat(automation): add Phase 3 agentic features - autonomous agent, action chaining, error recovery
- feat(automation): add Phase 2 agentic features - selector cache, structured outputs, map API
- feat(automation): add simplified agentic APIs - act(), observe(), extract()
- feat(automation): add agentic memory for multi-round automation
- feat(automation): add prompt-based website configuration
- perf(website): remove unnecessary clones and allocations
- test(automation): add tests for JSON extraction from thinking models
- fix(automation): improve JSON extraction with proper brace matching
- feat(automation): add llm_json for robust JSON parsing from LLM outputs
- feat(automation): add HTTP extraction support without chrome feature
- fix(automation): improve best_effort_parse_json_object parsing
- fix: extract JSON from LLM responses with reasoning text
- feat(automation): add ExtractionSchema for structured extraction output
- fix(page): add remote_multimodal fields to decentralized Page struct
- feat(automation): add token usage tracking and extraction support for RemoteMultimodalEngine
- chore: update to spider_fingerprint 2.38.0 from crates.io
- feat(webdriver): integrate spider_fingerprint stealth spoofs
- docs: refresh README with streaming, Chrome, and WebDriver examples
- feat(webdriver): add automation script support
- chore(spider): bump version to 2.41.0
- feat(spider): add WebDriver support via thirtyfour crate
- test(page): add feature flag test coverage
- fix(website): gate seeded page link extraction for decentralized feature
- chore(spider): bump version to 2.40.4
- fix(website): extract links and metadata from seeded pages
- fix(page): fix metadata extraction in streaming functions
- fix(page): add metadata extraction for HTTP requests
- feat(chrome): add remote multi_modal web automation
- chore(page): add missing gemini
- chore(page): add forbidden validation retry
- chore(page): add set_duration_elasped
- chore(solvers): add lemin
- chore(website): add warm model
- feat(solver): add built in gemini nano support
- chore(mod): add gemini solve
- chore(smart): add handles
- chore(smart): add imperva handle
- chore(smart): add sessionStorage detection
- release(minor): 2.39.0
- Add `with_on_should_crawl_callback_closure` (#346)
- chore(website): add modify http client builder
- chore(website): add with_referrer builder
- chore(chrome): add extra cleanup caps
- chore(mod): fix request error compile
- chore(website): add hint setters
- Fix breakage on openai example (#345)
- chore(chrome): add header cleanup
- chore(smart): add cypher mismatch check retry
- chore(cookies): add cookie jar re-use
- chore(chrome): fix bytes calculation smart rendering
- chore(website): add extra_information flag
- feat(gemini): add Gemini AI support for dynamic browser scripting (#344)
- chore(website): add extra info storing
- feat(website): add cmd crawling
- fix(chrome): add automatic chrome executable detection (#343)
- chore(network): add blacklist patterns
- chore(page): fix smart rendering
- chore(website): add pub buildable crawler
- chore(website): add copy crawl status
- chore(smart): add xml request detection
- chore(chrome): add whitelist patterns
- chore(chrome): add click automations
- chore(page): add empty trim check
- chore(validation): add to list
- chore(page): add false validation
- chore(page): add template false 403
- chore(website): add initial bytes tracking
- chore(page): add empty html
- chore(chrome): add emulation check
- chore(chrome): add cache cfg
- chore(smart): add react app ssr check
- chore(smart): fix empty page assign
- chore(page): add common type list ignore
- chore(smart): add hydration list
- chore(smart): add pierced nodes selecting frames
- chore(page): add gateway timeout
- chore(timeout): add gateway delay
- chore(page): add page empty fix
- chore(smart): add nuxt detection
- chore(smart): fix redirect assign
- chore(smart): fix page rendering on re-render
- chore(smart): revert watch methods
- chore(chrome): fix rewrite content base tag
- chore(smart): fix script tag removal
- chore(smart): fix redirect page content
- chore(configuration): fix with_wait_for_idle_network0 and with_wait_for_almost_idle_network0 builder
- chore(cache): add waf url check
- chore(real_browser): fix cf detection
- chore(page): fix http network error status code
- chore(chrome): add initial document status code fix
- chore(chrome): fix compile remote cache
- Fix "real_browser" disabled (#336)
- chore(cache): fix cache_options disk flag [#334]
- feat(chrome): add remote cache support
- chore(website): add connect error status
- chore(crate): add chrome_serde_stacker flag
- chore(website): fix chrome initial crawl status
- feat(chrome): add idle_network0 almost_idle_network0 wait for
- perf(chrome): add command send only closing chrome
- chore(chrome): add default ua & ref launch chrome headers
- chore(chrome,cache): add time check retrieval
- chore(website): add dynamic link find
- chore(website): add timeout builder methods
- chore(chrome): remove extra client header
- chore(chrome): fix cache compile
- chore(chrome): fix cache options compile
- chore(chrome): add shared hybrid cache retrieval
- chore(cli): add proxy_url [#330]
- chore(chrome): fix handler config [#329]
- chore(chrome): add event data handling internally
- chore(chrome): add wire bytes blocking
- chore(cli): fix download unicode handling [#325]
- chore(chrome): add stop_loading byte limit
- chore(chrome): fix compile [#328]
- chore(parser): add base tag respecting [#323]
- spider_cli: fix duplicated argument -r (#324)
- chore(chrome): fix max page bytes cancelling
- chore(page): fix invalid dns retry handling
- chore(chrome): add type automation
- chore(website): add get_extra_links
- feat(disk): add shared_state multi profiling
- chore(chrome): add click point
- chore(chrome): fix guard lock [#315]
- chore(chrome): fix clickable selector default
- chore(page): fix title metadata extracting
- chore(tracker): add event metadata
- chore(utils): fix duration elasped handling http [#306]
- chore(chrome): fix compile auto geolocation [#311]
- chore(website): fix platform interface binding [#309]
- chore(time): fix page duration tracking [#304]
- chore(utils): fix protocol handling invalid utf8
- chore(crate): fix compile spider_fingerprint
- chore(builder): add interface and local_address binding
- chore(cli): add status_code and remote_addr flags [#303]
- chore(cli): add time feature flag [#302]
- chore(real_browser): add simple click protected
- chore(sitemap): add xml check pre-parse
- chore(crate): bump quick-xml@0.38
- chore(cli): add return_headers [#301]
- chore(cli): add full-resources arg
- chore(utils): remove unused BufMut
- chore(utils): fix fetch advance out of bounds panic
- chore(utils): fix bufmut capacity panic
- chore(website): add csp enable/disable builder
- chore(chrome): add cdp handling cf
- feat(chrome): add auto_geo detection
- chore(emulation): add cdp set hardware_concurrency
- chore(emulation): add back userAgentData emulation
- chore(docs): fix spider_utils documentation crate
- chore(crate): cleanup crate purpose
- chore(chrome): add click_and_drag
- chore(docs): update relevant info
- chore(chrome): add full user_agent_metadata handling
- chore(chrome): add drag, double_click, and click_modifiers
- chore(chrome): fix firewall flags compile
- chore(chrome): re-export firewall-rustls
- chore(chrome): re-export firewall
- chore(fingerprint): add to gpu list
- chore(chrome): add navigator.platform emulating
- feat(chrome): add max_page_bytes control
- chore(chrome): add default flag serde0 pdl
- chore(chrome): add backwards compat scroll_by
- chore(chrome): fix delta for mousewheel
- chore(chrome): add to blocklist
- chore(chrome): add to blocklist
- chore(spider_chrome): add set_extra_headers and screencast
- chore(spider_chrome): add page set headers
- chore(spider_chrome): add rust-tls flag
- chore(fingerprint): fix firefox spoof
- chore(fingerprint): allow reference viewport
- chore(chrome): add scroll method
- chore(fingerprint): fix referer defaulting
- chore(fingerprint): fix empty agent handling
- chore(fingerprint): fix referrer header
- chore(fingerprint): add header detail control
- chore(fingerprint): add public title_case_header
- feat(fingerprint): add emulate_headers
- chore(fingerprint): remove openssl build defaults
- chore(fingerprint): add optional serde compile
- chore(fingerprint): add __pwInitScripts optional spoof
- feat(fingerprint): add centralized emulate
- chore(fingerprint): fix hardwareConcurrency name
- chore(fingerprint): fix hardwareConcurrency worker handling
- chore(fingerprint): add ua_full_version automation
- chore(fingerprint): fix chrome spoof
- chore(docs): fix chromiumoxide fork intro docs
- chore(fingerprint): add navigator.hardwareConcurrency spoof
- chore(examples): fix anti_bot with_user_agent
- chore(fingerprint): fix device_pixel_ratio mac defaults
- chore(fingerprint): fix hide chrome
- chore(fingerprint): prep fingerprint canvas noise
- chore(fingerprint): add profiles start
- chore(fingerprint): add env section
- chore(fingerprint): fix userAgentData getHighEntropyValues
- chore(chrome): add to block list
- chore(fingerprint): fix webgl worker spoof gpu
- chore(chrome): default enable service_workers
- chore(fingerprint): fix navigator.userAgentData prototype
- chore(fingerprint): fix plugins spoof toJSON
- chore(fingerprint): revert plugins
- chore(fingerprint): fix plugins spoof
- chore(fingerprint): add touchscreen spoof
- chore(chrome): add to block list
- chore(fingerprint): fix screen spoof color_depth
- chore(spoof): add plugins JSON.stringify fix
- chore(fingerprint): add spoof media labels
- chore(chrome): add thread rng re-use
- chore(chrome): add to block list
- chore(chrome): add to block list
- chore(docs): fix spoof_ref rustdoc
- chore(fingerprint): add re-export spider_fingerprint
- chore(fingerprint): add spider_fingerprint workspace
- chore(chrome): add permissions and notifications spoof
- chore(spoof): fix rust doc spoof_referrer_google
- chore(chrome): add spoof media codecs
- chore(chrome): add spoof screen
- chore(real_browser): add header merging
- chore(chrome): add to block list
- chore(crate): add __basic flag no tls [#297
- chore(crate): add http2 feature flag reqwest [#297]
- chrome_remote.rs: Add note about headless-browser (#296)
- chore(docs): add chrome-headless-shell recommendation
- chore(chrome): bump cdp revision 1457408
- chore(chrome): update browser,js pdl files
- chore(page): fix returning content on errors
- chore(chrome): remove unused cfg-if
- chore(chrome): add to block list
- chore(website): add initial_waf_detected and initial_page_retried
- chore(website): add set_status
- chore(chrome): add to block list
- chore(crate): bump sonic-rs@0.5
- chore(crate): bump lock backtrace@0.3.75
- chore(chrome): add browser dispose on end
- chore(chrome): add mobile chrome headers
- chore(chrome): fix concurrent context creation
- chore(chrome): fix create new context
- chore(chrome): add frame_id navigate params
- chore(page): remove unused og meta
- chore(page): add compact_str metadata
- chore(serde): add back serde serialize AIResults
- feat(page): add metadata extracting
- chore(chrome): fix cfg chrome_args
- chore(sitemap): add auto sitemap include whitelist
- chore(real_browser): add rng init re-use
- chore(http): add default error status
- chore(chrome): add to block list
- chore(chrome): add with_stealth_advanced stealth control
- chore(chrome): add webdriver real spoof
- chore(chrome): add gpu requestAdapter spoof
- chore(chrome): split mac and linux gpu spoof
- chore(chrome): add real native userAgentData spoof
- release(minor): 2.36.114 serde configuration defaults
- Add serde(default) for Configuration. (#290)
- chore(chrome): fix plugins script
- chore(crawler): add apache,openresty 403 detection
- chore(examples): add default fingerprint disable full pass anti-bot
- chore(chrome): fix stealth fingering mixing
- chore(chrome): fix wait_for_dom
- chore(chrome): add spoof fingerprint gpu with_fingerprint_advanced
- chore(openai): box gpt configs
- chore(openai): fix run openai call
- chore(chrome): fix sitemap crawling xml files
- chore(chrome): prep gpu fingerprint disable
- chore(chrome): add plugins spoof entry
- chore(chrome): fix missing bitness
- chore(website): add requires_javascript detection
- chore(disk): add optional sqlite builder method
- chore(chrome): add to block list
- chore(stealth): fix userAgentData setting
- chore(crawler): add headers clear method
- chore(chrome): fix turnstile detection
- chore(chrome): fix pdfViewerEnabled stealth
- chore(adblock): fix adblock compile [#288]
- chore(chrome): fix fingerprint script
- chore(chrome): fix header casing
- chore(chrome): fix stealth mode initial webdriver set
- chore(chrome): add spoof navigator userAgent high entro
- chore(website): add with_dismiss_dialogs builder method
- chore(page): fix duration elasped default compile
- chore(chrome): disable runtime default
- chore(chrome): fix viewport handling
- chore(sitemap): fix budget tracking
- chore(page): add 404 status parsing
- chore(spoof): fix trailing slash ref
- chore(client): fix aggresive keep-alive
- chore(website): add get_initial_anti_bot_tech
- feat(antibot): add anti-bot detection
- chore(chrome): add anti-bot detection
- chore(xml,page): fix xml parsing html
- chore(chrome): add unique isolated world generation
- chore(chrome,sitemap): fix xml parsing sitemap
- chore(sitemap): fix crawling sitemap return html response
- chore(cookies): fix duplicate cookie provider setup
- chore(headers): add viewport width
- chore(real_browser): add header preserving
- chore(website): fix client host header
- chore(decentralized): fix compile set_url_parsed_direct_empty
- chore(chrome,sitemap): add chrome rendering sitemap
- chore(page): fix error status code handling
- chore(website): fix xml parsing links [#284]
- release: 2.36.61
- fix: errors with page.authenticate (#286)
- chore(chrome): fix sitemap xml parsing content
- chore(chrome): revert enable stealth with agent
- chore(chrome): fix custom event paused event listener
- chore(chrome): fix cache disable duplicate
- chore(real_browser): fix arch and model headers
- chore(website): fix http dup ua
- chore(headers): fix header casing http
- chore(chrome): fix unknown status code security tracking
- chore(chrome): fix gpu spoof
- chore(chrome): fix spoof webdriver linux
- chore(chrome): fix spoof runtime
- chore(real_browser): add sec bitness,form,scheme default headers
- chore(chrome): fix invalid redirects
- chore(real_browser): add missing Priority header
- chore(chrome): add rtf file handling
- chore(real_browser): add linux fp target
- chore(real_browser): fix linux default headers
- chore(chrome): fix default accept lang
- chore(chrome,headers): fix compile store headers
- chore(chrome): add missing linux headers
- chore(chrome): fix default fp overwrites
- chore(chrome): fix wait for dom exit
- chore(chrome): fix compile fp
- chore(chrome): add platform fp handling
- chore(chrome): fix casing headers raw real_browser
- chore(chrome): add to block list
- chore(chrome): add dynamic header preserve order
- chore(chrome): fix hybrid cache storing valid headers
- chore(page): fix invalid page handling content bytes
- chore(page): fix document status code handling
- chore(real_browser): add accept-encoding client-hint
- chore(client): revert ref redirect
- chore(real_browser): fix referer header order
- chore(real_browser): fix ref header order
- chore(website): add firewall bypass
- chore(real_browser): fix chrome Sec-Ch-Ua-Platform case
- chore(chrome): fix timeout subtracting
- chore(chrome): fix request_timeout default
- chore(transformations): add <document check
- chore(real_browser): add accept-language header
- chore(real_browser,chrome): add missing chrome headers
- chore(chrome): add real browser loading
- chore(chrome): add request timeout bytes ignore
- chore(crate): fix default features
- chore(decentralized): fix compile event map
- chore(chrome): add unknown status code mask
- chore(chrome): fix wait_for_dom updating
- chore(chrome): fix events 599
- chore(chrome): fix sonic-rs utf8 parse
- chore(chrome): fix generic unknown status firewall handling
- chore(chrome): add buffer timeout cancelled request
- chore(chrome): fix content hang
- feat(client): add rquest [#280]
- chore(chrome): fix queue [#282]
- release: 2.34.5
- feat(chrome): add support for wss connection (#279)
- chore(chrome): add page content asset fallback
- chore(chrome): add exact media extraction [#278]
- chore(chrome): add masked bytes env
- feat(chrome): add req/resp tracker [#254]
- chore(chrome): fix asset downloading [#278]
- chore(utils): fix DocumentSelectors serde compile error
- feat(chrome): add xhr block custom list
- chore(lib): re-export spider_firewall
- chore(chrome): add to block list
- feat(website): add crawl_timeout and with_crawl_timeout builder method
- chore(website): add robots parser reference method
- chore(chrome): add 598 chrome error
- feat(chrome): add ValidateChain web automation
- chore(chrome): fix automation scripts wait_for selector
- chore(chrome): add to block list
- chore(crate): revert fork reqwest-middleware
- chore(examples): cleanup mut evalulate
- chore(chrome): add to block list [#276]
- chore(cache): fix http-cache compile
- feat(nix): add default.nix spider,spider_cli [#266]
- chore(crate): bump async-openai@0.28
- feat(chrome): add asset handling pages [#275]
- chore(lib): remove jemalloc from lib
- chore(chrome): add ws dispose
- feat(chrome): add fetch_chrome_persisted reuse browser
- feat(website): add light website.fetch_chrome
- chore(control): remove clone on reader
- feat(builder): add with_no_control_thread
- chore(website): add setup website norobots
- feat(send): add crawl raw send
- feat(website): add chrome crawl_chrome_send thread safe
- chore(chrome): add to block list
- chore(decentralized): fix compile bytes
- chore(utils): add serde serialize DocumentSelectors
- perf(page,chrome): add direct bytes handling
- chore(chrome): fix wait_for events sequence
- chore(chrome): add navigation network cancel
- chore(chrome): fix chrome headed proxy [#272]
- chore(budget): fix shutdown
- chore(chrome): fix launch timeout
- chore(crawl): add shutdown set join_task clear
- chore(chrome): add initial page retry
- perf(conf): remove box indirection proxies, whitelist, and blacklist
- chore(chrome): fix initial page status code set
- chore(chrome): add infinite loop document reload protection
- chore(chrome): add to block list
- chore(chrome): add localhost proxy bypass
- chore(chrome): fix firewall compile
- chore(chrome): revert dispose quite context
- chore(chrome): revert dispose drop
- chore(chrome): fix ws handler disconnect dispose
- chore(chrome): add handler drop on dispose
- chore(sub): fix guard handling
- chore(chrome): add handler abort
- chore(chrome): fix default sender dispose context message
- chore(chrome): fix closing handler drop
- chore(chrome): add handler close ws errors
- chore(chrome): add web socket error closing
- chore(chrome): add about:blank proxy_bypass
- chore(chrome): remove loopback local
- feat(chrome): add bad website block chrome networking
- chore(chrome): add html error extracting
- chore(smart): fix retry client errors
- perf(smart): add base tag rewrite
- perf(bytes): remove BytesMut
- chore(chrome): add dispose_on_detach closing
- chore(smart): fix missing bytes transferred
- feature(smart): add lazy load chrome
- chore(website): add reset_status
- chore(utils): re-export is_bad_website_url
- chore(firewall): add automated list
- chore(chrome): add to block list
- chore(chrome): bump connection retries
- chore(examples): add subscribe_multiple
- chore(website): fix page empty status
- chore(chrome): add close browser cleanup
- chore(website): add new_page direct drop
- chore(docs): cleanup firewall readme
- feat(firewall): add start of spider_firewall
- chore(website): add same tasks joining
- chore(chrome): add to block list
- chore(website): fix scrape hang [#268]
- chore(glob): fix compile [#267]
- chore(chrome): add optional cache disable
- chore(chrome): add service worker default disable
- chore(chrome): add preset init commands
- chore(smart): add initial http fallback
- chore(chrome): add connection retry ws
- release(minor): add cli arg help v2.27.54
- chore(cli): trigger help page on missing arguments (#265)
- chore(chrome): add to block list
- chore(chrome): add info log errors
- chore(chrome): add WEBSOCKET_DEFAULTS env variable
- chore(website): add direct proxy control
- chore(crawler): add force proxy special protocol
- chore(proxies): add sock:// ignore http
- chore(chrome): add to block list
- chore(website): fix disk compile feature
- chore(transform): fix default features [#264]
- chore(website): fix queue ignore [#256]
- chore(chrome): add wait_for check
- chore(website): add set_url_only
- chore(chrome): add get_cookies browser context id
- chore(chrome): revert content handling
- chore(chrome): fix outer_html target
- chore(request): bump max_duration 5 mins
- chore(chrome): fix duration overflow
- chore(chrome): add cdp DOM.getOuterHTML
- chore(chrome): fix duration page handling timeout
- docs: fix unclear documentation around scraping (#262)
- chore(chrome): fix wait_for delay ordering
- chore(chrome): add to default allowed scripts
- chore(page): fix full_resources DOM watch compile [#260]
- chore(page): fix full_resources smart compile [#260]
- chore(abs): clear query pairs [#257]
- chore(chrome): fix rand@0.9 compile
- chore(crate): update rand@0.9
- chore(chrome): handle socks protocol default
- chore(page): return error for status
- chore(crate): update cron@0.15
- chore(page): cleanup unknown status code usage
- chore(chrome): add optional simd flag
- chore(chrome): enable tcp keep alive ws connection client
- chore(website): fix insert signature compile
- chore(page): add port validation links
- chore(chrome): update tokio-tungstenite@0.26
- perf(chrome): add simd json entry
- chore(chrome): add binary message handling
- chore(website): add normalize builder method
- chore(transformations): update fast_html2md@0.0.44
- chore(serde): fix configuration compile cron
- chore(page): fix empty base link joining
- chore(utils): add back trim on abs path convert
- chore(website): fix relative path merge  building
- chore(website): add normalize handling chrome
- chore(website): fix subscribe guard handling queues
- chore(utils): add normalizing start
- chore(config): set default depth 25
- chore(page): remove extra clone external domains
- feat(page): add relative directory url handling
- chore(page): add status code from error
- ci(bench): fix local server install
- chore(worker): fix background runtime start
- chore(website): fix invalid protocol repair default https
- perf(website): add direct page selectors
- chore(website): add fast hostname with_external_domains
- perf(chrome): add direct about:blank check
- chore(website): fix dequeue validation [#251]
- chore(website): fix dequeue budget tracking
- chore(utils): remove extra check get_path_from_url
- chore(website): fix depth validation
- chore(page): remove duplicate asset check links validation
- chore(website): fix initial q drain [#250]
- chore(utils): fix meta content css select
- chore(utils): add meta css extracting content
- chore(utils): fix meta parsing css extraction
- chore(ignore): add xml and yaml ignore
- chore(sitemap): add chrome sitemap auto-correct
- chore(cache_chrome_hybrid): fix compile flag [#245]
- chore(sitemap): add auto find sitemap url path
- chore(chrome): add to block list
- chore(tests): fix transformations build parse
- perf(cli): add page links direct return
- chore(page): remove duplicate abs
- feat(page): add optional page base passing
- chore(page): fix parsing url detection
- perf(page): remove parsing url directly
- chore(docs): fix spider cargo docs
- chore(page): fix build reqwest error_for_status cache flag
- chore(page): fix feat decentralize compile error status
- chore(page): add new_page_streaming error
- feat(page): add page_error_status_details [#244]
- chore(connect): fix compile missing libc
- chore(license): bump 2025
- feat(cargo): add basic feature flag
- chore(chrome): fix compile fs flag
- chore(page): fix crawling initial redirect
- chore(decentralized): fix page blocked_crawl
- chore(fs): remove tentril crate
- chore(disk): fix cfg aho_corasick
- feat(page): add blocked_crawl [#242]
- feat(website): add on_should_crawl_callback [#241]
- chore(fs): fix feature flag compile fs
- chore(statics): remove scoped concurrent access lazy_statics
- chore(disk): add optional SQLite init query
- perf(smart): add oneshot rewrite handling
- chore(page): prep outputsink
- feat(io_uring): add uring for connect_phase linux
- perf(runtime): add dedicated thread for request connect
- perf(page): add max limit streaming chunk size
- chore(website): add connect layer concurrency limit
- chore(page): add goaway retry
- perf(page): add dynamic rewriter chunk size
- chore(crate): remove htm5ever crate
- chore(docs): fix crate features disk and balance
- chore(test): fix crawl subscription
- chore(static): fix concurrency access lazy_static
- chore(smart): fix re-rendering wait_for
- chore(disk): fix encoding url SQLite
- feat(disk): add hybrid disk storing links
- chore(abs): fix handling protocol
- chore(chrome): add to block list
- chore(website): fix http crawl
- chore(chrome): add to block list
- chore(abs): add safety check joining
- chore(abs): fix abs handling domain
- chore(page): fix block asset handling
- chore(chrome): add to block list
- chore(chrome): ignore block_js on top pages
- chore(chrome): fix adblock compile
- chore(chrome): fix allowed critical rendering script interception
- chore(chrome): fix cdp serde support test
- chore(chrome): fix frame inline navigation
- chore(chrome): add to block list
- chore(chrome): add build generate blocking
- chore(chrome): add to block list
- chore(docs): fix budget feature flag [#236]
- chore(chrome): add to block list
- chore(chrome): add to block list
- chore(chrome): add to block list
- chore(chrome): add to block list
- chore(crate): bump ego-tree@0.10
- chore(chrome): add reusable remote connect client
- chore(chrome): add tokio-tungstenite
- chore(balance): add simple rebalance
- feat(page): add remote_addr feature flag [#226]
- chore(chrome): add to block list
- chore(chrome): add to block list
- chore(crawler): add to block list
- feat(balance): add global semaphore balancing
- chore(chrome): add ad and analytics block list
- chore(chrome): add ad and analytics block list
- chore(crawler): add blocklist items
- chore(crawler): add blocklist items
- chore(chrome): fix xhr interception
- chore(chrome): fix bytes timeout
- chore(chrome): add block list items
- chore(chrome): add block list items
- chore(chrome): add linkd custom blocker
- chore(chrome): revert adblock-rs default
- chore(chrome): add adblock items
- chore(chrome): default adblock engine
- chore(chrome): fix other and manifest ignore
- perf(chrome): add hydration ignore and custom website interception layer
- chore(smart): add meteor.js detection
- chore(chrome): add ads and tracking ignore list
- feat(chrome): add page bytes collecting
- perf(chrome): add xhr interception
- chore(crawler): add full analytics ignore
- chore(page): ignore js void
- chore(chrome): add analytics ignore scripts
- chore(transform): fix streaming repeated text
- feat(transform): add send text transform streaming
- feat(transform): add transform_content_send async streaming
- chore(interning): add optional string-interning
- chore(page): fix url encode handling mismatch
- chore(website): fix crawl establish domain removal [#233]
- chore(page): fix page link return full urls
- chore(clippy): fix lint
- chore(clippy): fix website display
- chore(website): add url log info
- chore(website): fix throttling request
- chore(page): add content-type encoding respect streaming rewriter
- chore(website): fix delay handling [#232]
- chore(examples): fix default entry
- chore(crawler): add default js enable intercept
- chore(website): fix limit budget breaking
- chore(crawler): fix backoff thread infinite loop
- chore(tracing): fix tracing ignore
- chore(backoff): add backoff max timeout
- chore(chrome): add join wait
- chore(chrome): add max wait for
- chore(website): add semaphore cancel release
- chore(chrome): add init state handling poll
- chore(abs): fix page paths infinite handling
- chore(tokio): remove parking_lot
- chore(chrome): revert page valid url parsing
- chore(chrome): add invalid network status code handling
- chore(redirect): add final url handling errors
- chore(logs): fix fetch error message
- chore(tracing): skip attempt navigation
- chore(chrome): fix browser close on initial erorr
- chore(abs): add ignore protocol list
- chore(abs): remove extra asset handling
- perf(abs): remove extra clone on abs handling
- chore(website): fix abs char boundary handling
- perf(website): add conditional url parsing budget check
- chore(website): fix protocol merging resources
- chore(website): fix budget immediate break
- chore(crawler): fix absolute url handling
- chore(website): fix shutdown break loop
- chore(tracing): add shutdown tracing
- chore(tracing): fix establish link call
- chore(tracing): add skip params
- chore(tracing): add fetch tracing
- chore(chrome): fix intercept passing ignore
- chore(examples): add chrome remote connection
- chore(tracing): add main entry trace
- chore(transform): fix text transform ignore elements
- chore(cache): fix build default cache backend
- chore(transform): fix text ignore handler
- chore(chrome): fix request interception auth enabling
- chore(chrome): fix connection loop hanging
- chore(tracing): add named tasks
- chore(website): remove extra chrome crawl method
- chore(chrome): fix semaphore drop on error
- chore(trace): add tracing tasks/sets
- chore(crate): add tokio tracing flag
- chore(transform): fix text transform ignore
- chore(page): fix full_resources [#230]
- chore(website): fix streaming hang
- chore(crawler): add rewriter text extracting
- chore(cache): fix http mem cache backend
- chore(semaphore): fix semaphore handling tasks
- chore(chrome): add proxy-authorization cdp errors
- chore(chrome): remove set_headers extra clone
- chore(chrome): add loopback only localhost
- chore(chrome): add localhost loopback remote connection
- perf(transform): add rewriter markdown handling
- chore(chrome): prep capturing anchor event listeners no hrefs
- chore(page): add safe char handling extensions
- perf(page): collect bytes without sink
- perf(page): add stream full resources
- perf(smart): add atomic re-render handling
- perf(page): add streaming chunks rewrite
- chore(chrome): add only success check handling
- chore(page): add box html bytes
- perf(page): pin html pre-parsing
- feat(config): add concurrency limits builder method
- perf(smart): add ahocorasick js handling search
- perf(smart): fix js need determination
- feat(page): add nextjs build ssg path handling
- chore(chrome): add page url final redirect handling
- perf(chrome): add skip other resources
- chore(website: add full resources only_html check
- chore(config): add only_html builder config
- chore(config): add only_html configuration
- chore(chrome): add reqeust timeout
- chore(chrome): add serde device type
- chore(chrome): add dismiss_dialogs config
- chore(chrome): add alert polyfill
- feat(real_browser): add randomize viewport
- chore(website): add missing with_wait_for_idle_dom builder
- feat(chrome): add web automation wait for dom
- chore(page): fix page absolute join
- chore(website): add increased timeout error handling
- chore(crawl): fix joinset blocking task
- chore(transformations): fix filter markdown/commonmark
- chore(openai): add json_structure handling js browser events
- chore(smart): fix establishing whitelist/blacklist
- chore(budget): fix whitelist/blacklist budgeting
- chore(transform): add dynamic rewriting html
- chore(transforms): add optional ignore tags
- chore(smart): add layui.js and jquery detection
- chore(fs,chrome): fix compile viewport handling
- chore(openai): fix response_format types
- chore(transformations): add filter_svg
- chore(encoding): fix auto encoding detection
- chore(page): add network error status code defaulting
- chore(website): add initial status code handling
- chore(smart): fix smart page links handling
- chore(page): fix link handling protocols
- chore(utils): fix try exact match
- chore(chrome): add automation all routes handling
- chore(crawler): fix tld and subdomain handling
- chore(chrome): fix window navigator stealth handling
- chore(transform): revert default output transformed encoding
- chore(encoding): add html lang auto parsing
- chore(transformations): add html file extensions list
- chore(transformations): add root selector across formats
- chore(page): add waf_check exposure
- chore(transformations): add filter svg markdown
- chore(links): fix default robots handling check
- chore(encoding): fix regex build auto_encoder
- chore(transformations): fix script encoding
- chore(transformations): add html_to_xml format
- chore(utils): fix serde serializing selectors
- chore(config): cleanup unused interception configs
- feat(utils): add xpath support
- chore(chrome): add bezier mouse movements
- chore(config): add serializable crawl configuration
- chore(page): add empty page validation
- chore(backoff): add gateway backoff retrying
- chore(status): add rate limit and server status tracking
- chore(page): add full handling status codes retrying
- chore(page): fix indexing convert abs
- chore(chrome): fix smart mode wait for page
- chore(page): fix retry handling
- chore(chrome): add smart mode wait for navigation
- chore(chrome): add return headers and cookies
- chore(encoding): fix smart mode loss encoding
- chore(page): fix url join empty slash
- chore(chrome): patch logs stealth mode
- chore(chrome): add waf mouse movement
- chore(chrome): add spoofed plugins
- chore(smart): add smart mode chrome retry
- chore(chrome): add extra http headers
- chore(chrome): fix request interception remote sockets
- feat(website): add retry and rate limiting handling
- chore(chrome): add cookie setting browser
- chore(chrome): fix concurrent websocket page connecting
- chore(chrome): fix proxy handling websockets
- chore(website): add semaphore negating
- release(minor): 2.6.19
- chore(website): add drain_links
- chore(crate): fix all feature flag compiling
- chore(page): add initial start domain tracking
- chore(crate): fix hickory typo
- chore(crate): default tls reqwest_native_tls_native_roots
- chore(config): add preserve host header configuration
- chore(links): add default type
- chore(spider): revert parent host change
- chore(page): fix link parsing redirected domains
- feat(transformations): add spider_transformation crate
- perf(cpu): add bucket backend string interning
- perf(mem): add buffer backend string interning
- perf(website): add string interning links visited
- chore(chrome): add public interception
- chore(smart_mode): revert set_content
- chore(docs): update intro title about
- chore(utils): add trie match all base path
- chore(examples): add css scraping example
- perf(chrome): add major screenshot performance custom command
- chore(chrome): fix chrome interception setup
- perf(chrome): add concurrent request interception
- chore(chrome): fix navigation setup sub pages
- chore(chrome): add chrome response headers
- chore(chrome): fix frame establishing
- chore(encoding): add html parse auto encoding
- chore(sitemap,chrome): fix compile intercept
- chore(chrome): remove intercept tasks on complete
- release(minor): 2.2.13
- Fix infinite loop (#213)
- chore(chrome): fix limit 1 handler hang
- chore(chrome): fix handler hang
- chore(chrome): add valid page check
- chore(chrome): fix screenshot timeout
- chore(chrome): add lazy chrome env connection url
- chore(deps): update auto_encoder@0.1.4 binary file check fix
- chore(deps): add chromiumoxide@0.7.0 and auto_encoder
- chore(page): revert page links mut
- perf(page): add pre-condition asset check parsing
- feat(encoding): add auto detect encoding [#207]
- chore(config): fix crawl budget manual set
- chore(config): fix budget reset on crawl end
- chore(chrome): fix remote connect dispose
- chore(chrome): fix remove connect channel send handler break
- feat(page): add return page links configuration
- chore(chrome): fix handler shutdown on ws unexpected
- chore(decentralized): fix smart mode compile
- perf(chrome): remove extra browser clone
- chore(chrome): fix chrome new page hang
- chore(scrape): add sitemap and smart [#206]
- chore(chrome): fix chrome handler break page hangs
- chore(cookies): fix missing cookies flag
- chore(crate): fix zstd wasm build
- chore(page): add http cookies map return
- chore(deps): re-export encoding_rs
- Update README.md (#203)
- chore(utils): release@0.2.0
- chore(scrape): fix visited_links storing
- chore(page): add get_chrome_page chrome page ref
- chore(v2): fix scrape API
- chore(minor): remove accidental typing parent_host
- chore(deps): update quick-xml@0.36.0
- chore(crate): add wasm entry target
- feat(chrome): add trie for execution_maps and web_automation_maps
- feat(chrome): add web automation scripts
- feat(adblock): add network adblocking
- chore(website): add website build error
- chore(fs,chrome): fix chrome fs storing [#198]
- chore(chrome): fix fingerprint and initial eval scripting
- perf(smart): add js assets phf set
- Fixing clap issues #195 (#196)
- chore(openai): fix max_tokens calc
- chore(chrome): add request allow channel cancelled send
- chore(page): add take parsed url
- chore(utils): add clean text element extracting content
- feat(spider_utils): add util crate for css scraping
- chore(regex): fix extra ref compact string
- chore(deps): update compact_str@0.8.0
- perf(cli): add crawl performant crawl streaming
- chore(cli): add sync flag crawl handling
- perf(cli): add subscription streaming scraping and storing
- perf(robots): add fast top pass robots check
- chore(chrome): fix wait_for_selector and wait_for_event
- feat(chrome): add exact page execution_scripts support
- chore(serde): add wait_for page_navigations defaults
- chore(chrome): add serde wait_for
- chore(robots): fix robots text path matching root
- chore(docs): add discord chat badge
- chore(bench): add debian wget install check [#191]
- chore(utils): add max bytes limiting env
- chore(robots): fix robot disallow respect
- chore(cli): add headless and smartmode install examples
- chore(website): fix sitemap regex compile
- chore(minor): fix license path
- chore(deps): update regex@1.10.5
- feat(whitelist): add whitelist only paths
- chore(budget): fix budget overflow
- feat(sem): add shared queue strategy
- chore(chrome): fix non chrome agents spoofing
- chore(website): fix type blacklist
- chore(page): add invalid url handling
- chore(cli): fix params builder optional handling
- chore(chrome): fix missing fingerprint initial establish
- perf(crawl): add limit detection crawling
- chore(chrome_intercept): fix intercept spawn handling
- perf(website): fix semaphore distribution
- chore(website): fix chrome stealth handling agent
- chore(website): add safe semaphore handling
- chore(website): fix crawl status persisting
- release: 1.95.27
- chore(robots): add max delay respect
- chore(website): fix crawl limit immediate exit
- chore(cli): fix limit respecting
- chore(crate): unpin smallvec
- chore(robots): fix respect robots [#184]
- chore(website): add dynamic sempahore multiplier
- chore(deps): bump chromiumoxide@0.6.0
- feat(reqwest): add zstd support
- chore(cli): fix serde_json pin
- chore(openai): fix gpt_config default serde deserialization
- chore(examples): fix subscribe stdout end sequence
- chore(page): add set_url page
- chore(openai): add tracking, ads, and meta default clean
- chore(openai): remove hidden content from request
- chore(openai): compress base prompt
- chore(chrome): remove unnecessary page activation
- chore(openai): add link and iframe to clean removal
- chore(openai): add default clean on resource
- chore(openai): fix smart mode passing target url
- chore(openai): revert prompt spaces
- feat(openai): add error tracking request
- chore(openai): fix spaces base prompt
- chore(html5ever): add comment endline tokenizer parse fix
- feat(real_browser): add http simulation headers
- feat(cache): add chrome caching between http
- chore(cache): add chrome hybrid caching
- feat(subscriptions): add semapre capacity matching
- chore(guard): add subscription auto drop cancel
- chore(openai): add fallback full context reduction
- feat(cache): add top level cache manager
- chore(chrome): fix blocked content waf detection
- chore(chrome): add interception handling waf detection
- chore(website): add missing smart mode init status handling
- chore(website): fix blocked status code reset
- chore(chrome): add proper status code detection
- feat(chrome): add dynamic chrome connections
- feat(spoof): add referrer spoofing
- feat(ua_generator): add verified real user-agents ua_generator@0.4.1
- chore(real_browser): add single call to handle iframe
- chore(chrome): fix intercept hang
- chore(chrome): fix domain host validation interception
- chore(real_browser): fix frame click action
- chore(real_browser): add duration timeout wait iframe
- chore(real_browser): add explicit form iframe checking protected bypass
- chore(docs): fix benchmarks target url
- chore(website): fix url getter function name
- chore(crate): unpin reqwest 0.12.4 - 0.12
- chore(examples): update default domain spider.cloud
- chore(docs): fix repo badges
- chore(openai): add messages trim_start
- chore(worker): fix headers flag compile
- chore(docs): update cloud hosted link - spider.cloud
- chore(chrome): fix proxy server headless connecting
- feat(openai): add screenshot js execution after effects
- feat(openai): add response caching
- docs: fix broken glob url link (#179)
- chore(openai): add detailed gpt results output
- chore(parser): update fast_html5ever@0.26.4 crate compile
- chore(docs): add subscribe_guard feat flag req note
- chore(docs): add examples cargo doc
- chore(openai): add custom api key config
- chore(docs): add subscribe example
- chore(docs): fix chrome missing link wait_for_delay
- chore(website): add url consistency
- chore(openai): fix prompt deserialization
- chore(configs): add prompt deserialization
- chore(configs): add option openai configs simple serde
- chore(openai): add path matching prompt determination
- chore(config): cleanup zero cost sub features
- chore(crate): remove hard pins on common crates
- Fix typo in README file (#176)
- chore(website): fix sitemap chrome build
- chore(crate): remove serde pin
- chore(openai): fix compile chrome
- feat(rss): add rss support
- feat(fingerprint): add fingerprint id configuration
- chore(page): return all page content regardless of status
- chore(openai): fix svg removal
- feat(openai): add extra data gpt curating
- chore(openai): add credits used response
- feat(openai): add multi driven prompts
- feat(real_browser): add real browser feature flag for chrome
- chore(crate): default enable http cookies
- chore(page): add cold fusion file detection
- chore(openai): add html_rewriting for max_tokens
- chore(links): allow extra links assigning all states
- chore(screenshots): fix screenshot format casing
- chore(examples): fix chrome_scheenshot_with_config intro
- chore(openai): add serde gpt configs
- chore(docs): update clarity on openai flag  env key
- chore(docs): add OpenAI section
- chore(openai): add explicit model checking
- chore(openai): add GPTConfigs new shortcut
- Openai/chrome driver (#174)
- chore(tokio): enable io-std feature
- chore(chrome): fix semaphore limiting scrape
- chore(docs): remove discount hosted
- chore(deps): update reqwest@0.11.26
- feat(chrome): add evaluate on new document script handling
- bench(deps): update criterion@0.5.1
- chore(guard): add sub guard full crawls
- chore(examples): fix queue gathering alt pages
- feat(q): add mid crawl queue
- chore(website): add set extra links extended crawls
- chore(website): add set_url re-usablity
- chore(smart): fix smart mode http defaults
- feat(chrome): add auth challenge response
- chore(sitemap): fix chrome sitemap handling
- perf(chrome): add direct page navigation
- bench(docs): fix page count
- feat(chrome): add chrome_headless_new flag
- chore(chrome): fix page concurrency
- chore(page): fix network wait_for_event
- chore(chrome): add pathbuf outdir
- chore(chrome): fix page chrome close
- chore(chrome): add conditional chrome close flag chrome_store_page
- chore(chrome): fix wait_for_idle_network break
- chore(chrome): fix intercept early exit
- chore(chrome): fix page hang
- chore(clippy): fix join_handle warnings
- feat(chrome): add wait_for configuration delay, selector, and idle network
- chore(chrome): add prep for custom wait_for_events
- chore(chrome): fix final_url page when wait_for_network_idle
- chore(chrome): fix page url determination
- bench(perf): add local dev server for testing (#167)
- chore(chrome): fix wait_for_network page extracting
- chore(chrome): fix bytes handling
- chore(website): remove extra conditional builder methods
- chore(website): add direct assign http client method
- chore(docs): add spider hosted coupon code
- chore(docs): fix warnings across feature flags
- chore(deps): update tokio@1.36.0
- release(minor): add response headers feature
- Felix engl/main (#164)
- chore(cli): add accept_invalid_certs arg
- chore(page): fix subdomain detection
- feat(config): add with_danger_accept_invalid_certs builder method
- chore(website): add url empty whitespace handling
- chore(cli): fix full resource include external domains
- feat(cli): add crawl page limit
- chore(page): fix external domains include
- chore(page): add shtml resource allow list
- perf(semaphore): fix dynamic sem limit scaling
- perf(runtime): remove yielding switching
- feat(chrome): add new waitforidlenetwork timeout control
- chore(sitemap): fix gathering all links
- feat(chrome): add wait_for_network_idle
- chore(lib): fix clippy suggestions
- chore(examples): add subscribe_download example
- chore(examples): fix download file path
- chore(sitemap): fix extracting multiple sitemap urls
- chore(website): fix un-controlled shutdown
- chore(website): fix http scrape persist links between crawls
- chore(website): re-use crawl handle check
- chore(docs): add subscribe_guard example
- chore(chrome): fix chrome initial scrape page set
- chore(chrome): fix chrome glob compile
- chore(scrape): fix store initial page
- perf(scrape): add shared arc
- chore(website): add error log scrape
- chore(sync): add subscription guard
- chore(chrome): add screenshot default compilation
- chore(sync): add unsubscribe channels
- chore(chrome): fix chrome screenshot storing
- chore(chrome): return bytes screenshot helper
- feat(chrome): add viewport configuration
- chore(docs): cleanup website & config
- feat(config): add full_resources manual set
- chore(config): fix cache compile chrome
- perf(page): fix inline hot link extracting
- chore(chrome): fix cache enabling/disabling
- test(cache): fix enable cache
- chore(config): fix limit configuration builder
- chore(config): add simple crawl limit builder method
- feat(chrome): add override locale emulation
- feat(chrome): add timezone override config
- chore(chrome): fix request interception execute self hosted scripts
- chore(examples): add glob example
- feat(config): add re-usable configuration building
- perf(cron): add async_job@0.1.4 lock fix
- chore(deps): update async_job@0.1.3 panic removal
- chore(sitemap): add persistable chain crawls
- feat(sitemap): add crawl_sitemap
- perf(encoding): add html size buffer allocation streaming
- feat(encoding): add dynamic streaming encoding html
- chore(configuration): add full builder confs
- chore(sitemap): fix scrape raw handling
- chore(website): add full builder defaults
- perf(budget): remove extra string allocation validation
- perf(robots): remove extra mem allocation link validation
- perf(control): add rwlock crawl control
- chore(stealth): add default method
- feat(redirects): add redirect limit expose with server respects
- chore(control): add crawl_id for explicit website control
- feat(depth): add crawl depth level control
- chore(chrome_intercept): fix intercept redirect initial domain
- feat(cache): add caching backend feat flag (#156)
- chore(chrome_intercept): fix host match blocking assets
- chore(redirects): fix subdomain checking domain
- feat(chrome_intercept): add chrome_intercept request interception flag
- feat(redirect): add transparent top redirect handling
- chore(decentralize): fix glob build
- perf(scrape): remove extra clone page broadcast
- feat(chrome_store_page): add feat flag chrome_store_page and screenshot helper
- chore(smart): fix nextjs static app detection
- perf(smart): add re-render html dispose
- chore(website): add smart mode method default
- perf(smart): prevent static app re-rendering
- Smart mode (#154)
- chore(website): add with_cookies builder method
- chore(website): add auto handle transport
- chore(bench): fix criterion table count
- chore(website): revert raw method flags
- chore(control): fix control task abort after crawl
- feat(chrome): add chrome_screenshot feature flag
- chore(website): add website.stop handling shutdown
- chore(crate): re-export async_job
- chore(website): add re-crawling clearing
- chore(cron): move cron feature to async_job crate
- chore(exports): add smallvec re-export
- chore(website): add chrome flag over raw HTTP methods
- chore(website): add enum string crawlstatus crontype
- chore(docs): fix fs feature default
- chore(docs): add node.js port docs
- chore(website): add crawl invalid state
- feat(client): add experimental http3 support
- chore(cron): add ref runner stop
- chore(feat): add public features
- chore(napi): add rust_tls flag
- feat(napi): add napi support
- chore(cron): remove default feature
- feat(cron): add cron feature flag (#153)
- feat(cookies): add cookie jar optional feature
- chore(website): remove public expose setup
- chore(chrome): fix raw page build
- chore(deps): bump tokio@1.34.0
- chore(ahash): bump ahash@0.8.6 AES ARM instructions fix
- chore(website): add crawl_concurrent_raw method (#152)
- chore(chrome): add main chromiumoxide crate
- release(minor): 1.49.5
- Return status code on error (#151)
- test(website): fix invalid configuration test
- chore(website): fix crawl re-run host determination
- chore(cli): add error output incorrect website url
- chore(cli): add http check cli website url
- chore(chrome): fix chrome headless headful args
- chore(docs): cleanup website
- chore(feat): add reqwest_json feature flag re-export use
- chore(scraper): fix text processing ignore script/style
- feat(page): add status code and error message page response (#148)
- chore(page): fix subdomain entry point handling root (#146)
- chore(docs): add missing sync default
- chore(config): add missing with_on_link_find_callback [#145]
- chore(chrome): add error log instance launch
- chore(docs): spider cloud hosted intro
- chore(chrome): update chromiumoxide_fork@0.5.12 safe ws connection handling
- chore(chrome): fix tungstenite remote headers ddos vulnerability
- feat(external): add wildcard handling all domains found
- chore(docs): remove early discount
- feat(cli): add external domains grouping
- feat(cli): add crawl budgeting
- chore(docs): fix spider hosted promo
- chore(docs): add hosted early launch oss discount
- chore(scraper): add element attribute extracting
- docs(fix): fix error in "scrape/gather html" section (#137)
- chore(deps): remove matches
- bench(node): disable bench invalid output
- chore(sync): fix build missing broadcast
- chore(docs): add respect_robots_txt delay note
- ci(bench): add full scopes
- chore(docs): add budget feature flag req
- chore(worker): fix scrape fetching content
- chore(redirect): add fast chrome redirect determination
- chore(chrome): fix build page missing mut
- chore(budget): allow budget public access
- perf(budget): use flag to prevent degradation
- feat(budget): add crawl budgeting pages
- feat(page): add page redirect destination exposure [#127]
- chore(worker): fix description of the crate
- chore(glob): fix link callback #136
- chore(docs): add hosted pay as you go example display
- chore(cli): fix links json output
- feat(external): add external domains grouping
- chore(benches): default proxy to spider worker
- chore(deps): update bytes@1.5.0
- chore(sitemap): fix sender drop
- feat(sitemap): add sitemap crawling feature flag
- chore(chrome): fix ip connecting
- chore(chrome): fix remote page closing
- chore(chrome): fix proxy remote interface connections
- chore(chrome): fix remote browser droping
- chore(chrome): fix remote debug ws find
- feat(chrome): add CHROME_URL env var remote connections
- chore(crawler): fix chrome_cpu flag launch args
- chore(chrome): add CHROME_BIN env variable handling
- feat(chrome): add stealth mode
- feat(chrome): add proxy networking
- feat(chrome): add chrome_cpu feature flag
- chore(chrome): fix content extracting after navigate
- chore(fetch): remove duplicate funcs
- chore(api): move chrome to features
- chore(crawl): remove crawl sync api
- chore(chrome): fix chrome browser shutdown
- feat(chrome): enable chrome rendering page content
- feat(website): add crawl status
- chore(docs): add missing feature flag info
- chore(exports): add tokio_stream re-export
- feat(page): add duration tracking
- chore(docs): add new hosted service link
- chore(website): add domain parsed getter
- chore(website): add domain name getter
- chore(page): fix trailing slash url getter
- chore(full_resources): fix capturing link tag
- chore(cli): fix resource extensions [130]
- chore(pages): fix full_resource gathering scripts
- feat(page): add byte storing resource (#131)
- chore(cli): add regex and full_resources flags
- chore(sync): fix initial link sub capture
- chore(sync): fix subscribe handling
- feat(sync): add broadcast channel subscriptions v1.36.0
- chore(utils): fix cfg control feature
- chore(lib): add full re-exports
- chore(scrape): add resource in on_link_find_callback
- chore(page): fix domain name clean handling
- perf(url): update percent-encoding@2.3.0
- perf(css): improve link tree parsing nodes (#126)
- chore(website): fix crawl establish link handling
- chore(website): fix cfg crawl establish
- chore(bench): fix env var massive bench
- feat(headers): add http header configuration
- perf(regex): use regex set one pass (#125)
- chore(deps): pin versions and update tokio@1.29.1
- chore(docs): add builder pattern configuration usage
- feat(builder): add builder pattern [#115]
- chore(spider): add .jsp file detection support
- chore(bench): add env crawl massive target url
- chore(feats): remove spoof ua_generator default
- chore(regex): fix compilation
- feat(cli): add static markup downloading
- chore(deps): update jsdom@0.0.11-alpha.1 SO fix
- chore(deps): update deps to latest
- chore(docs): update bench speed compare
- perf(workers): add increased semaphore limits (#123)
- chore(decentralized): revert semaphore
- chore(page): fix extra url parse selector gathering
- chore(page): remove double process conditional check
- chore(page): fix asset detection
- release(minor): add feature js script link extracting v1.31.0
- feat(js): add init of script parsing (#121)
- chore(page): fix crawl processing ignore assets
- chore(docs): fix crawl sequential docs
- chore(worker): fix cfg scrape build
- perf(worker): drop semaphore usage decentralized
- feat(worker): add tls support
- test(glob): fix http proxy test case
- chore(robots): fix robots parser link gathering
- chore(request): add custom domain redirect policy
- chore(glob): fix glob crawl establish
- chore(crawl): fix crawl asset detection and trailing start
- perf(strings): use case_insensitive_string as primary string (#117)
- chore(page): fix trailing slash links find
- chore(decentralized): fix transport type response return
- bench(decentralized): add decentralized worker split case (#116)
- release(minor): feat url glob and hybrid temp disk support
- chore(docs): add glob section
- feat(fs): add temp storage resource handling (#112)
- chore(glob): fix base subdomain tld crawls
- URL globbing  (#113)
- chore(network): fix status code handling valid resource
- chore(proxy): fix proxy interception and handling
- chore(socks5): fix proxy handling
- ci(proxy): add local socks5 server
- test(spider): fix crawl proxy case
- release(spider): 1.28.0 proxy support
- feat(proxy): add request proxy support
- perf(io): add bytes_stream body handling
- chore(docs): fix benchmark page count
- perf(sem): fix semaphore count (#109)
- chore(crawl): fix link domain handling
- chore(blacklist): fix option handling
- feat(config): add http2_prior_knowledge support for http2 request
- perf(regex): compile regex start of crawl
- chore(worker): fix selector handling
- chore(docs): fix feature headings
- chore(docs): fix website user_agent assign
- feat(flexbuffers): add flexbuffers serde decentralization (#105)
- perf(crawl): remove unused selectors building
- feat(full_resources): add flag for gathering all content related to domain
- feat(serde): add serde de/serialize case-insenstive strings
- chore(website): fix getting pages
- chore(worker): add string_concat url format
- perf(decentralized): add link gathering decentralization (#103)
- feat(control): move crawl control under feature flag
- chore(decentralized): fix proxy traffic handling
- feat(decentralize): add start of decentralized worker
- chore(html5ever): add create_comment stub
- chore(crawl): handle invalid domain panic
- chore(minor): build client without checking
- perf(crawl): remove extra yield on task join
- perf(async): add current handle runtime set
- chore(examples): fix htr react example
- perf(crawl): fix page link stream switching logic
- perf(scraper): add html5ever fork ignore comments + errors
- perf(task): fix semaphore task limiting (#102)
- perf(crawl): add single shared arc concurrent
- chore(page): add optional html
- bench(parse): add stream scraper bench marks
- chore(crawl): fix scheme mismatch crawling
- perf(crawl): reverse logic for extension handling
- bench(crawl): add massive crawl benchmark init
- chore(crawl): fix fragment handling of url ego walk
- chore(crawl): fix extension handling of domains
- chore(crawl): fix safe ignoring extra dot ext
- perf(parser): add fast concurrency setup (#100)
- chore(crawl): fix crawl reset active links
- perf(scraper): add fast element node creation
- perf(scraper): fix cache and hasher
- build(deps): add scraper to packages
- perf(crawl): reduce memory usage gathering links
- perf(crawl): reduce memory usage and make cb optional
- perf(css): update css3 selectors fast_scraper@0.14.3
- perf(crawl): reduce mem allocations css selectors and network client
- perf(ua): fix ua memory alloc for client
- chore(crawl): fix link callback handling on pre gather
- chore(robots): fix sitemap last write time
- perf(crawl): add join handle task management (#99)
- perf(scrape): add compact selectors
- perf(scraper): add ahash imp of scraper fork
- perf(crawl): add memory efficient strings (#98)
- feat(accuracy): improve crawl handling very large websites
- chore(crawl): add optional duration to disable timeout
- chore(docs): add reference examples instead of slow page gathering
- chore(docs): add missing scrape section
- build(tokio): remove unused net feature
- perf(client): add mass scan configs (#97)
- perf(crawl): add pre link gathering boost
- perf(crawl): remove mem usage on robot parser
- perf(crawl): add optional heap blacklist_url
- chore(crawl): remove double hashmap concurrent runs
- chore(crawl): add heap links_visited usage (#95)
- chore(crawl): remove active crawl link storing in self
- chore(crawl): fix loop break
- perf(tokio): enable parking_lot (#93)
- chore(crawl): add end breaking
- chore(minor): remove unnecessary await on handler config
- perf(inlining): fix cold path inline removal
- benches(cargo): fix install deps
- chore(hash): fix borrow handling links visited (#91)
- perf(inline): add inline opts (#90)
- perf(links): add ascii case-insensitive handling (#89)
- perf(parsing): add fast link serialization extracting
- chore(docs): fix crawl doc
- chore(scrape): add optional page setup
- chore(docs): fix robot parser example
- perf(crawl): remove static url handling robotparser
- perf(crawl): add top level subdomain tld base extracting
- chore(crawl): add case insensitive and trap handling
- Converted clippy warned single character strings to chars (#87)
- chore(crawl): add top throttle setup
- chore(crawl): fix link delay between
- perf(crawl): add top level selector building
- chore(bench): add hyperfine mac benchmarks
- chore(ignore): add img and xlsx to ignore
- Compiler Warnings (#82)
- chore(parser): add .mov file ignore

## What's Changed
* feat: add on_link_blocked_callback for robots.txt blocked URLs by @zanmato in https://github.com/spider-rs/spider/pull/394
* fix(website): crawl_raw uses raw sitemap chain on chrome builds by @zanmato in https://github.com/spider-rs/spider/pull/393
* Fix 500ms timeout for proxy error statuses 598/599 by @waamm in https://github.com/spider-rs/spider/pull/395
* • Fix Unbounded Task Spawning in Sitemap Crawl by @Uinelj in https://github.com/spider-rs/spider/pull/397
* chore(deps): bump actions/checkout from 4 to 7 by @dependabot[bot] in https://github.com/spider-rs/spider/pull/399
* browser: per-crawl control map for optional chrome enhancements by @j-mendez in https://github.com/spider-rs/spider/pull/400
* fix: concurrency/memory/panic audit hardening (v2.52.7) by @j-mendez in https://github.com/spider-rs/spider/pull/401

## New Contributors
* @zanmato made their first contribution in https://github.com/spider-rs/spider/pull/394
* @waamm made their first contribution in https://github.com/spider-rs/spider/pull/395
* @Uinelj made their first contribution in https://github.com/spider-rs/spider/pull/397
* @dependabot[bot] made their first contribution in https://github.com/spider-rs/spider/pull/399

**Full Changelog**: https://github.com/spider-rs/spider/compare/v2.48.4...v2.52.2