Starstruk Logo MarkStarstruk

v2.47.75

anthropics/anthropic-tokenizer-typescriptv2.47.75Mar 20, 2026by j-mendez

AI Summary

This release introduces extensible crawl pipelines with PageData and Crawler traits, adds proxy support for LLM HTTP requests, and enhances Chrome remote debugging capabilities with remote caching. Performance optimizations include SIMD-accelerated byte scanning and zero-alloc DNS cache hits.

Key Highlights

  • PageData & Crawler trait abstractions for extensible crawl pipelines
  • Proxy support for LLM HTTP requests
  • Chrome remote_addr via CDP `Network.responseReceived`
  • Remote cache for Chrome responses with dump & fallback support

New Features

  • SIMD-accelerated byte scanning (memchr)
  • Trie: `Box<str>` keys + manual byte-walk + memchr dot scan
  • Bloom filter bitmask addressing + inline early-exit
  • Zero-alloc DNS cache hits via `Arc<[SocketAddr]>`
  • Batch LRU cache eviction to reduce write-lock hold time
  • Skip robots.txt for single-page crawls

Full Release Notes

## What's New

- **PageData & Crawler trait abstractions** for extensible crawl pipelines
- **Proxy support for LLM HTTP requests** (#378)
- **Chrome remote_addr** via CDP `Network.responseReceived`
- **Remote cache for Chrome responses** — dump & fallback support

## Performance

- SIMD-accelerated byte scanning (memchr), unrolled FNV hash
- Trie: `Box<str>` keys + manual byte-walk + memchr dot scan
- Bloom filter bitmask addressing + inline early-exit
- Zero-alloc DNS cache hits via `Arc<[SocketAddr]>`
- Skip robots.txt for single-page crawls, TCP keepalive always
- Batch LRU cache eviction to reduce write-lock hold time
- `cache_mem` auto-enables `skip_browser` for Chrome mode
- Remove unnecessary Box allocations and redundant string conversions

## Fixes

- Fall back to HTTP crawl when Chrome is unavailable (#373)
- Skip redirect-caused `net::ERR_ABORTED` instead of aborting navigation
- Deterministic CDP event listener shutdown via watch channel
- Remote cache fallback + 3s timeout for chrome_remote_cache path
- Add decentralized stubs for `set_url_parsed_direct` and `links_full`

## Deps

- strum 0.28, phf 0.13, gemini-rust 1.7, chromey 2, llm_models_spider 0.1

**Full Changelog**: https://github.com/spider-rs/spider/compare/v2.47.52...v2.47.75