Starstruk Logo MarkStarstruk

v2.11.20

spider-rs/spiderv2.11.20Oct 31, 2024by j-mendez

AI Summary

Major performance boost by processing crawling tasks concurrently, enabling better handling of Next.js SSG sites.

Key Highlights

  • Concurrent task processing for performance
  • Add Next.js SSG path handling
  • Add `wait_for_dom` for DOM updates
  • Add `with_block_assets` config

New Features

  • Add concurrent task processing
  • Add Next.js SSG path handling
  • Add `wait_for_dom`
  • Add `with_block_assets`
  • Add ignore assets config

Full Release Notes

# Whats Changed

Major performance improvement on crawling processing pending tasks concurrently. Now you can get all Next.js SSG pages on initial crawl for websites that do not expose links and have dynamic event listeners for routing.

* fix loop blocking tasks
* improve crawl performance processing tasks concurrent
* fix page absolute link joining
* add wait_for_dom to target element updates chrome
* add alert polyfill blocking prevention
* add missing chrome navigate request timeout for http future
* add ignore assets when crawling http
* add with_block_assets builder config for Server response non html
* perf(chrome): add skip other resources
* feat(page): add nextjs build ssg path handling

**Full Changelog**: https://github.com/spider-rs/spider/compare/v2.11.0...v2.11.20