Starstruk Logo MarkStarstruk

v2.22.19

spider-rs/spiderv2.22.19Dec 24, 2024by j-mendez

AI Summary

Introduces SQLite for improved memory handling with disk features, allowing for billions of resources. Also fixes concurrency issues.

Key Highlights

  • SQLite integration for improved memory management.
  • Hybrid memory/performance approach.
  • Fixed concurrent initial scoped access to `lazy_static!`.
  • Added connect layer concurrency limit.

New Features

  • SQLite integration
  • Connect layer concurrency limit

Full Release Notes

# Whats Changed

This release brings in a SQLite for improved memory handling with the feature flags `disk_native_tls`, `disk`, and `disk_aws`.
SQLite is set to be used in a hybrid manner with memory in order to maintain performance.

With disk handling and our string interning urls crawled can entire the billions of resources or infinite with EFS attached.

## Other Changes

* chore(website,page): fix concurrent initial scoped access to `lazy_static!`
* chore(chrome): add more network block layers for chrome
* chore(chrome): remove smart mode default idle_dom usage
* perf(page): dynamic rewriter chunk size
* chore(website): add connect layer concurrency limit apply
* perf(runtime): add dedicated thread for request connect

**Full Changelog**: https://github.com/spider-rs/spider/compare/v2.21.33...v2.22.19