Starstruk Logo MarkStarstruk

v2.27.50

spider-rs/spiderv2.27.50Jan 25, 2025by j-mendez

AI Summary

Introduced web page normalization to prevent duplicate content and crawl traps, and enabled crawling of websites on ports other than the default 80 and 443.

Key Highlights

  • Added relative directory URL handling
  • Fixed relative page merging links
  • Added port validation
  • Updated dependencies

New Features

  • Page normalization
  • Port validation

Full Release Notes

# Whats Changed

Web page normalizing to prevent all duplicate content, crawl traps, and more pages from being crawled repeatedly.
We can now crawl websites that target ports outside 80 and 443.

1. feat(page): add relative directory url handling
1. chore(website): fix relative page merging links
1. chore(serde): fix cron compile configuration
1. chore(chrome): update tokio-tungestite@0.26
1. chore(page): add port validation links
1. chore(website): fix signature compile non disk feature flag
1. chore(rand): update rand@0.9
1. [chore(abs): clear query pairs [#257]

**Full Changelog**: https://github.com/spider-rs/spider/compare/v2.26.27...v2.27.50