v2.10.27

spider-rs/spiderv2.10.27Oct 23, 2024by j-mendez

AI Summary

Fixes crawling bugs and adds support for specific JS frameworks and OpenAI configuration options.

Key Highlights

Fix protocol and TLD handling
Support for Layui UI and jQuery
Add OpenAI JSON schema configs

New Features

Add optional ignore tags
Add Layui UI support
Add OpenAI JSON schema option

Full Release Notes

# Whats Changed

* fix protocol handling valid links to crawl
* fix subdomains and tld handling matching
* add empty server response retry
* add initial request status code storing
* fix auto-encoding detection for html
* fix openai compile and fs compile
* add layui ui js frameworks and smartmode handling jquery
* chore(transforms): add optional ignore tags
* chore(budget): fix whitelist/blacklist budgeting
* chore(smart): fix whitelist/blacklist establish
* chore(openai): add json_schema option gpt configs

**Full Changelog**: https://github.com/spider-rs/spider/compare/v2.10.6...v2.10.27