v2.45.20
spider-rs/spiderv2.45.20Feb 5, 2026by j-mendez
AI Summary
Introduces relevance gate feature for remote multimodal crawling that allows the LLM to determine page relevance and refund budget credits for irrelevant pages, enabling more efficient crawling of relevant content.
Key Highlights
- relevance_gate config to enable LLM relevance filtering
- relevance_prompt for custom relevance criteria
- Budget credit refund system for irrelevant pages
- Atomic relevance_credits counter
- relevant field in AutomationResult
New Features
- relevance_gate and relevance_prompt in RemoteMultimodalConfig
- Atomic relevance_credits counter in RemoteMultimodalConfigs
- relevant: Option<bool> in AutomationResult and AutomationResults
- System prompt extension with relevance gate instructions
- restore_wildcard_budget() for budget refund
- Drain relevance credits in crawl loop dequeue
Full Release Notes
## What's New
### Relevance Gate for Remote Multimodal Crawling
Added a `relevance_gate` config that instructs the LLM to return a `"relevant": true|false` field in its JSON response. When a page is deemed irrelevant, its wildcard budget credit is refunded so the crawler discovers more relevant content.
**New config fields:**
- `relevance_gate: bool` — enables the feature
- `relevance_prompt: Option<String>` — optional custom relevance criteria
**How it works:**
1. When enabled, the system prompt instructs the LLM to include `"relevant": true|false`
2. If the model returns `false`, a budget credit is atomically accumulated
3. Credits are drained in the crawl loop to restore the wildcard budget
4. Default fallback is `true` (assume relevant) if the model omits the field
**Example:**
```rust
let cfgs = RemoteMultimodalConfigs::new(api_url, model)
.with_relevance_gate(Some("Only pages about Rust programming".into()));
```
### Full Changelog
- feat(agent): add `relevance_gate` and `relevance_prompt` to `RemoteMultimodalConfig`
- feat(agent): add atomic `relevance_credits` counter to `RemoteMultimodalConfigs`
- feat(agent): add `relevant: Option<bool>` to `AutomationResult` and `AutomationResults`
- feat(agent): extend system prompt and extraction with relevance gate instructions
- feat(spider): add `restore_wildcard_budget()` for budget refund
- feat(spider): drain relevance credits in crawl loop dequeue