v2.44.16
spider-rs/spiderv2.44.16Feb 5, 2026by j-mendez
AI Summary
Adds JSON Schema support for structured extraction in `RemoteMultimodalEngine`, allowing users to define strict schemas for the data they want to extract from web pages.
Key Highlights
- New `ExtractionSchema` struct with name, description, and schema fields
- JSON Schema definition support for precise data extraction
- Strict mode enforcement to ensure data integrity
- Integration with `RemoteMultimodalConfigs` for easy setup
New Features
- Structured extraction using JSON Schema definitions
- Strict validation of extracted data
- Configurable schema descriptions
Full Release Notes
## What's New
### Relevance Gate for Remote Multimodal Crawling
Added a `relevance_gate` config that instructs the LLM to return a `"relevant": true|false` field in its JSON response. When a page is deemed irrelevant, its wildcard budget credit is refunded so the crawler discovers more relevant content.
**New config fields:**
- `relevance_gate: bool` — enables the feature
- `relevance_prompt: Option<String>` — optional custom relevance criteria
**How it works:**
1. When enabled, the system prompt instructs the LLM to include `"relevant": true|false`
2. If the model returns `false`, a budget credit is atomically accumulated
3. Credits are drained in the crawl loop to restore the wildcard budget
4. Default fallback is `true` (assume relevant) if the model omits the field
**Example:**
```rust
let cfgs = RemoteMultimodalConfigs::new(api_url, model)
.with_relevance_gate(Some("Only pages about Rust programming".into()));
```
### Full Changelog
- feat(agent): add `relevance_gate` and `relevance_prompt` to `RemoteMultimodalConfig`
- feat(agent): add atomic `relevance_credits` counter to `RemoteMultimodalConfigs`
- feat(agent): add `relevant: Option<bool>` to `AutomationResult` and `AutomationResults`
- feat(agent): extend system prompt and extraction with relevance gate instructions
- feat(spider): add `restore_wildcard_budget()` for budget refund
- feat(spider): drain relevance credits in crawl loop dequeue