v2.44.16

spider-rs/spiderv2.44.16Feb 5, 2026by j-mendez

AI Summary

Adds JSON Schema support for structured extraction in `RemoteMultimodalEngine`, allowing users to define strict schemas for the data they want to extract from web pages.

Key Highlights

  • New `ExtractionSchema` struct with name, description, and schema fields
  • JSON Schema definition support for precise data extraction
  • Strict mode enforcement to ensure data integrity
  • Integration with `RemoteMultimodalConfigs` for easy setup

New Features

  • Structured extraction using JSON Schema definitions
  • Strict validation of extracted data
  • Configurable schema descriptions

Full Release Notes

## What's New

### Relevance Gate for Remote Multimodal Crawling

Added a `relevance_gate` config that instructs the LLM to return a `"relevant": true|false` field in its JSON response. When a page is deemed irrelevant, its wildcard budget credit is refunded so the crawler discovers more relevant content.

**New config fields:**
- `relevance_gate: bool` — enables the feature
- `relevance_prompt: Option<String>` — optional custom relevance criteria

**How it works:**
1. When enabled, the system prompt instructs the LLM to include `"relevant": true|false`
2. If the model returns `false`, a budget credit is atomically accumulated
3. Credits are drained in the crawl loop to restore the wildcard budget
4. Default fallback is `true` (assume relevant) if the model omits the field

**Example:**
```rust
let cfgs = RemoteMultimodalConfigs::new(api_url, model)
    .with_relevance_gate(Some("Only pages about Rust programming".into()));
```

### Full Changelog
- feat(agent): add `relevance_gate` and `relevance_prompt` to `RemoteMultimodalConfig`
- feat(agent): add atomic `relevance_credits` counter to `RemoteMultimodalConfigs`
- feat(agent): add `relevant: Option<bool>` to `AutomationResult` and `AutomationResults`
- feat(agent): extend system prompt and extraction with relevance gate instructions
- feat(spider): add `restore_wildcard_budget()` for budget refund
- feat(spider): drain relevance credits in crawl loop dequeue