Starstruk Logo MarkStarstruk

v0.0.31

getmaxun/maxunv0.0.31Jan 4, 2026by amhsirak

AI Summary

Introduction of autonomous web discovery and search capabilities through two new robots: Crawl and Search.

Key Highlights

  • Crawl robot for intelligent website discovery via sitemap parsing and link following
  • Search robot for turning search queries into structured datasets
  • Granular scope control (Domain, Subdomain, Path-specific modes)
  • Regex pattern filtering for content inclusion/exclusion
  • Dual search modes: Discover (metadata) and Scrape (full content)

New Features

  • Crawl robot with intelligent discovery, depth control, and advanced filtering
  • Search robot with query-based web search and recency filters
  • Hybrid access via no-code dashboard and Developer SDK

Full Release Notes

v0.0.31 introduces two powerful new robots to the Maxun engine: **Crawl** and **Search**. These features allow you to automate data discovery at scale, whether you are mapping entire domains or researching the web via natural language.

## 🕸️Crawl

Stop manually feeding URLs. Crawl intelligently discovers and extracts entire websites.

* **Intelligent Discovery:** Uses both **Sitemap parsing** and **Link following** to find every relevant page.
* **Granular Scope Control:** Target exactly what you need with **Domain, Subdomain, or Path-specific** modes.
* **Advanced Filtering:** Use Regex patterns to include or exclude specific content (e.g., skip `/admin`, target `/blog/*`).
* **Depth Control:** Define how many levels deep the robot should navigate from your starting URL.


https://github.com/user-attachments/assets/d3e6a2ca-f395-4f86-9871-d287c094e00c



## 🔍 Search

Turn search engine queries into structured datasets.

* **Query Based:** Search the web with a search query - same as you would type in a search engine.
* **Dual Modes:** Use **Discover Mode** for fast metadata/URL harvesting, or **Scrape Mode** to automatically visit and extract full content from every search result.
* **Recency Filters:** Narrow down data by time (Day, Week, Month, Year) to find the freshest content.


https://github.com/user-attachments/assets/9133180c-3fbf-4ceb-be16-d83d7d742e1c



## Hybrid Access
Manage these robots through our no-code dashboard or trigger them programmatically via our Developer SDK.

## What's Changed
* fix: remove `onCancel` button from robot config page by @amhsirak in https://github.com/getmaxun/maxun/pull/929
* feat: cleaner alert notifs by @amhsirak in https://github.com/getmaxun/maxun/pull/927
* feat(ui): match theme of standard & outlined alerts by @amhsirak in https://github.com/getmaxun/maxun/pull/928
* feat: !show rows per page for runs & robots by @amhsirak in https://github.com/getmaxun/maxun/pull/930
* fix: show formats selected for scrape by @amhsirak in https://github.com/getmaxun/maxun/pull/931
* chore: pass prompt by @RohitR311 in https://github.com/getmaxun/maxun/pull/932
* feat: auto name lists for ai mode by @RohitR311 in https://github.com/getmaxun/maxun/pull/933
* feat: !wrap robots table w table head by @amhsirak in https://github.com/getmaxun/maxun/pull/934
* feat: display api key creation date by @amhsirak in https://github.com/getmaxun/maxun/pull/935
* feat: run scroll customization by @amhsirak in https://github.com/getmaxun/maxun/pull/936
* feat(core): real time progress update for runs by @amhsirak in https://github.com/getmaxun/maxun/pull/937
* feat: add crawl and search  by @RohitR311 in https://github.com/getmaxun/maxun/pull/938
* chore: pre-release v0.0.31 by @amhsirak in https://github.com/getmaxun/maxun/pull/940


**Full Changelog**: https://github.com/getmaxun/maxun/compare/v0.0.30...v0.0.31