Starstruk Logo MarkStarstruk

v1.6.0

opendatalab/MinerUv1.6.0May 4, 2026by dogancanbakir

AI Summary

This release adds automatic login support, per-domain page budgets, and adaptive rate limiting to the web crawler, alongside fixes for headless Chrome and resource cleanup.

Key Highlights

  • Automatic login support
  • Per-domain page budget to limit crawl scope
  • Per-host rate limiting with adaptive backoff
  • Page load strategy option (heuristic, load, domcontentloaded, networkidle)

New Features

  • Automatic login support
  • Per-domain page budget
  • Per-host rate limiting
  • Page load strategy configuration

Full Release Notes

## What's Changed

### 🎉 New Features
* Added automatic login support by @dogancanbakir in https://github.com/projectdiscovery/katana/pull/1620
* Added per-domain page budget to limit pages crawled per domain (0 = unlimited) by @dogancanbakir in https://github.com/projectdiscovery/katana/pull/1612
* Added per-host rate limiting with adaptive backoff by @dogancanbakir in https://github.com/projectdiscovery/katana/pull/1614
* Made jsluice cgo-free by @Mzack9999 in https://github.com/projectdiscovery/katana/pull/1570
* Added page load strategy by @fail-open in https://github.com/projectdiscovery/katana/pull/1514

`-pls, -page-load-strategy string  page load strategy (heuristic, load, domcontentloaded, networkidle, none) (default "heuristic")`

### 🐞 Bug Fixes
* Fixed cleanup of crawler resource lifecycles by @x-stp in https://github.com/projectdiscovery/katana/pull/1634
* Fixed headless Chrome profile persistence with `-no-incognito` and `-chrome-data-dir` by @iamkunal9 in https://github.com/projectdiscovery/katana/pull/1608
* Fixed gracefully handling DOM retrieval timeout in headless mode by @trasnake87 in https://github.com/projectdiscovery/katana/pull/1592
* Fixed ignored headless options since v1.4.0 by @dogancanbakir in https://github.com/projectdiscovery/katana/pull/1622
* Fixed onclick link discovery for dynamic JS navigations by @dogancanbakir in https://github.com/projectdiscovery/katana/pull/1624
* Fixed preserving response body/raw and waiting after form submission in headless mode by @XTeam-Wing in https://github.com/projectdiscovery/katana/pull/1585
* Fixed output of unvisited urls by @Mzack9999 in https://github.com/projectdiscovery/katana/pull/1595

### Other Changes
* Added `none` to filters by @Mzack9999 in https://github.com/projectdiscovery/katana/pull/1594
* Added helper functions + tests by @Mzack9999 in https://github.com/projectdiscovery/katana/pull/1596
* Added deduplication of headless mode output by @dogancanbakir in https://github.com/projectdiscovery/katana/pull/1602
* Updated Ubuntu installation instructions by @ecdeveloper in https://github.com/projectdiscovery/katana/pull/1628



## New Contributors
* @trasnake87 made their first contribution in https://github.com/projectdiscovery/katana/pull/1592
* @iamkunal9 made their first contribution in https://github.com/projectdiscovery/katana/pull/1608
* @ecdeveloper made their first contribution in https://github.com/projectdiscovery/katana/pull/1628
* @x-stp made their first contribution in https://github.com/projectdiscovery/katana/pull/1634
* @XTeam-Wing made their first contribution in https://github.com/projectdiscovery/katana/pull/1585

**Full Changelog**: https://github.com/projectdiscovery/katana/compare/v1.5.0...v1.6.0