Starstruk Logo MarkStarstruk

v3.18.0

openmeterio/openmeterv3.18.0Aug 4, 2026by apify-service-account

AI Summary

This release focuses on bug fixes for core crawling functionality, including request handling, persistence, cookies, and Cloudflare challenges, along with improvements to sitemap and Stagehand support.

Key Highlights

  • Fixes for `enqueueLinksByClickingElements` and duplicate persistence.
  • Prevention of re-submitting already-enqueued requests.
  • Updated Cloudflare challenge handling.

New Features

  • Support for any `clickOptions` in enqueue links
  • Fix for duplicate final crawler persistence
  • Request list re-submission prevention
  • Redirect handler for session cookie parity
  • Sitemap index support
  • Stagehand page registration wait
  • Cloudflare challenge markup update
  • Sitemap metadata leak prevention

Full Release Notes

# [3.18.0](https://github.com/apify/crawlee/compare/v3.17.0...v3.18.0) (2026-08-04)

### Bug Fixes

* allow any `clickOptions` in `enqueueLinksByClickingElements` ([#3823](https://github.com/apify/crawlee/issues/3823)) ([abcbfb1](https://github.com/apify/crawlee/commit/abcbfb158ab3edbb9e7e4c5c6f1d937dea9bb4dd))
* avoid duplicate final crawler persistence ([#3866](https://github.com/apify/crawlee/issues/3866)) ([79dd6c4](https://github.com/apify/crawlee/commit/79dd6c41eff144914ddb7fa6e04a65e9b8b762e0)), closes [#3167](https://github.com/apify/crawlee/issues/3167)
* **basic-crawler:** declare `@apify/datastructures` as a dependency ([#3815](https://github.com/apify/crawlee/issues/3815)) ([4e9d227](https://github.com/apify/crawlee/commit/4e9d227d7711f10662800d4079c4f958368bc063))
* **core:** avoid re-submitting already-enqueued requests in addRequestsBatched ([#3843](https://github.com/apify/crawlee/issues/3843)) ([704c269](https://github.com/apify/crawlee/commit/704c269806503d36ad2c629a63f3a4944109930c)), closes [#2456](https://github.com/apify/crawlee/issues/2456) [#2456](https://github.com/apify/crawlee/issues/2456) [#2456](https://github.com/apify/crawlee/issues/2456) [#3120](https://github.com/apify/crawlee/issues/3120)
* **core:** cap retries for unprocessed requests in addRequestsBatched ([#3765](https://github.com/apify/crawlee/issues/3765)) ([b3170a6](https://github.com/apify/crawlee/commit/b3170a60c5c0dbb53f4078555df7551cd0b77967)), closes [#3764](https://github.com/apify/crawlee/issues/3764)
* **core:** clear `AutoscaledPool.pause()` polling interval on timeout ([#3847](https://github.com/apify/crawlee/issues/3847)) ([42b8914](https://github.com/apify/crawlee/commit/42b89146a372b639eaaf50fbda9ff16b026b3344))
* **core:** ignore explicitly undefined baseUrl in enqueueLinks helpers ([#3924](https://github.com/apify/crawlee/issues/3924)) ([a58a9c1](https://github.com/apify/crawlee/commit/a58a9c1417c34c2a45a10f3855c4bee721a480ae))
* **core:** keep crawler enqueue limits when options carry explicit undefined ([#3927](https://github.com/apify/crawlee/issues/3927)) ([5f1b0ab](https://github.com/apify/crawlee/commit/5f1b0ab480f297100d2db70f646825b92f70f21a)), closes [#3924](https://github.com/apify/crawlee/issues/3924) [#3924](https://github.com/apify/crawlee/issues/3924)
* **core:** keep same-domain filtering after an off-domain redirect ([#3923](https://github.com/apify/crawlee/issues/3923)) ([9b312ef](https://github.com/apify/crawlee/commit/9b312efd14d607437f6a3865eea8bb67ba4cf05c)), closes [#3921](https://github.com/apify/crawlee/issues/3921)
* **core:** remove `RequestList` persist state listener on `teardown` ([#3846](https://github.com/apify/crawlee/issues/3846)) ([63394ee](https://github.com/apify/crawlee/commit/63394ee4bca950db1ee1678d3ea2b95ce45f95c5))
* **core:** skip malformed cookie fragments in mergeCookies ([#3822](https://github.com/apify/crawlee/issues/3822)) ([b3c3ef2](https://github.com/apify/crawlee/commit/b3c3ef2cab175ec0af4e5777bc6d0979b5e71d40))
* **core:** unblock backpressured sitemap load on persistState ([#3863](https://github.com/apify/crawlee/issues/3863)) ([0b31a8b](https://github.com/apify/crawlee/commit/0b31a8b57cdeee4cead83db6c48febf48b988f6f))
* declare missing dependencies ([#3817](https://github.com/apify/crawlee/issues/3817)) ([5a9a661](https://github.com/apify/crawlee/commit/5a9a661401a3aa895738c79a8310d9cc18e28826))
* filter sitemap-derived URLs by enqueue strategy ([#3797](https://github.com/apify/crawlee/issues/3797)) ([9dc3160](https://github.com/apify/crawlee/commit/9dc3160e6b83c549c29c188b7a88fdc8a612be62))
* **impit-client:** honor redirect handler for session cookie parity ([#3938](https://github.com/apify/crawlee/issues/3938)) ([9371301](https://github.com/apify/crawlee/commit/9371301b21f733a5b5f5c2a092512ec34f511843))
* make crawler terminal status message reliably delivered ([#3733](https://github.com/apify/crawlee/issues/3733)) ([7aed264](https://github.com/apify/crawlee/commit/7aed264f1cd9e993ffab93e314e9c7318b30d43d))
* **memory-storage:** don't increment `handledRequestCount` when updating a handled request ([#3827](https://github.com/apify/crawlee/issues/3827)) ([4739d48](https://github.com/apify/crawlee/commit/4739d4886e3c52f90814b9e5342c4ec2cbaa7587))
* **memory-storage:** prevent storage names from escaping the storage directory ([#3715](https://github.com/apify/crawlee/issues/3715)) ([a04c297](https://github.com/apify/crawlee/commit/a04c29766e3165534a5789d59a3a7bef5a8f5b5a))
* **memory-storage:** use platform basename in key-value store warning ([#3864](https://github.com/apify/crawlee/issues/3864)) ([dcefe01](https://github.com/apify/crawlee/commit/dcefe0196d0ba9bd89770e0a9ff5503295dc274b))
* **playwright,puppeteer:** only intercept requests for non-GET methods or payloads ([#3819](https://github.com/apify/crawlee/issues/3819)) ([1942f1e](https://github.com/apify/crawlee/commit/1942f1ec827cb69953a95abae2086400498b32d8))
* **playwright:** silence transient errors in `enqueueLinksByClickingElements` ([#3732](https://github.com/apify/crawlee/issues/3732)) ([7b98cc3](https://github.com/apify/crawlee/commit/7b98cc350f8ef7bd56bda309b5aff5664dd6fed5)), closes [#3216](https://github.com/apify/crawlee/issues/3216)
* read properties directly from `GotScrapingHttpClient` response ([#3807](https://github.com/apify/crawlee/issues/3807)) ([c673c8d](https://github.com/apify/crawlee/commit/c673c8d7771a43640e7714e929a95df0404ceed2))
* respect falsy transformRequestFunction return in context enqueueLinks ([#3925](https://github.com/apify/crawlee/issues/3925)) ([6cb4fc3](https://github.com/apify/crawlee/commit/6cb4fc392f43e7ad42d56114372eb8ad79aeff74)), closes [#3920](https://github.com/apify/crawlee/issues/3920)
* **sitemap:** recognize sitemap_index as a valid input sitemap url ([#3909](https://github.com/apify/crawlee/issues/3909)) ([eacab5e](https://github.com/apify/crawlee/commit/eacab5e155aeb5dca62cd0c03ef3553be65dd32b))
* **stagehand:** wait for Stagehand to register a page before using it ([#3865](https://github.com/apify/crawlee/issues/3865)) ([238214f](https://github.com/apify/crawlee/commit/238214f9e1586d63837c0b2c7b0a2cfc9a945ae4))
* update handleCloudflareChallenge for new Cloudflare challenge markup ([#3717](https://github.com/apify/crawlee/issues/3717)) ([5baa8bd](https://github.com/apify/crawlee/commit/5baa8bdd7baba927871ed20859405e14312297b6)), closes [#3629](https://github.com/apify/crawlee/issues/3629) [apify/camoufox-js#283](https://github.com/apify/camoufox-js/issues/283) [#3629](https://github.com/apify/crawlee/issues/3629)
* **utils:** advance the CPU tick baseline in systemInfoV2 getCurrentCpuTicks ([#3824](https://github.com/apify/crawlee/issues/3824)) ([bb23800](https://github.com/apify/crawlee/commit/bb238007e34009371807742ba11ecbb9974887b3))
* **utils:** allow apostrophes and asterisks in extracted URLs ([#3790](https://github.com/apify/crawlee/issues/3790)) ([e9baffd](https://github.com/apify/crawlee/commit/e9baffd78ec42079d5c9a8a6e62d43a594ae42c8)), closes [#2755](https://github.com/apify/crawlee/issues/2755)
* **utils:** prevent ReDoS in social email regex ([#3845](https://github.com/apify/crawlee/issues/3845)) ([d6a2b39](https://github.com/apify/crawlee/commit/d6a2b39252c0f00290e75a73def057fd39625525))
* **utils:** prevent sitemap metadata leak across url without loc ([#3795](https://github.com/apify/crawlee/issues/3795)) ([895ed62](https://github.com/apify/crawlee/commit/895ed6264d7eadc5306b92303aeb99bff41cb286))
* **utils:** use article:* selectors for article Open Graph properties ([#3789](https://github.com/apify/crawlee/issues/3789)) ([b660506](https://github.com/apify/crawlee/commit/b6605065fc4d7b692bcdac3ee2004042e8de7457))

### Features

* **core:** infer default-route handler userData from its schema ([#3919](https://github.com/apify/crawlee/issues/3919)) ([37d0ccb](https://github.com/apify/crawlee/commit/37d0ccbb2199bc05620bf4e359b0769bf6f1e05e))
* **impit-client:** add `cacheClients` option to disable `Impit` client reuse ([#3771](https://github.com/apify/crawlee/issues/3771)) ([af9cb8c](https://github.com/apify/crawlee/commit/af9cb8c68c8299e39fd2c40c3ff411e35d8a03f3)), closes [#3769](https://github.com/apify/crawlee/issues/3769) [#3769](https://github.com/apify/crawlee/issues/3769)
* opt-in schema validation of request userData per router label ([#3851](https://github.com/apify/crawlee/issues/3851)) ([c7347f9](https://github.com/apify/crawlee/commit/c7347f94f72980b7c424fb4fc7dc8f990f360d70)), closes [#3747](https://github.com/apify/crawlee/issues/3747) [#3748](https://github.com/apify/crawlee/issues/3748) [#3747](https://github.com/apify/crawlee/issues/3747) [#3082](https://github.com/apify/crawlee/issues/3082)
* support `puppeteer@25` ([#3869](https://github.com/apify/crawlee/issues/3869)) ([1430062](https://github.com/apify/crawlee/commit/1430062c2ba49c32ecb4cd4a14edd2d9a71c7ac3)), closes [#3772](https://github.com/apify/crawlee/issues/3772)
* type-safe router labels via per-label userData map ([#3747](https://github.com/apify/crawlee/issues/3747)) ([1124aca](https://github.com/apify/crawlee/commit/1124aca3c5eb527c097decd7263a6816c411369e)), closes [#3082](https://github.com/apify/crawlee/issues/3082)

### Performance Improvements

* **core:** improve `RequestList` initialization speed on large arrays ([#3754](https://github.com/apify/crawlee/issues/3754)) ([c451c13](https://github.com/apify/crawlee/commit/c451c13a9c9b5c884ce70e0d1257e15597fcb10b)), closes [#3750](https://github.com/apify/crawlee/issues/3750)