v0.2.7
D4Vinci/Scraplingv0.2.7Nov 26, 2024by D4Vinci
AI Summary
Adds stealth-specific features like ad blocking and improves proxy support for basic requests to enhance scraping capabilities.
Key Highlights
- Added `disable_ads` argument to `StealthyFetcher` (enabled by default)
- Added `locale` argument to `PlayWrightFetcher`
- Added proxy support in basic requests (`http://user:pass@host` format)
- Improved stealth mode for `PlayWrightFetcher`
New Features
- Fixed proxy logic for `PlayWrightFetcher` to be independent of stealth mode
- Fixed `ResponseEncoding` to default to `utf-8` if detected encoding fails
Full Release Notes
# What's changed ### New features - Now if you used the `wait_selector` argument with `StealthyFetcher` and `PlayWrightFetcher` classes, Scrapling will wait again for the JS to fully load and execute like normal. If you used the `network_idle` argument, Scrapling will wait for it again too after waiting for all of that. If the states are all fulfilled then no waiting happens, of course. - Now you can enable and disable ads on `StealthyFetcher` with the `disable_ads` argument. This is enabled by default and it installs the `ublock origin` addon. - Now you can set the locale used by `PlayWrightFetcher` with the `locale` argument. The default value is still `en-US`. - Now the basic requests done through `Fetcher` can accept proxies in this format `http://username:password@localhost:8030`. - The stealth mode improved a bit for `PlayWrightFetcher`. ### Bugs Squashed/Improvements 1. Now enabling proxies on the `PlayWrightFetcher` class is not tied to the `stealth` mode being on or off (Thanks to [@AbdullahY36](https://github.com/AbdullahY36) for pointing that out) 2. Now the `ResponseEncoding` tests if the encoding returned from the response can be used with the page or not. If the returned encoding triggered an error, Scrapling defaults to `utf-8` --- > [!NOTE] > A friendly reminder that maintaining and improving `Scrapling` takes a lot of time and effort which I have been happily doing for months even though it's becoming harder. So, if you like `Scrapling` and want it to keep improving, you can help by supporting me through the [Sponsor button](https://github.com/sponsors/D4Vinci).