v0.2.7

D4Vinci/Scraplingv0.2.7Nov 26, 2024by D4Vinci

AI Summary

Adds stealth-specific features like ad blocking and improves proxy support for basic requests to enhance scraping capabilities.

Key Highlights

  • Added `disable_ads` argument to `StealthyFetcher` (enabled by default)
  • Added `locale` argument to `PlayWrightFetcher`
  • Added proxy support in basic requests (`http://user:pass@host` format)
  • Improved stealth mode for `PlayWrightFetcher`

New Features

  • Fixed proxy logic for `PlayWrightFetcher` to be independent of stealth mode
  • Fixed `ResponseEncoding` to default to `utf-8` if detected encoding fails

Full Release Notes

# What's changed
### New features
- Now if you used the `wait_selector` argument with `StealthyFetcher` and `PlayWrightFetcher` classes, Scrapling will wait again for the JS to fully load and execute like normal. If you used the `network_idle` argument, Scrapling will wait for it again too after waiting for all of that. If the states are all fulfilled then no waiting happens, of course.
- Now you can enable and disable ads on `StealthyFetcher` with the `disable_ads` argument. This is enabled by default and it installs the `ublock origin` addon.
- Now you can set the locale used by `PlayWrightFetcher` with the `locale` argument. The default value is still `en-US`.
- Now the basic requests done through `Fetcher` can accept proxies in this format `http://username:password@localhost:8030`.
- The stealth mode improved a bit for `PlayWrightFetcher`.


### Bugs Squashed/Improvements
1. Now enabling proxies on the `PlayWrightFetcher` class is not tied to the `stealth` mode being on or off (Thanks to [@AbdullahY36](https://github.com/AbdullahY36) for pointing that out)
2. Now the `ResponseEncoding` tests if the encoding returned from the response can be used with the page or not. If the returned encoding triggered an error, Scrapling defaults to `utf-8`
---

> [!NOTE] 
> A friendly reminder that maintaining and improving `Scrapling` takes a lot of time and effort which I have been happily doing for months even though it's becoming harder. So, if you like `Scrapling` and want it to keep improving, you can help by supporting me through the [Sponsor button](https://github.com/sponsors/D4Vinci).