v0.2

D4Vinci/Scraplingv0.2Nov 11, 2024by D4Vinci

AI Summary

A major version introducing Fetcher classes (Fetcher, StealthyFetcher, PlayWrightFetcher) and new bulk operation methods for selectors and text handling.

Key Highlights

  • Introduced three new Fetcher classes for different browser strategies.
  • Added `find_all`/`find` methods for element selection.
  • Added `filter` and `search` methods for bulk operations.
  • Added `TextHandlers` class for bulk text operations.

Breaking Changes

  • Renamed `css_selector` to `generate_css_selector`.
  • Renamed `xpath_selector` to `generate_xpath_selector`.
  • Restructured code into a `core` subpackage.

New Features

  • Fetcher class for basic HTTP requests.
  • StealthyFetcher using Camoufox.
  • PlayWrightFetcher with various modes (Vanilla, Stealth, Real Chrome, NSTBrowser).
  • New selector generation methods.
  • Restructured tests folder.

Full Release Notes

# What's changed
### New features
1. Introducing the `Fetchers` feature with 3 new main types to make Scrapling fetch pages for you with a LOT of options!
   - The `Fetcher` class for basic HTTP requests
   - The `StealthyFetcher` class is a completely stealthy fetcher that uses a [stealthy modified version of Firefox](https://github.com/daijro/camoufox).
   - The `PlayWrightFetcher` class that allows doing browser-based requests with Vanilla PlayWright, PlayWright with stealth mode made by me, Real browsers through CDP, and [NSTBrowser](https://app.nstbrowser.io/r/1vO5e5)'s [docker browserless](https://hub.docker.com/r/nstbrowser/browserless)!
2. Added the completely new `find_all`/`find` methods to find elements easily on the page with dark magic!
3. Added the methods `filter` and `search` to the `Adaptors` class for easier bulk operations on `Adaptor` object groups.
4. Added methods `css_first` and `xpath_first` methods for easier usage.
5. Added the new class type `TextHandlers` which is used for bulk operations on `TextHandler` objects like the `Adaptors` class.
6. Added `generate_full_css_selector` and `generate_full_xpath_selector` methods.

### Bugs Squashed
1. Now the `Adaptors` class version of `re_first` returns the first result that matches in all `Adaptor` objects inside instead of the faulty logic of returning the results of `re_first` of all `Adaptor` objects.
2. Now if the user selects a text-type content to be returned from selected elements (like css `::text` function) with any method like `.css` or `.xpath`. The `Adaptor` object will return the `TextHandlers` class instead of returning a list of strings like before. So now you can do `page.css('something::text').re_first(r'regex_pattern').json()` instead of `page.css('something::text')[0].re_first(r'regex_pattern').json()`
3. Now `Adaptor`/`Adaptors` re/re_first arguments are consistent with the `TextHandler` ones. So now you have `clean_match` and `case_sensitive` arguments. 
4. Now the `auto_match` argument is enabled by default in the initialization of `Adaptor` but still you have to enable it while selecting elements if you want to enable it. (Not a bug but a design decision)
5. A lot of type-annotations corrections here and there for better auto-completion experience while you are coding with Scrapling.

### Quality of life changes
1. Renamed both `css_selector` and `xpath_selector` methods to `generate_css_selector` and `generate_xpath_selector` for clarity and to not interrupt the auto-completion while coding.
2. Restructured most of the old code into a `core` subpackage and other design decisions for cleaner and easier maintenance in the future.
3. Restructured the tests folder into a cleaner structure and added tests for the new features. Also now tox environments are cached on GitHub for faster automated tests with each commit.