py-1.13.0

pola-rs/polarspy-1.13.0Nov 12, 2024by github-actions[bot]

AI Summary

A major feature release introducing SQL support improvements, performance optimizations for cloud scans and joins, and new streaming engine capabilities.

Key Highlights

  • Improve `DataFrame.sort().limit/top_k` performance
  • Improve hive partition pruning with datetime predicates from SQL
  • Add SQL support for `RIGHT JOIN`
  • Add `show_graph` to display a GraphViz plot for expressions

New Features

  • Try to support native SAP HANA driver via `read_database`
  • Implement max/min methods for dtypes
  • Add SQL support for `RIGHT JOIN`
  • Streamline use of predicates connected by `&` with IEJoin
  • Support use of `is_between` range predicate with IEJoin

Full Release Notes

## 🚀 Performance improvements

- Improve `DataFrame.sort().limit/top_k` performance (#19731)
- Improve cloud scan performance (#19728)
- Fix quadratic 'with\_columns' behavior (#19701)
- Improve hive partition pruning with datetime predicates from SQL (#19680)
- Allow for arbitrary skips in Parquet Dictionary Decoding (#19649)
- Reorder conditions in is\_leap\_year (#19602)
- Rechunk in DataFrame.rows if needed (#19628)
- Dispatch Parquet Primitive PLAIN decoding to faster kernels when possible (#19611)
- Use faster iteration in 'starts\_with'/'ends\_with' (#19583)
- Branchless Parquet Prefiltering (#19190)
- Reduce size of IdxVec from 24 -> 16 bytes (#19550)

## ✨ Enhancements

- Try to support native SAP HANA driver via `read_database` (#19733)
- Implement max/min methods for dtypes (#19494)
- Improve `n_chunks` typing (#19727)
- Improve hive partition pruning with datetime predicates from SQL (#19680)
- Identify inefficient use of Python string `removeprefix`, `removesuffix`, and `zfill` in `map_elements` (#19672)
- Automatically use boto3 / google-auth if installed when scanning cloud (#19677)
- Identify inefficient use of Python string `replace` in `map_elements` (#19668)
- Parallel IPC sink for the new streaming engine (#19622)
- Add SQL support for `RIGHT JOIN`, fix an issue with wildcard aliasing (#19626)
- Add show\_graph to display a GraphViz plot for expressions (#19365)
- Streamline use of predicates connected by `&` with IEJoin (`join_where`) (#19552)
- Support use of `is_between` range predicate with IEJoin operations (`join_where`) (#19547)

## 🐞 Bug fixes

- Use `cls` for `to_python` (#19726)
- Fix validation for inner and left join when join\_nulls unflaged (#19698)
- SQL `ELSE` clause should be implicitly `NULL` when omitted (#19714)
- Improve `n_chunks` typing (#19727)
- Ensure `NoDataError` raised consistently between engines for Excel reads (#19712)
- In group\_by\_dynamic, period and every were getting applied in reverse order for the window upper boundary (#19706)
- Only allow `list.to_struct` to be elementwise when width is fixed (#19688)
- Make Array arithmetic ops fully elementwise (#19682)
- Address inconsistency with use of Python types in frame-level `cast` (#19657)
- Update line-splitting logic in batched CSV reader (#19508)
- Fix incorrect lazy schema for `explode()` in `agg()` (#19629)
- Fix fill null types (#19656)
- Fix filter incorrectly pushed past struct unnest when unnested column name matches upper column name (#19638)
- Fix typing for SchemaDefinition (#19647)
- Ensure `mean_horizontal` raises on non-numeric input (#19648)
- Reorder conditions in is\_leap\_year (#19602)
- Copy height in .vstack() for empty dataframes (#19641) (#19642)
- Correct wildcard and input expansion for some more functions (#19588)
- Allow `.struct.with_fields` inside `list.eval` (#19617)
- Sortedness was incorrectly being preserved in dt.offset\_by when offsetting by non-constant durations in the timezone-naive case (#19616)
- Fix incorrect `scan_parquet().with_row_index()` with non-zero slice or with streaming collect (#19609)
- Fix mask and validity confusion in Parquet String decoding (#19614)
- Parquet decoding of nested dictionary values (#19605)
- Do not attempt to load default credentials when `credential_provider` is given (#19589)
- Fix gather len in group-by state (#19586)
- Added input validation for `explode` operation in the array namespace (#19163)
- Improve error message (#19546)
- Fix predicate pushdown into inequality joins (#19582)
- Correct categorical namespace error message (#19558)
- Fix performance regression for sort/gather on list/array columns (#19564)
- Ignore quoted newlines when skipping lines in CSV (#19543)
- Incorrect gather for FixedSizeList with outer validity but no inner validities (#19489)
- Make Duration parsing fallible and not panic (#19490)

## 📖 Documentation

- Revise and rework user-guide/expressions (#19360)
- Update Excel page of user guide to refer to fastexcel as the default engine (#19691)
- Alter examples for round\_sig\_figs to make behaviour clearer (#19667)
- Assorted fixes to Rust API docs (#19664)
- Improve `replace` and `replace_all` docstring explanation of the "$" character with reference to capture groups (vs use as a literal) (#19529)
- Add credential provider section and examples to user guide (#19487)
- Fix various instances of repeated words in docs and comments (#19516)

## 📦 Build system

- Bump Rust toolchain to `nightly-2024-10-28` (#19492)

## 🛠️ Other improvements

- Remove unused Excel code (#19710)
- Use `Column` for the `{try,}_apply_columns{_par,}` functions on `DataFrame` (#19683)
- Remove more `@scalar-opt` (#19666)
- Move Series bitops to `std::ops::Bit...` (#19673)
- Mark test\_parquet.py test\_dict\_slices as slow (#19675)
- Get `Column` into `polars-expr` (#19660)
- Streamline internal SQL join condition processing (#19658)
- Factor out logic for re-use by new streaming CSV source (#19637)
- Configure grouped Dependabot updates (#19604)
- Fix PyO3 error in CI (#19545)
- Update nightly compiler version (#19590)
- Added input validation for `explode` operation in the array namespace (#19163)
- Fix lint (#19584)
- Add a `Column::Partitioned` variant (#19557)
- Move to fast-float2 (#19578)
- Only run remote bench on rust changes (#19581)
- Remove unsafe \*\_release functions (#19554)
- Fix `test_rolling_by_integer` not using parameterized dtype (#19555)
- Add `mindebug-dev` rust profile (#19524)
- Add CI step to process benchmark results (#19530)
- Add CI benchmark on merge (#19518)
- Skip client check with env var (#19517)
- Improve makefile build commands (#19498)

Thank you to all our contributors for making this release possible!
@3tilley, @HansBambel, @MarcoGorelli, @alexander-beedie, @barak1412, @braaannigan, @cmdlineluser, @coastalwhite, @corwinjoy, @dependabot, @dependabot[bot], @eitsupi, @janpipek, @jqnatividad, @letkemann, @max-muoto, @nameexhaustion, @orlp, @ritchie46, @rodrigogiraoserrao, @siddharth-vi, @stinodego and @wence-