py-1.17.0

pola-rs/polarspy-1.17.0Dec 8, 2024by github-actions[bot]

AI Summary

Stabilizes new-streaming engine features like `maintain_order` joins and adds support for reading Enum dtype from CSV and Python objects.

Key Highlights

  • Stabilize maintain_order joins
  • Support reading Enum dtype from csv
  • Enable view arrow export in write_delta
  • Add lazy support for pl.select
  • Experimental cloud write support

New Features

  • Add `maintain_order` parameter to joins
  • Support reading Enum dtype from csv
  • Improve dtype inference and load for DataFrame cols constructed from Python Enum values
  • Add `drop_nans` method to DataFrame and LazyFrame
  • Allow for `to_datetime` / `strftime` to automatically parse dates with single-digit hour/minute/second
  • Add lazy support for `pl.select`
  • Enable view arrow export in `write_delta`

Full Release Notes

## 🚀 Performance improvements

- Add fast paths for series.arg\_sort and dataframe.sort (#19872)
- Much faster `Series` construction from subclasses of standard Python types (#20166)
- Utilize the RangedUniqueKernel for Enum/Categorical (#20150)
- Reduce memory copy when scanning from Python objects (#20142)
- Construct `Series` for bytes/binary data 10x faster when dtype not explicitly set (#20157)
- Don't instantiate validity mask when unneeded in Parquet (#20149)

## ✨ Enhancements

- Retry with reloaded credentials on cloud error (#20185)
- Support reading Enum dtype from csv (#20188)
- Improve dtype inference and load for `DataFrame` cols constructed from Python Enum values (#20180)
- Allow sorting of lists and arrays (#20169)
- Add `maintain_order` parameter to joins (#20026)
- Allow for `to_datetime` / `strftime` to automatically parse dates with single-digit hour/minute/second (#20144)
- Issue warning when using `to_struct()` without a list of field names (#20158)
- Experimental cloud write support (#20129)
- Add lazy support for `pl.select` (#20091)
- Enable view arrow export in `write_delta` (#20092)

## 🐞 Bug fixes

- Don't trigger length check in array construction (#20205)
- Allow row encoding for 32-bit architectures (e.g. WASM) (#20186)
- Properly project unordered column in parquet prefiltered (#20189)
- Csv stop simd cache if eol char is hit (#20199)
- Estimated size for object (#20191)
- Respect parallel argument in parquet (#20187)
- Only validate UTF-8 for selected items when all below len 128 (#20183)
- Serialize categories of Enum in arrow metadata (#20181)
- Don't use RLE encoding for Parquet Boolean (#20172)
- Invalid `bitwise_xor` for ScalarColumn (#20140)
- Series construct with large nested `u64` (#20167)
- Add temporal feature gate in `is_elementwise_top_level` (#20177)
- Column name mismatch or not found in Parquet scan with filter (#20178)
- Raise if apply returns different types (#20168)
- Deal with masked out list elements (#20161)
- Fix index out of bounds in uniform\_hist\_count (#20133)
- Implement `arg_sort` for Null series (#20135)
- Handle slice pushdown in PythonUDF GroupBy (#20132)
- Check shape for `*_horizontal` functions (#20130)
- Properly coerce types in lists (#20126)
- Incorrect aggregation of empty groups after slice (#20127)
- DataFrame `.get_column` after `drop_in_place` (#20120)
- Subtraction with underflow on empty FixedSizeBinaryArray (#20109)
- Materialize smallest dyn ints to use feature gate for i8/i16 (#20108)
- Return null instead of 0. for rolling\_std when window contains a single element and ddof=1 and there are nulls elsewhere in the Series (#20077)
- Only slice after sort when slice is smaller than frame length (#20084)
- Preserve Series name in \_\_rpow\_\_ operation (#20072)
- Allow nested `is_in()` in `when()/then()` for full-streaming (#20052)

## 📖 Documentation

- Add more Rust examples to User Guide (#20194)
- Expand plotting docs (#19719)
- Fix Rust examples in user guide (#20075)
- Update `by` param description for rolling\_\*\_by functions (#19715)
- Correct supported compression formats (#20085)
- Specify strictness in cast (#20067)

## 📦 Build system

- Upgrade `sqlparser-rs` from version `0.49` to `0.52` (#20110)
- Bump `memmap2` to version `0.9` (#20105)
- Bump `object_store` to version `0.11` (#20102)
- Bump `fs4` to version `0.12` (#20101)
- Bump `thiserror` to version `2` (#20097)
- Bump `atoi_simd` to version `0.16` (#20098)
- Bump `chrono-tz` to `0.10` (#20094)
- Update Rust dependency `ndarray` to `0.16` (#20093)
- Bump Rust toolchain to `nightly-2024-11-28` (#20064)

## 🛠️ Other improvements

- Deprecate ddof parameter for correlation coefficient (#20197)
- Move Bitwise aggregations to FunctionExpr (#20193)
- Add ragged lines test (#20182)
- Set delta version check higher (#20153)
- Fix typo in assertion in datatype copy test (#20121)
- Move horizontal methods to polars-ops (#20134)
- Remove useless SeriesTrait::get implementations (#20136)
- Add a bunch more automated row encoding sortedness tests (#20056)

Thank you to all our contributors for making this release possible!
@DzenanJupic, @MarcoGorelli, @YichiZhang0613, @alexander-beedie, @coastalwhite, @dependabot, @dependabot[bot], @flowlight0, @henryharbeck, @iharthi, @ion-elgreco, @jqnatividad, @lukapeschke, @lukemanley, @mcrumiller, @nameexhaustion, @ptiza, @ritchie46, @siddharth-vi, @stijnherfst, @stinodego and @wsyxbcl