py-1.22.0
pola-rs/polarspy-1.22.0Feb 8, 2025by github-actions[bot]
AI Summary
Major updates to the new streaming engine including multi/hive scans, Unity catalog support, and `merge_sorted` for binary types.
Key Highlights
- Multi/Hive scans in new streaming engine
- Unity catalog support (credentials, write_table)
- Implement `merge_sorted` for binary
New Features
- Multi/Hive scans in new streaming engine
- Add `linear_spaces`
- IO plugins suppport lazy schema
- Add `write_table()` function to Unity catalog client
- Add `is_object` method to Polars `DataType` class
- Implement `merge_sorted` for binary
- Add CredentialProviderAzure parameter to accept user-instantiated azure credential classes
- Expose unity catalog dataclasses and type aliases
Full Release Notes
## 🚀 Performance improvements
- Reduce sharing in stringview arrays in new-streaming equijoin (#21129)
- Implement native Expr.count() on new-streaming (#21126)
- Speed up list operations that use amortized\_iter() (#20964)
- Use Cow as output for rechunk and add rechunk\_mut (#21116)
- Reduce arrow slice mmap overhead (#21113)
- Reduce conversion cost in chunked string gather (#21112)
- Enable prefiltered by default for new streaming (#21109)
- Enable parquet column expressions for streaming (#21101)
- Deduplicate buffers again in stringview concat kernel (#21098)
- Add dedicated concatenate kernels (#21080)
- Rechunk only once during join probe gather (#21072)
- Micro-optimise internal `DataFrame` height and width checks (#21071)
- Speed up from\_pandas when converting frame with multi-index columns (#21063)
- Change default memory prefetch to MADV\_WILLNEED (#21056)
- Remove cast to boolean after comparison in optimizer (#21022)
- Split last rowgroup among all threads in new-streaming parquet reader (#21027)
- Recombine into larger morsels in new-streaming join (#21008)
- Improve `list.min` and `list.max` performance for logical types (#20972)
- Ensure count query select minimal columns (#20923)
## ✨ Enhancements
- Add projection pushdown to new streaming multiscan (#21139)
- Implement join on struct dtype (#21093)
- Use unique temporary directory path per user and restrict permissions (#21125)
- Enable ingest of objects supporting the PyCapsule interface via `from_arrow` (#21128)
- Enable new streaming multiscan for CSV (#21124)
- Environment `POLARS_MAX_CONCURRENT_SCANS` in multiscan for new streaming (#21127)
- Ensure AWS credential provider sources AWS\_PROFILE from environment after deserialization (#21121)
- Multi/Hive scans in new streaming engine (#21011)
- Add `linear_spaces` (#20941)
- IO plugins suppport lazy schema (#21079)
- Add `write_table()` function to Unity catalog client (#21089)
- Add `is_object` method to Polars `DataType` class (#21074)
- Implement `merge_sorted` for binary (#21045)
- Hold string cache in new streaming engine and fix row-encoding (#21039)
- Add CredentialProviderAzure parameter to accept user-instantiated azure credential classes (#21047)
- Expose unity catalog dataclasses and type aliases (#21046)
- Support max/min method for Time dtype (#19815)
- Implement a streaming merge sorted node (#20960)
- Automatically use temporary credentials API for scanning Unity catalog tables (#21020)
- Add negative slice support to new-streaming engine (#21001)
- Allow for more RG skipping by rewriting expr in planner (#20828)
- Rename catalog `schema` to `namespace` (#20993)
- Add functionality to create and delete catalogs, tables and schemas to Unity catalog client (#20956)
- Allow custom JSONEncoder for the `json_normalize` function, minor speedup (#20966)
- Support passing `aws_profile` in `storage_options` (#20965)
- Improved support for KeyboardInterrupts (#20961)
- Make the available `concat` alignment strategies more generic (#20644)
- Extract timezone info from python datetimes (#20822)
- Add hint for `POLARS_AUTO_USE_AZURE_STORAGE_ACCOUNT_KEY` to error message (#20942)
- Filter Parquet pages with `ParquetColumnExpr` (#20714)
- Expose descending and nulls last in window order-by (#20919)
## 🐞 Bug fixes
- Fix `Expr.over` applying scale incorrectly for Decimal types (#21140)
- Fix IO plugin predicate with failed serialization (#21136)
- Ensure `lit` handles datetimes with tzinfo that represents a fixed offset from UTC (#21003)
- Correctly implement take\_(opt\_)chunked\_unchecked for structs (#21134)
- Restore printing backtraces on panics (#21131)
- Use microseconds for Unity catalog datetime unit (#21122)
- Fix incorrect output height for SQL `SELECT COUNT(*) FROM` (#21108)
- Validate/coerce types for comparisons within join\_where predicates (#21049)
- Do not auto-init credential providers if credential fetch returns error (#21090)
- Fix `join_where` incorrectly dropping transformations on RHS of equality expressions (#21067)
- Quadratic allocations when loading nested Parquet column metadata (#21050)
- Invalidate sortedness flag when sorting from pl.Categorical to pl.Categorical("lexical") (#21044)
- Calling `top_k` on list type panics (#21043)
- Fix rolling on empty DataFrame panicking (#21042)
- Fix `set_tbl_width_chars` panicking with negative width (#20906)
- Ensure `write_excel` recognises the Array dtype and writes it out as a string (#20994)
- Fix `merge_sorted` producing incorrect results or panicking for some logical types (#21018)
- Fix all-null list aggregations returning Null dtype (#20992)
- Ensure scalar-only with\_columns are broadcasted on new-streaming (#20983)
- Improve SQL interface behaviour when `INTERVAL` is not a fixed duration (#20958)
- Address minor regression for one-column DataFrame passed to `is_in` expressions (#20948)
- Add Arrow Float16 conversion DataType (#20970)
- Revert length check of `patterns` in `str.extract_many()` (#20953)
- Add maintain order for flaky new-streaming test (#20954)
- Allow for respawning of new streaming sinks (#20934)
- Ensure Function name correctness in cse (#20929)
- Don't consume c\_stream as iterable (#20899)
- Validate `pl.Array` shape argument types (#20915)
- Fix `from_numpy` returning Null dtype for empty 1D numpy array (#20907)
- Consider the original dtypes when selecting columns in `write_excel` function (#20909)
- Handle boolean comparisons in Iceberg predicate pushdown (#18199)
- Fix `map_elements` panicking with Decimal type (#20905)
## 📖 Documentation
- Replace pandas `where` with `mask` in Migrating -> Coming from Pandas (#21085)
- Correct Arrow misconception (#21053)
- Add example showing use of `write_delta` with `delta_lake.WriterProperties` (#20746)
- Add missing `shape` param to `Array` docstring (#20747)
- Add IO plugins to Python API reference (#21028)
- Document IO plugins (#20982)
- Ensure `set_sorted` description references single-column behavior (#20709)
## 📦 Build system
- Speed up CI by running a few more tests in parallel (#21057)
## 🛠️ Other improvements
- Add test for equality filters in Parquet (#21114)
- Add various tests for open issues (#21075)
- Upgrade packages and apply latest formatting (#21086)
- Move python dsl and builder\_dsl code to dsl folder (#21077)
- Organize python related logics in polars-plan (#21070)
- Improve binary dispatch (#21061)
- Skip physical order test (#21060)
- Fix new ruff lints (#21040)
- Added test to check for the computation of list.len for null (#20938)
- Add make fix for running cargo clippy --fix (#21024)
- Add tests for resolved issues (#20999)
- Update code coverage workflow to use macos-latest runners (#20995)
- Remove unused arrow file (#20974)
- Deprecate the old streaming engine (#20949)
- Move `dt.replace` tests to dedicated file, add "typing :: typed" classifier, remove unused testing function (#20945)
- Extract merge sorted IR node (#20939)
- Update copyright year (#20764)
- Move Parquet deserialization to `BitmapBuilder` (#20896)
- Also publish polars-python (#20933)
- Remove verify\_dict\_indices\_slice from main (#20928)
- Add tests for already resolved issues (#20921)
- Fix the `verify_dict_indices` codegen (#20920)
- Add ProjectionContext in projection pushdown opt (#20918)
Thank you to all our contributors for making this release possible!
@FBruzzesi, @MarcoGorelli, @aberres, @alexander-beedie, @arnabanimesh, @bschoenmaeckers, @coastalwhite, @deanm0000, @dependabot[bot], @dimfeld, @eitsupi, @etiennebacher, @henryharbeck, @itamarst, @lmmx, @lukemanley, @mcrumiller, @mullimanko, @nameexhaustion, @orlp, @petrosbar, @ritchie46, @siddharth-vi, @skritsotalakis, @taureandyernv and [dependabot[bot]](https://github.com/apps/dependabot)