py-2.0.0

## 💥 Breaking changes

- Run SQL window functions over grouped rows on the aggregated rows, and evaluate QUALIFY before the projection (#29746)
- Type exact SQL numeric literals as Decimal and truncate SQL `%`/DIV (#29536)
- Allow deterministic expression plugins to opt into CSE/CSPE (#29428)
- Read Parquet ENUM type as pl.String (#29331)
- More map operations (#29296)
- Deprecate cut/qcut (#29329)

## 🚀 Performance improvements

- Fix phases ending too soon due to join sampling (#29752)
- Group record batch fetches of remote IPC scans (#29750)
- Mark group-by node as memory intensive pipeline blocker (#29749)
- Don't inline slow OOC memory drift path (#29745)
- Only prefetch on unspill when there is memory headroom (#29701)
- Pushdown filters to scan\_lance (#29538)
- Do not materialize equal `ScalarColumn`s in `Column::append`/`extend` (#28989)
- Evaluate scalar windows in the streaming window node (#29698)
- Don't materialize scalar columns in `DataFrame::estimated_size` (#29711)
- Defer cached pread from prefetch to decode for Parquet (#29702)
- Decode IPC scans out of order when order is not observed (#29675)
- IPC metadata suffix fetch (#29692)
- Speed up null tracking in streaming group-by sum, min and max (#29691)
- Cache Iceberg manifest files across scans (#29623)
- Improve pushdown support for fallible Hive predicates (#29594)
- Streamline `BytecodeParser` method rewriting (#29632)
- Stream `from_dicts` records straight into column buffers (#29656)
- Read correlated SQL aggregates from the outer join when the subquery repeats it (#29685)
- Emit files unordered for remote IPC files when order not observed (#29678)
- Don't keep input order for order-insensitive windows in all engines (#29680)
- Fix parallelization in top-k reducer (#29684)
- Share one bloom filter between the build threads of a runtime filter (#29677)
- Add streaming (out-of-core) sort (#29657)
- Insert keys that start a run of equal keys into the hot group-by table right away (#29673)
- Prefetch key-row hash table lookups in blocks (#29663)
- Lower count-guarded sums to a sum that is null on empty input (#29668)
- Use single PUT for small cloud uploads (#29603)
- Don't cache plain scans in the streaming engine (#29654)
- Change single-key group-by hash table layout and add prefetching (#29661)
- Use the integer ranges of sampled parquet row groups in filter estimates (#29655)
- Raise HTTP rate-limiter read init default (#29584)
- Change join hash table layout and add prefetching (#29653)
- Add a fast-path for `as_list` with only a single input (#29592)
- Use one runtime filter builder per thread for a finished sample (#29646)
- Compute decimal addition, subtraction and multiplication in i64 when values fit (#29638)
- Spread decimal sums and means over lanes and check the sum's 38 digits once (#29637)
- Compute decimal addition, subtraction and multiplication without per-row validity (#29634)
- Skip or reuse UDF disassembly in `BytecodeParser` (#29625)
- Rescale a single decimal operand once in addition and subtraction (#29631)
- Dynamic hot table size in streaming GroupBy (#29629)
- Lower gathering from literal as elementwise (#29605)
- Execute fixed-size rolling windows on streaming engine (#29602)
- Use the finite rule of equi joins to determine build side of semi/anti (#29617)
- Drop the byte limit on runtime filter build sides (#29618)
- Add eviction-free path to GroupedReduction, with lanes (#29613)
- Improve window functions (#29591)
- Push down `is_nan`/`is_not_nan` filters to Iceberg (#29579)
- Compute shared group-by agg input subexpressions once (#29574)
- Emit files unordered for remote Parquet files when order not observed (#29581)
- Filter rows by the runtime join key range (#29576)
- Update few-group sums, means and counts in lanes (#29573)
- Loop over 4k arrays in group-by update (#29572)
- Decode parquet scans out of order when order is not observed (#29546)
- Don't always row-encode multi-column group-by (#29570)
- Push down `str.starts_with` filters to Iceberg (#29545)
- Trace join runtime filters through unions (#29565)
- Fold SQL temporal literals into plain values (#29562)
- Don't rehash stored key on every TotalIndexMap probe (#29561)
- Fuse drops into all filters on streaming engine (#29550)
- Join probe on Arrays directly (#29558)
- Better build side selection in equi-join sampler (#29524)
- Large optimisation for `BytecodeParser` rewrite/dispatch (#29548)
- Parallelize expressions in in-memory map fallback (#29535)
- Push down `!=` filters to Iceberg (#29455)
- Improve predicate stats (#29495)
- Scale bloom filters from keys seen (#29477)
- Optimize simple like queries (#29459)
- Insert head slice below select(len() \<cmp> n) to enable slice pushdown (#29209)
- Parallelize row encoding in the row\_encode expression (#29434)
- Degrade outer join optimization (#29432)
- Optionally reorder semi join (#29443)
- Borrow the cached regex instead of cloning it per page (#29422)
- Lower SQL LIKE '%x%' to a literal substring match (#29431)
- Optimize `is_in` expression (#29425)
- Pushdown bloom filters in probe side (#29423)
- Narrow the decimal rescale to 64 bits when the value fits (#29396)
- Use a thread-local copy of a pushed-down regex predicate (#29411)
- Sample anti-join build side (#29416)
- Improve performance importing from arrow (#29412)
- Sample table size to determine build side in semi joins (#29375)
- Optimize row-encoding (#29405)
- Order pushed parquet predicate columns by measured selectivity (#29397)
- Coerce float literals to decimal instead of casting the column (#29395)
- Fix OOM on TPCH SQL and fix fuzzing errors (#29389)
- Restrict correlated SQL aggregates to requested keys (#29383)
- Prepare Parquet scans for row-group splitting in Polars Cloud (#29295)
- Skip row groups by join runtime ranges without a statistics frame (#29370)
- Reduce copy in `scan_lines` (#29310)
- Increase HTTP read rate-limit default (#29363)
- Evaluate a pushed parquet predicate sequentially per conjuct (#29352)
- Disable system certificates for `CloudScheme::Http` sources (#29284)
- Reduce rechunk in `sort_in_place` (#29343)
- Dynamic predicates for hash joins (#29312)
- Use HTTP suffix range for Parquet size and footer (#29308)
- Derive predicates from join conditions (#29304)
- Lower uncorrelated subqueries to semi joins and push semi/anti joins below inner joins (#29289)
- Make leaf name iterator unique (#29291)
- Don't clone the full frame per arm in when/then/otherwise (#29258)
- Push inner joins before outer joins and rewrite left-join-is-null to anti join (#29277)
- Use stats to decide cross join buffering side (#29270)
- Improve cache-removal and join-order cost estimates (#29263)
- Improve CSPE cost evaluation  (#29250)
- Fuse group-by pre-select into node after partition (#29251)
- Inline hot small functions (#29244)
- Fix plan-time regressions in projection pushdown for wide frames (#28724)
- Rechunk before selecting in group-by pre-select (#29219)
- Reuse Iceberg data file sizes (#29063)

## ✨ Enhancements

- Python Polars 2.0 (#29723)
- Enable OOC by default with 80% of available RAM as threshold (#29741)
- Support window frames in SQL FIRST\_VALUE and LAST\_VALUE, and add NTH\_VALUE and a LAG/LEAD default (#29740)
- Set default OOC disk budget to 64 GB (#29734)
- Add POLARS\_OOMKILL\_THRESHOLD\_MB (#29735)
- Improve estimated DataFrame memory usage (#29712)
- Extend pipe\_with\_dtype for multiple expressions (#29676)
- Attribute physical nodes to IR nodes (#29522)
- Make approx\_quantile sketch states mergeable across processes (#29660)
- Add unstable scan\_lance (#29413)
- New Expression and function: pipe\_with\_dtype (#29547)
- Resolve `is_in` and Map lookup coercion from dtypes alone (#29487)
- Cast the needle of `is_in` and Map lookups exactly or not at all (#29486)
- Support `Map` in JSON and NDJSON reading and writing (#29478)
- Add `scan_external_reader` (#29232)
- Add `struct.eval` (#29454)
- Improve `BytecodeParser` UDF variable resolution (#29553)
- Type exact SQL numeric literals as Decimal and truncate SQL `%`/DIV (#29536)
- Make Iceberg sink commits idempotent across re-executions (#29512)
- Allow deterministic expression plugins to opt into CSE/CSPE (#29428)
- Improve cloud object\_store IO error types and messages (#29471)
- Add erf(c) function (#29501)
- Update `BytecodeParser` for Python 3.15 (#29491)
- Expose the registered source as `scan_fn.io_source` (#28897)
- Support selectors in `join` keys (#29233)
- Honor Iceberg sort orders in native sinks (#29318)
- Add `APPROX_QUANTILE` to the SQL frontend (#29288)
- Add support for `approx_quantile` in the streaming engine (#29237)
- Read Parquet ENUM type as pl.String (#29331)
- Expose more `scan_iceberg/delta`-related attributes in the visitor for cudf\_polars (#29297)
- More map operations (#29296)
- Binning functions (#28888)
- Support `collect` and `collect_batches` using `RemoteEngine` (#28914)
- Support GROUP BY GROUPING SETS, ROLLUP, CUBE and GROUPING() (#29278)
- Fix tpch SQL issues  (#29269)
- Fuse filters in (inner) join operation (#29218)
- Add in-memory support for approximate quantile (#29206)
- Disable casts from `String` to `Time` (#29215)

## 🐞 Bug fixes

- Find SQL aggregates by their function name, and raise on nested aggregate calls (#29751)
- Run SQL window functions over grouped rows on the aggregated rows, and evaluate QUALIFY before the projection (#29746)
- Match categories added after `is_in` prepares a string haystack (#29744)
- Give tied rows the same running aggregate in SQL windows, and support more window frames (#29736)
- Fix scan\_lance CI error (#29737)
- Rank NULLs and ties in SQL window functions, and add PERCENT\_RANK, CUME\_DIST and NTILE (#29729)
- Read from database without requiring SQLAlchemy `asyncio` extra (#29713)
- Don't slice the input of a select that has a window expression (#29725)
- Fix named SQL windows and raise for window shapes that gave wrong results (#29720)
- Preserve projections across nested caches (#29710)
- Allow the same equi-join key pair twice in a join condition (#29715)
- Defer fallible Hive pruning until after partition rewriting (#29700)
- Make dynamic window bounds data-independent (#29706)
- Panic on empty windows in dynamic group-by (#29704)
- Only reserve builder room for the rows a sort bucket receives (#29696)
- Fix handling of pruned input nodes during graph traversal (#29532)
- Minor fixes for `DataFrame` "orient" behaviour (#29679)
- Fix `list.contains` on multi-chunk sliced lists (#29635)
- Make `is_sorted` order nested types as `sort` does (#29683)
- Fix compile errors after attributing physical nodes to IR nodes (#29687)
- Raise on ragged rows instead of silently dropping values (#29658)
- Exclude IPC dictionary batches from record batch stats row count (#29672)
- Handle reordered groups in sort\_by (#29639)
- Don't recompile regex (#29597)
- Fix null and mixed-dtype edge cases in `is_in` (#29593)
- Errors in str.to\_lowercase and str.to\_titlecase (#29607)
- Make str.zfill consistent with pad for unicode (#29604)
- Empty group-by argmin/max on scalar (#29608)
- Support nested by in grouped `min_by`/`max_by` in streaming engine (#29628)
- Guard `is_nan`/`is_not_nan` pushdown against Decimal columns in Iceberg (#29610)
- Fix read\_excel from\_arrow error (#29492)
- Improve default S3 endpoint resolution (#29470)
- Don't let a SQL derived table alias replace a registered table (#29577)
- Write a single Avro header when a DataFrame has multiple chunks (#29569)
- Fix bug in `group_by` predicate pushdown (#29499)
- Preserve literal state in list/arr eval under group\_by (#29557)
- Don't evict when updating an existing `LRUCache` key (#29551)
- Use SQL decimal result scales for `*` and `/` (#29520)
- Compute mixed-scale Decimal operations without a lossy common cast (#29519)
- Keep Decimal results within their declared precision and round Decimal to float casts correctly (#29539)
- Suggest `str.contains` for string containment in `map_elements` UDFs (#25472)
- Keep a landed Iceberg snapshot's manifests when a commit response is lost (#29511)
- Allow deterministic expression plugins to opt into CSE/CSPE (#29428)
- Keep requested length in streaming negative slice (#29399)
- Match engine output dtype in planner for `Decimal` and integer arithmetic (#29515)
- Push down `!=` filters to Iceberg (#29455)
- Reject non-numeric input and preserve `Decimal` scale in `entropy` (#29228)
- Honor env var in Parquet/IPC pipeline budget (#29518)
- Stop casting data to a Decimal needle in `is_in`, allow `Array(Null)` casts (#29485)
- Clamp inflight budget from HTTP rate-limiter (#29497)
- Don't propagate RUSTFLAGS into dependencies in coverage (#29510)
- Read legacy Parquet LIST structures according to the spec (#29469)
- Address Python 3.15 compatibility issue with `_is_empty_method` (#29489)
- Insert head slice below select(len() \<cmp> n) to enable slice pushdown (#29209)
- Fix loading negative decimals from iceberg statistics (#29452)
- Support `NaN` comparison pushdown to Iceberg (#29453)
- Stop parse\_version from splicing a pre-release suffix into the number (#29401)
- Produce correct parquet statistics for enum columns (#29364)
- Don't restart predicate pushdown if the predicate was not pushable (#29418)
- Add regression test for Decimal/integer arithmetic schema (#29120)
- Dynamic boundary schema mismatch (#29448)
- Avoid panic on Enum and Categorical literals in Parquet predicates (#29421)
- Fix SQL count star on CSV performance regression (#29427)
- Respect include\_bounaries on in-mem empty dynamic (#29440)
- Fix over ordering for multiple cols (#29417)
- Enable dtype-i128 alongside dtype-decimal in polars-stream (#29410)
- Prevent bias in `'req_double'` at the median by actually merging the sketches (#29326)
- Fix OOM on TPCH SQL and fix fuzzing errors (#29389)
- Size row-index table statistics from the statistics frame (#29381)
- Preserve categories object in pyo3-polars (#29385)
- Strip the leading slash from Windows Delta table roots (#29386)
- Isolate expanded Python dataset scans (#29378)
- Make Iceberg bucket sort keys serializable (#29376)
- Apply "schema\_overrides" in `read_database` for Arrow-based drivers (#29273)
- Remove duplicated word in rate-limit comment (#29372)
- Check whether Datetime is monotonically increasing dynamically (#29293)
- Keep input order of unmatched build rows in ordered streaming equi join (#29371)
- Unaliased constants in SQL SELECT with GROUP BY (#29367)
- Consistent `Date` and `Decimal` means between the streaming and in-memory engines (#29359)
- Respect string statistics for enum columns during parquet scanning (#29366)
- Support selectors properly in DataFrame n\_unique (#29360)
- Decimal Parquet statistics for decimal/i128/f16 (#29350)
- Early check for converting integer map keys (#29348)
- Lowering for input-independent filter (#29340)
- Correlation of constant column returning non-NaN for larger inputs (#29319)
- Incorrect height in multi-input GroupBy (#29332)
- Fix panic in scan\_iceberg for snapshot\_id before a schema change (#28895)
- Resolve arithmetic `Struct` supertypes per-field (#29261)
- Fix comparison expression method comment (#29257)
- Don't panic on an empty or null quantile expression input (#29240)
- Detect list-valued quantile literals in approx\_quantile `auto` (#29236)
- Various map issues (#29147)
- Rolling quantile should respect window trimming (#29200)
- Ensure ignored columns are excluded from dtype `Wildcard` selector (#29220)

## 📖 Documentation

- Add engines and other missing items to API reference (#29493)
- Update migration guide with rc2 changes (#29415)
- Clarify that ambiguous parameter refers to DST transitions (#28873)
- Document gzip/zstd compression for read\_csv and scan\_csv (#29387)
- Add LazyFrameResolver to reference guide (#29379)
- Add user-guide for new enable\_monitoring feature (#29358)
- Improve join\_where engine tag (#29234)

## 🛠️ Other improvements

- Hold the category mapping in the categorical `is_in` lookup (#29747)
- Derive serde for moment states (#29742)
- Give each crate its own temporary column name counter (#29689)
- Apply `A-io-partitioning` label for Hive-related issues and PRs (#29659)
- Use explicit match arms in `LogSeries` operations \& improve error messages (#29575)
- Improve reliability of a slightly flaky test (#29587)
- Split `has_joins_or_unions` into `has_joins` and `has_unions` (#29338)
- Remove scheduled cache cleaning (#29517)
- Add window placement to group by rolling (#29529)
- Add window placement to group by dynamic (#29527)
- Fix flaky test (#29530)
- Improve cloud object\_store IO error types and messages (#29471)
- Fix iceberg-related test failures (#29533)
- Add IR structs for temporal group-by options (#29526)
- Workflow tweaks (#29502)
- Share the temporal group-by index space (#29498)
- Compute the first dynamic window start in one place (#29484)
- Fix anti/semi join test row with invalid result order expectation (#29467)
- Additional negative Decimal coverage for Iceberg (#29468)
- Add regression test for Decimal/integer arithmetic schema (#29120)
- Tidy the staged parquet predicate (#29408)
- Add dymanicpredicates to preferred build sides (#29335)
- Deprecate cut/qcut (#29329)
- Mark test as slow (#29336)
- Bump maturin (#29282)
- Bump object\_store crate to 0.14.2 (#29317)
- Update rustls dependency to version `0.23.45` (#29303)
- Bump build deps used in ARM64 Windows release pipeline (#29280)
- Fix the stalling `test_fused_many_morsels_and_skew` test (#29290)
- Use `SpillFrame`s in `DataFrameSearchBuffers` (#29207)
- More obvious `MapChunked` storage handling (#29248)
- Centralize hoisted aggregate bookkeeping and clarify grouping predicates (#29287)
- Update analytics endpoint config (#29222)
- Add a blanket `lf.collect().schema == lf.collect_schema()` check (#29224)
- Add expand\_paths parameter to toggle path expansion (#29210)

Thank you to all our contributors for making this release possible!
@ATL2001, @Aidavdw, @AlessandroKuz, @JakubValtar, @Kevin-Patyk, @MarcoGorelli, @Punisheroot, @Rodrigo-Palma, @SatvikMishra08, @TNieuwdorp, @aarushkandukoori, @abokhalill, @alexander-beedie, @atharva7905k, @ayushh0110, @borchero, @c-peters, @cBournhonesque, @carnarez, @dancsi, @dominikandreasseitz, @dsprenkels, @fsimkovic, @gautamvarmadatla, @hadrian-reppas, @jonasdedden, @kafka1991, @kdn36, @krithikashreeL, @lun3x, @madsbk, @matthewbayer, @mikhail5555, @mroeschke, @nameexhaustion, @orlp, @r-brink, @ritchie46, @shaneraphel, @vgvr0 and @wtn