v0.19.0

pathwaycom/pathwayv0.19.0Feb 20, 2025by github-actions[bot]

AI Summary

Introduced fully asynchronous UDFs with Futures support, added Iceberg/S3 backend support, and enabled output sorting for all connectors.

Key Highlights

  • Added fully asynchronous UDF executor and Future data type.
  • Iceberg connectors now support S3 and AWS Glue catalog.
  • All output connectors now support `sort_by` field.

Breaking Changes

  • Changed interface of LLMReranker (removed `use_logit_bias`, etc.).
  • LLMReranker no longer inherits from pw.UDF.
  • AsyncTransformer.output_table now returns Future data type.

New Features

  • Fully asynchronous UDFs.
  • Iceberg/S3 support.
  • Output connector sorting support.

Full Release Notes

### Added
- `LLMReranker` now supports custom prompts as well as custom response parsers allowing for other ranking scales apart from default 1-5.
- `pw.io.kafka.write` and `pw.io.nats.write` now support `ColumnReference` as a topic name. When a `ColumnReference` is provided, each message's topic is determined by the corresponding column value.
- `pw.io.python.write` accepting `ConnectorObserver` as an alternative to `pw.io.subscribe`.
- `pw.io.iceberg.read` and `pw.io.iceberg.write` now support S3 as data backend and AWS Glue catalog implementations.
- All output connectors now support the `sort_by` field for ordering output within a single minibatch.
- A new UDF executor `pw.udfs.fully_async_executor`. It allows for creation of non-blocking asynchronous UDFs which results can be returned in the future processing time.
- A Future data type to represent results of fully asynchronous UDFs.
- `pw.Table.await_futures` method to wait for results of fully asynchronous UDFs.
- `pw.io.deltalake.write` now supports partition columns specification.

### Changed
- **BREAKING**: Changed the interface of `LLMReranker`, the `use_logit_bias`, `cache_strategy`, `retry_strategy` and `kwargs` arguments are no longer supported.
- **BREAKING**: LLMReranker no longer inherits from pw.UDF
- **BREAKING**: `pw.stdlib.utils.AsyncTransformer.output_table` now returns a table with columns with Future data type.
- `pw.io.deltalake.read` can now read append-only tables without requiring explicit specification of primary key fields.