v0.19.0
pathwaycom/pathwayv0.19.0Feb 20, 2025by github-actions[bot]
AI Summary
Introduced fully asynchronous UDFs with Futures support, added Iceberg/S3 backend support, and enabled output sorting for all connectors.
Key Highlights
- Added fully asynchronous UDF executor and Future data type.
- Iceberg connectors now support S3 and AWS Glue catalog.
- All output connectors now support `sort_by` field.
Breaking Changes
- Changed interface of LLMReranker (removed `use_logit_bias`, etc.).
- LLMReranker no longer inherits from pw.UDF.
- AsyncTransformer.output_table now returns Future data type.
New Features
- Fully asynchronous UDFs.
- Iceberg/S3 support.
- Output connector sorting support.
Full Release Notes
### Added - `LLMReranker` now supports custom prompts as well as custom response parsers allowing for other ranking scales apart from default 1-5. - `pw.io.kafka.write` and `pw.io.nats.write` now support `ColumnReference` as a topic name. When a `ColumnReference` is provided, each message's topic is determined by the corresponding column value. - `pw.io.python.write` accepting `ConnectorObserver` as an alternative to `pw.io.subscribe`. - `pw.io.iceberg.read` and `pw.io.iceberg.write` now support S3 as data backend and AWS Glue catalog implementations. - All output connectors now support the `sort_by` field for ordering output within a single minibatch. - A new UDF executor `pw.udfs.fully_async_executor`. It allows for creation of non-blocking asynchronous UDFs which results can be returned in the future processing time. - A Future data type to represent results of fully asynchronous UDFs. - `pw.Table.await_futures` method to wait for results of fully asynchronous UDFs. - `pw.io.deltalake.write` now supports partition columns specification. ### Changed - **BREAKING**: Changed the interface of `LLMReranker`, the `use_logit_bias`, `cache_strategy`, `retry_strategy` and `kwargs` arguments are no longer supported. - **BREAKING**: LLMReranker no longer inherits from pw.UDF - **BREAKING**: `pw.stdlib.utils.AsyncTransformer.output_table` now returns a table with columns with Future data type. - `pw.io.deltalake.read` can now read append-only tables without requiring explicit specification of primary key fields.