v0.17.0
pathwaycom/pathwayv0.17.0Jan 31, 2025by github-actions[bot]
AI Summary
This release introduces Apache Iceberg support, a new Docling parser for complex PDFs, and enhanced connector naming for monitoring. It also consolidates sorting logic and removes a significant number of deprecated APIs and legacy join methods.
Key Highlights
- Added `pw.io.iceberg.read` for reading Apache Iceberg tables.
- New `DoclingParser` supporting parsing of PDFs with tables and images.
- Connectors now include an optional `name` parameter for logs and monitoring.
- Sorting functionality consolidated into `pw.Table.sort`.
Breaking Changes
- `pw.io.deltalake.read` now requires explicit specification of primary key fields.
- `pw.xpacks.llm.question_answering.BaseRAGQuestionAnswerer` now returns a dictionary from `pw_ai_answer` endpoint.
- `pw.indexing.build_sorted_index`, `pw.indexing.retrieve_prev_next_values`, `pw.indexing.sort_from_index` and `pw.indexing.SortedIndex` are removed.
- Deprecated methods `pw.Table.unsafe_promise_same_universe_as`, `pw.Table.left_join`, `pw.Table.right_join`, `pw.Table.outer_join`, etc., are removed.
- Removed deprecated arguments from input connectors: `value_columns`, `primary_key`, `types`, `default_values`.
New Features
- `pw.io.iceberg.read` method.
- `pw.io.postgres.write` and `pw.io.postgres.write_snapshot` with `init_mode` argument.
- `pw.io.deltalake.read` now supports serialization and deserialization for all Pathway data types.
- `pathway.xpacks.llm.parsers.DoclingParser`.
- Output connectors now include an optional `name` parameter.
Full Release Notes
### Added - `pw.io.iceberg.read` method for reading Apache Iceberg tables into Pathway. - methods `pw.io.postgres.write` and `pw.io.postgres.write_snapshot` now accept an additional argument `init_mode`, which allows initializing the table before writing. - `pw.io.deltalake.read` now supports serialization and deserialization for all Pathway data types. - New parser `pathway.xpacks.llm.parsers.DoclingParser` supporting parsing of pdfs with tables and images. - Output connectors now include an optional `name` parameter. If provided, this name will appear in logs and monitoring dashboards. - Automatic naming for input and output connectors has been enhanced. ### Changed - **BREAKING**: `pw.io.deltalake.read` now requires explicit specification of primary key fields. - **BREAKING**: `pw.xpacks.llm.question_answering.BaseRAGQuestionAnswerer` now returns a dictionary from `pw_ai_answer` endpoint. - `pw.xpacks.llm.question_answering.BaseRAGQuestionAnswerer` allows optionally returning context documents from `pw_ai_answer` endpoint. - **BREAKING**: When using delay in temporal behavior, current time is updated immediately, not in the next batch. - **BREAKING**: The `Pointer` type is now serialized to Delta Tables as raw bytes. - `pw.io.kafka.write` now allows to specify `key` and `headers` for JSON and CSV data formats. - `persistent_id` parameter in connectors has been renamed to `name`. This new `name` parameter allows you to assign names to connectors, which will appear in logs and monitoring dashboards. - Changed names of parsers to be more consistent: `ParseUnstrutured` -> `UnstructuredParser`, `ParseUtf8` -> `Utf8Parser`. `ParseUnstrutured` and `ParseUtf8` are now deprecated. ### Fixed - `generate_class` method in `Schema` now correctly renders columns of `UnionType` and `None` types. - a bug in delay in temporal behavior. It was possible to emit a single entry twice in a specific situation. - `pw.io.postgres.write_snapshot` now correctly handles tables that only have primary key columns. ### Removed - **BREAKING**: `pw.indexing.build_sorted_index`, `pw.indexing.retrieve_prev_next_values`, `pw.indexing.sort_from_index` and `pw.indexing.SortedIndex` are removed. Sorting is now done with `pw.Table.sort`. - **BREAKING**: Removed deprecated methods `pw.Table.unsafe_promise_same_universe_as`, `pw.Table.unsafe_promise_universes_are_pairwise_disjoint`, `pw.Table.unsafe_promise_universe_is_subset_of`, `pw.Table.left_join`, `pw.Table.right_join`, `pw.Table.outer_join`, `pw.stdlib.utils.AsyncTransformer.result`. - **BREAKING**: Removed deprecated column `_pw_shard` in the result of `windowby`. - **BREAKING**: Removed deprecated functions `pw.debug.parse_to_table`, `pw.udf_async`, `pw.reducers.npsum`, `pw.reducers.int_sum`, `pw.stdlib.utils.col.flatten_column`. - **BREAKING**: Removed deprecated module `pw.asynchronous`. - **BREAKING**: Removed deprecated access to functions from `pw.io` in `pw`. - **BREAKING**: Removed deprecated classes `pw.UDFSync`, `pw.UDFAsync`. - **BREAKING**: Removed class `pw.xpack.llm.parsers.OpenParse`. It's functionality has been replaced with `pw.xpack.llm.parsers.DoclingParser`. - **BREAKING**: Removed deprecated arguments from input connectors: `value_columns`, `primary_key`, `types`, `default_values`. Schema should be used instead.