Starstruk Logo MarkStarstruk

v0.18.0

pathwaycom/pathwayv0.18.0Feb 7, 2025by github-actions[bot]

AI Summary

Added JSON serialization support for all Pathway types, fixed PostgreSQL serialization, and updated Pointer type handling.

Key Highlights

  • All Pathway types can now be serialized into JSON.
  • PostgreSQL connectors now handle PyObjectWrapper and Timedelta properly.
  • Pointer type serialized as string in Iceberg and Delta Lake.

Breaking Changes

  • Changed interface of UnstructuredParser.
  • Pointer type serialized as string in Iceberg and Delta Lake.
  • Bytes type serialized with base64 in JSON.
  • Array type serialized as object in JSON.
  • Removed undocumented `license_key` argument from pw.run.

New Features

  • JSON serialization support.
  • PostgreSQL serialization fixes.

Full Release Notes

### Added
- `pw.io.postgres.write` and `pw.io.postgres.write_snapshot` now handle serialization of `PyObjectWrapper` and `Timedelta` properly.
- New chunking options in `pathway.xpacks.llm.parsers.UnstructuredParser`
- Now all Pathway types can be serialized into JSON and consistently deserialized back.
- `table.col.dt.to_duration` converting an integer into a `pw.Duration`.
- `pw.Json` now supports storing datetime and duration type values in ISO format.

### Changed
- **BREAKING**: Changed the interface of `UnstructuredParser`
- **BREAKING**: The `Pointer` type is now serialized and deserialized as a string field in Iceberg and Delta Lake.
- **BREAKING**: The `Bytes` type is now serialized and deserialized with base64 encoding and decoding when the JSON format is used. A string field is used to store the encoded contents.
- **BREAKING**: The `Array` type is now serialized and deserialized as an object with two fields: `shape` denoting the shape of the stored multi-dimensional array and `elements` denoting the elements of the flattened array.
- **BREAKING**: Marked package as **py.typed** to indicate support for type hints.

### Removed
- **BREAKING**: Removed undocumented `license_key` argument from `pw.run` and `pw.run_all` methods. Instead, `pw.set_license_key` should be used.