Starstruk Logo MarkStarstruk

v0.21.0

pathwaycom/pathwayv0.21.0Mar 19, 2025by github-actions[bot]

AI Summary

Major release adding CSV serialization/deserialization support, updating RAG endpoints to v2, and improving Delta Lake metadata handling.

Key Highlights

  • All Pathway types can now be serialized/deserialized via CSV.
  • RAG endpoints updated to v2 (deprecated /v1, new /v2).
  • Delta Lake write now aligns output metadata with existing table metadata.

Breaking Changes

  • Updated endpoints in `pw.xpacks.llm.question_answering.BaseRAGQuestionAnswerer`.
  • RAG methods renamed and deprecated (e.g., `pw_ai_summary` -> `summarize`).
  • Bytes, Duration, tuple, and np.ndarray serialization rules changed in CSV.
  • Schema parameter is now optional for Delta Lake read.

New Features

  • CSV serialization and deserialization support.
  • RAG endpoint updates.
  • Delta Lake metadata preservation.

Full Release Notes

### Added
- All Pathway types can now be serialized to CSV using `pw.io.csv.write` and deserialized back using `pw.io.csv.read`.
- `pw.io.csv.read` now parses null-values in data when it can be done unambiguously.

### Changed
- **BREAKING**: Updated endpoints in `pw.xpacks.llm.question_answering.BaseRAGQuestionAnswerer`:
  - Deprecated: `/v1/pw_list_documents`, `/v1/pw_ai_answer`
  - New: `/v2/list_documents`, `/v2/answer`
- RAG methods under the `pw.xpacks.llm.question_answering.RAGClient` are re-named, and they now use the new endpoints. Old methods are deprecated and will be removed in the future.
  - `pw_ai_summary` -> `summarize`
  - `pw_ai_answer` -> `answer`
  - `pw_list_documents` -> `list_documents`
- When `pw.io.deltalake.write` creates a table, it also stores its metadata in the columns of the created Delta table. This metadata can be used by Pathway when reading the table with `pw.io.deltalake.read` if no `schema` is specified.
- The `schema` parameter is now optional for `pw.io.deltalake.read`. If the table was created by Pathway and the `schema` was not specified by user, it is read from the table metadata.
- `pw.io.deltalake.write` now aligns the output metadata with the existing table's metadata, preserving any custom metadata in the sink.
- **BREAKING**: The `Bytes` type is now serialized and deserialized with base64 encoding and decoding when the CSV format is used.
- **BREAKING**: The `Duration` type is now serialized and deserialized as a number of nanoseconds when the CSV format is used.
- **BREAKING**: The `tuple` and `np.ndarray` types are now serialized and deserialized as their JSON representations when the CSV format is used.

### Fixed
- `pw.io.csv.write` now correctly escapes quote characters.