v0.16.2

pathwaycom/pathwayv0.16.2Jan 2, 2025by github-actions[bot]

AI Summary

Introduces LLM prompt utilities and enhances Kafka and Delta Lake connectors with metadata support.

Key Highlights

  • New `RAGPromptTemplate` and `BaseContextProcessor` utilities for LLMs.
  • Kafka read now supports `with_metadata` flag.
  • Delta Lake read can now stream tables with deletions.

Breaking Changes

  • `pw.io.sharepoint.read` now explicitly terminates with an error if it fails to read data.
  • Prompt fields renamed from 'docs' to 'context' and 'query'.
  • Removed support for `short_prompt_template` and `long_prompt_template`.

New Features

  • `pw.xpacks.llm.prompts.RAGPromptTemplate`.
  • `pw.xpacks.llm.question_answering.BaseContextProcessor`.
  • `pw.io.kafka.read` now supports `with_metadata` flag.
  • `pw.io.deltalake.read` can now stream tables with deletions.

Full Release Notes

### Added
- `pw.xpacks.llm.prompts.RAGPromptTemplate`, set of prompt utilities that enable verifying templates and creating UDFs from prompt strings or callables.
- `pw.xpacks.llm.question_answering.BaseContextProcessor` streamlines development and tuning of representing retrieved context documents to the LLM.
- `pw.io.kafka.read` now supports `with_metadata` flag, which makes it possible to attach the metadata of the Kafka messages to the table entries.
- `pw.io.deltalake.read` can now stream the tables with deletions, if no deletion vectors were used.

### Changed
- `pw.io.sharepoint.read` now explicitly terminates with an error if it fails to read the data the specified number of times per row (the default is `8`).
- `pw.xpacks.llm.prompts.prompt_qa`, and other prompts expect 'context' and 'query' fields instead of 'docs'. 
- Removed support for `short_prompt_template` and `long_prompt_template` in `pw.xpacks.llm.question_answering.BaseRAGQuestionAnswerer`. These prompt variants are no longer accepted during construction or in requests.
- `pw.xpacks.llm.question_answering.BaseRAGQuestionAnswerer` allows setting user created prompts. Templates are verified to include 'context' and 'query' placeholders.
- `pw.xpacks.llm.question_answering.BaseRAGQuestionAnswerer` can take a `BaseContextProcessor` that represents context documents to the LLM. Defaults to `pw.xpacks.llm.question_answering.SimpleContextProcessor` which filters metadata fields and joins the documents with new lines.

### Fixed
- The input of `pw.io.fs.read` and `pw.io.s3.read` is now correctly persisted in case deletions or modifications of already processed objects take place.