v0.16.2
pathwaycom/pathwayv0.16.2Jan 2, 2025by github-actions[bot]
AI Summary
Introduces LLM prompt utilities and enhances Kafka and Delta Lake connectors with metadata support.
Key Highlights
- New `RAGPromptTemplate` and `BaseContextProcessor` utilities for LLMs.
- Kafka read now supports `with_metadata` flag.
- Delta Lake read can now stream tables with deletions.
Breaking Changes
- `pw.io.sharepoint.read` now explicitly terminates with an error if it fails to read data.
- Prompt fields renamed from 'docs' to 'context' and 'query'.
- Removed support for `short_prompt_template` and `long_prompt_template`.
New Features
- `pw.xpacks.llm.prompts.RAGPromptTemplate`.
- `pw.xpacks.llm.question_answering.BaseContextProcessor`.
- `pw.io.kafka.read` now supports `with_metadata` flag.
- `pw.io.deltalake.read` can now stream tables with deletions.
Full Release Notes
### Added - `pw.xpacks.llm.prompts.RAGPromptTemplate`, set of prompt utilities that enable verifying templates and creating UDFs from prompt strings or callables. - `pw.xpacks.llm.question_answering.BaseContextProcessor` streamlines development and tuning of representing retrieved context documents to the LLM. - `pw.io.kafka.read` now supports `with_metadata` flag, which makes it possible to attach the metadata of the Kafka messages to the table entries. - `pw.io.deltalake.read` can now stream the tables with deletions, if no deletion vectors were used. ### Changed - `pw.io.sharepoint.read` now explicitly terminates with an error if it fails to read the data the specified number of times per row (the default is `8`). - `pw.xpacks.llm.prompts.prompt_qa`, and other prompts expect 'context' and 'query' fields instead of 'docs'. - Removed support for `short_prompt_template` and `long_prompt_template` in `pw.xpacks.llm.question_answering.BaseRAGQuestionAnswerer`. These prompt variants are no longer accepted during construction or in requests. - `pw.xpacks.llm.question_answering.BaseRAGQuestionAnswerer` allows setting user created prompts. Templates are verified to include 'context' and 'query' placeholders. - `pw.xpacks.llm.question_answering.BaseRAGQuestionAnswerer` can take a `BaseContextProcessor` that represents context documents to the LLM. Defaults to `pw.xpacks.llm.question_answering.SimpleContextProcessor` which filters metadata fields and joins the documents with new lines. ### Fixed - The input of `pw.io.fs.read` and `pw.io.s3.read` is now correctly persisted in case deletions or modifications of already processed objects take place.