Starstruk Logo MarkStarstruk

v0.16.1

pathwaycom/pathwayv0.16.1Dec 12, 2024by github-actions[bot]

AI Summary

Enhances S3 connector to detect deletions and modifications in streaming mode.

Key Highlights

  • S3 connector now monitors object deletions and modifications in streaming mode.
  • S3 read now supports `with_metadata` flag.

New Features

  • `pw.io.s3.read` now monitors object deletions and modifications.
  • `pw.io.s3.read` now supports `with_metadata` flag.

Full Release Notes

### Changed
- `pw.io.s3.read` now monitors object deletions and modifications in the S3 source, when ran in streaming mode. When an object is deleted in S3, it is also removed from the engine. Similarly, if an object is modified in S3, the engine updates its state to reflect those changes.
- `pw.io.s3.read` now supports `with_metadata` flag, which makes it possible to attach the metadata of the source object to the table entries.

### Fixed
- `pw.xpacks.llm.document_store.DocumentStore` no longer requires `_metadata` column in the input table.