v0.15.2

pathwaycom/pathwayv0.15.2Oct 25, 2024by github-actions[bot]

AI Summary

Enhances Delta Lake support for HTTP endpoints and optimizes S3 caching and nearest neighbor indices.

Key Highlights

Delta Lake read supports custom S3 endpoints with HTTP.
Nearest neighbor indices now work on numpy arrays.
State persistence configuration simplified.

New Features

`pw.io.deltalake.read` now supports custom S3 Delta Lakes with HTTP endpoints.
Indices in `pathway.stdlib.indexing.nearest_neighbors` can now work on numpy arrays.
It is now possible to set the size limit of cache in `pw.udfs.DiskCache`.

Full Release Notes

### Added
- `pw.io.deltalake.read` now supports custom S3 Delta Lakes with HTTP endpoints.
- `pw.io.deltalake.read` now supports specifying both a custom endpoint and a custom region for Delta Lakes via `pw.io.s3.AwsS3Settings`.

### Changed
- Indices in `pathway.stdlib.indexing.nearest_neighbors` can now work also on numpy arrays. Previously they only accepted `list[float]`. Working with numpy arrays improves memory efficiency.
- `pw.io.s3.read` has been optimized to minimize new object requests whenever possible.
- It is now possible to set the size limit of cache in `pw.udfs.DiskCache`.
- State persistence now uses a single backend for both metadata and stream storage. The `pw.persistence.Config.simple_config` method is therefore deprecated. Now you can use the `pw.persistence.Config` constructor with the same parameters that were previously used in `simple_config`.

### Fixed
- `pw.io.bigquery.write` connector now correctly handles `pw.Json` columns.