v0.31.0
kazuto-07/TTS-cacherv0.31.0May 25, 2026by github-actions[bot]
Upgrading?
AI Summary
This update introduces a new SQLite write connector and improves Delta Lake compatibility by supporting decimals and dates. It also fixes bugs in HTTP and Delta Lake connectors while introducing a breaking change regarding timezone handling in AWS Glue.
Key Highlights
- New pw.io.sqlite.write connector for writing to SQLite databases.
- Full support for Delta Lake decimal and date types with lossless round-trips.
- Breaking: pw.io.iceberg.write to Glue catalog no longer accepts DateTimeUtc columns.
- Enhanced pw.io.deltalake.read to handle various Parquet integer widths (INT_8, INT_16, etc.).
- Improved SQLite reader to parse more Pathway value types including Duration and Pointer.
Breaking Changes
- pw.io.iceberg.write to a Glue catalog no longer accepts DateTimeUtc columns.
New Features
- pw.io.sqlite.write connector
- pw.io.deltalake.read support for decimal(p, s) and date columns
- pw.io.deltalake.write support for writing into existing Delta decimal columns
- UI panel widget parameters (page_size, table_height)
- pw.io.sqlite.read parsing improvements for complex types
Full Release Notes
### Added - `pw.io.sqlite.write` connector, which writes a Pathway table into a SQLite database file. Supports two modes: `stream_of_changes` (default) appends each event alongside `time`/`diff` metadata columns, while `snapshot` maintains the current state of the table via `INSERT ... ON CONFLICT DO UPDATE` on insertions and `DELETE` on retractions, keyed on the `primary_key` parameter. Values are encoded using the same storage-class mapping that `pw.io.sqlite.read` accepts, so `write` / `read` round-trips every supported Pathway type losslessly. `init_mode` controls whether the destination table is left as-is, auto-created, or replaced on start-up. - `pw.io.deltalake.read` now accepts Delta `decimal(p, s)` columns. The Pathway type declared in the schema chooses the projection: `float` converts each value through f64 (lossy in general — both because f64 is binary and because its mantissa carries only ~15–17 significant decimal digits) and emits a one-time warning at startup naming each affected column; `str` formats the unscaled integer with the column's scale and passes the resulting decimal text through unchanged, lossless for the full Delta precision range (up to 38 digits). - `pw.io.deltalake.write` accepts a Pathway `str` column when writing into an existing Delta `decimal(p, s)` column: each row's text is parsed as decimal and stored as the column's fixed-point value. Combined with the lossless `decimal → str` read path, a Delta `decimal` column can round-trip through a Pathway pipeline with no precision loss. A string that can't be parsed as a decimal of the column's shape fails the write with an error message naming the offending value, the column's precision and scale, and the specific constraint it violated. Tables that don't contain a `decimal` column (or that are being created fresh by Pathway) are unaffected. - `pw.io.deltalake.read` now accepts Delta `date` columns (mapped onto `DateTimeNaive` / `DateTimeUtc` at midnight on the calendar day, since Pathway has no native `Date` type) and `timestamp_millis` columns (mapped onto the same Pathway types with millisecond precision preserved). - The panel widget for table visualization now accepts `page_size` and `table_height` parameters. ### Changed - **BREAKING**: `pw.io.iceberg.write` to a Glue catalog no longer accepts `DateTimeUtc` columns. Glue's metastore has no timezone-aware timestamp type, so previous versions silently dropped the timezone on read-back; writes now fail with an explicit error instead of corrupting the zone. To store UTC timestamps in Glue, convert to `DateTimeNaive` with UTC-normalized values, or write through the REST catalog, which preserves the timezone. - `pw.io.sqlite.read` now parses every Pathway `Value` variant. In addition to `int`, `float`, `str`, `bytes`, `pw.Json`, and their `Optional` forms, the reader now accepts `bool`, `pw.DateTimeNaive`, `pw.DateTimeUtc`, `pw.Duration`, `pw.Pointer`, `pw.PyObjectWrapper`, homogeneous `tuple` / `list`, and `np.ndarray`. Composite types are stored as `TEXT` using the same JSON encoding that `pw.io.jsonlines.write` emits. Booleans additionally accept PostgreSQL-style textual literals (`true`/`false`, `yes`/`no`, `on`/`off`, `t`/`f`, `y`/`n`; case-insensitive, whitespace-trimmed), and `float` columns tolerate values stored with `INTEGER` storage class. - `pw.io.mssql.read` and `pw.io.mssql.write` now retry transient SQL Server errors automatically. ### Fixed - `pw.io.http.rest_connector` no longer raises `TypeError: Cannot instantiate typing.Any` when a request column has the inferred default schema type (`Any`). The cast step now skips columns typed as `Any` instead of attempting to call the type as a constructor. - `pw.io.deltalake.read` now accepts Delta tables whose integer columns use any of the standard Parquet integer widths (`INT_8`, `INT_16`, `INT_32`, unsigned variants), and whose floating-point columns use `FLOAT` (32-bit) or `FLOAT16`. Previously the row-level reader only matched `INT_64` and `DOUBLE`, so tables produced by Spark / DuckDB / pandas with explicit narrower casts read back as zero rows with per-row conversion errors. - `pw.io.deltalake.write` partition columns of type `pw.Pointer`, `pw.Duration`, and `pw.Json` now round-trip correctly through `pw.io.deltalake.read`. Previously the values were correctly placed in the partition path on write, but the reader had no decoder for those types and produced a conversion error for every row.
Related Repositories
public-apis/public-apis★ 491,268
A collective list of free APIs
NousResearch/hermes-agent★ 256,134
The agent that grows with you
yt-dlp/yt-dlp★ 200,318
A feature-rich command-line audio/video downloader
microsoft/markitdown★ 191,598
Python tool for converting files and office documents to Markdown.
Significant-Gravitas/AutoGPT★ 187,705
AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.