v2.3.2

Spice v2.3.2 makes point lookups and repeated queries faster with Spice Cayenne. SQL results-cache improvements apply to every cached query, regardless of its data source or accelerator. Across two benchmark rounds, the time for a small cache hit fell by 50-52% over HTTP and 44-48% over Flight SQL. For small cache hits, CPU time per server request fell by 58-61% over HTTP and 49-51% over Flight SQL.

Highlights in v2.3.2 include:

- [**Indexed Point Lookups**](#faster-point-lookups-with-cayenne-indexes) — Cayenne now honors `indexes`, so a lookup reads the matching rows instead of scanning
- [**Cayenne Data Layout**](#control-how-cayenne-lays-out-data) — `cayenne_cluster_by` groups related rows together across storage tiers
- [**Faster Queries After a Full Refresh**](#faster-queries-after-a-full-refresh) — refreshed tables are written so that filtered queries read fewer files
- [**Faster Cached Responses**](#faster-cached-query-responses) — these improvements apply to every query in the SQL results cache, regardless of its data source or accelerator. For small results, cache-hit time fell by 44-52%
- [**Faster Repeat Queries**](#faster-repeat-queries-on-cayenne) — Cayenne reuses its prepared view of a table until the data changes
- [**Localpod Fix**](#localpod-datasets-stay-in-sync-with-their-source) — a localpod dataset no longer serves data its source has replaced
- [**Catalog Fix**](#catalog-tables-no-longer-go-missing) — tables no longer go missing when datasets load at the same time
- [**Safer Cayenne Configuration**](#cayenne-detects-an-ambiguous-metastore-at-startup) — an ambiguous metastore configuration is reported instead of appearing as lost data
- [**Distributed HTTP Queries**](#distributed-queries-over-http-datasets) — async distributed queries can read HTTP datasets

## What's New in v2.3.2

### Faster Point Lookups with Cayenne Indexes

Every other accelerator already accepted `indexes`, and [Cayenne](https://spiceai.org/docs/components/data-accelerators/cayenne) logged that it ignored them. Cayenne now builds an index for each entry, in both `mode: file` and `mode: memory`:

```yaml
acceleration:
  engine: cayenne
  mode: file # or memory
  indexes:
    '(TenantId, ServiceId)': enabled
```

A query that pins every column of an index to a value, such as `WHERE TenantId = 7 AND ServiceId = 'a'`, now reads the matching rows directly. Without an index, Cayenne can only skip files whose minimum and maximum values rule the key out, which rarely helps when related rows are spread across the table — a lookup on a 5.4M-row test dataset had to open about half its files for nearly every key.

Indexes only reduce what a query reads, so results are identical either way. A query that uses a range, an `IN` list, an `OR`, or a cast on the indexed column reads the table as before. Index definitions are not stored with the table, so adding or removing one takes effect the next time the dataset loads. Floating-point columns cannot be indexed and are reported at load time.

`EXPLAIN` shows whether a query used an index, and the `cayenne_lookup_index_probe_total` metric counts lookups by outcome.

### Control How Cayenne Lays Out Data

`cayenne_cluster_by` stores rows with similar values near each other, so a filtered query reads fewer files. It applies to every storage tier, replacing `cayenne_datalake_clustering_columns`, which affected only the coldest tier.

```yaml
acceleration:
  engine: cayenne
  params:
    cayenne_cluster_by: 'tenant_id, event_time'
```

`CREATE TABLE ... CLUSTER BY (column, ...)` is also supported, and several cases it previously rejected — a single column in parentheses, and names whose capitalization differs from the column definition — now work. A column name that does not exist is reported when the dataset loads rather than failing later.

### Faster Queries After a Full Refresh

A [full refresh](https://spiceai.org/docs/features/data-acceleration/refresh-modes/full) previously spread each key across every file it wrote, so a filtered query had to open all of them even when it wanted a single row. A refreshed table is now written so that each file holds a distinct range of the data, and queries filtering on that range read only the files that can match.

This needs no configuration and applies to any dataset whose accelerated table is replaced by a refresh. Datasets that already set `cayenne_sort_columns` or `cayenne_cluster_by` keep their existing layout, and the very first load is unchanged.

### Faster Cached Query Responses

Before this release, the runtime planned a query before it checked the [SQL results cache](https://spiceai.org/docs/features/caching). It also repeated other work that a cache hit does not need. The runtime now checks the cache first and returns cached answers directly. These improvements apply to every cached query, regardless of its data source or accelerator.

Across rounds of benchmarking queries that returned a seven-row `GROUP BY` result, cache-hit time over HTTP fell by 50-52%. Over Flight SQL, cache-hit time fell by 44-48%. In one HTTP round, cache-hit time fell from 63.8 µs to 30.8 µs. In one round over Flight SQL, it fell from 55.3 µs to 30.9 µs.

With task history enabled, the server used 125 µs of CPU time for a small HTTP hit before the change. It used 49 µs after the change. For a small result over Flight SQL, the server used 330 µs before the change. It used 168 µs after the change. These changes reduced CPU time by 61% and 49%, respectively.

For a wide result over HTTP, the server used 241 µs of CPU time before the change. It used 116 µs after the change. This is a 52% reduction. Wide results did not reduce the runtime's cache-hit time in those runs. Some measurements were slower.

Raw cache entries now share each stored batch with streaming JSON HTTP and Flight SQL responses. Buffered HTTP formats — including CSV, plain text, `vnd.*` envelopes, and JSON with union columns — still clone batches while encoding.

A stream benchmark with eight 20-column batches fell from 1.8722 µs to 226.23 ns. This is an 8.3x speedup. One 200-column batch fell from 2.0190 µs to 131.94 ns. This is a 15x speedup. These figures measure the construction and full consumption of the stream. They do not measure query latency from start to finish.

For compressed entries, the first read keeps the entry compressed. The second read promotes it to raw data when the raw size fits the cache. Later reads avoid decompression. In a cache benchmark, a later read took 17.1-17.8 µs across three payload sizes. A decode took 39.1-154.8 µs. Cached answers, expiry times, and cache settings are unchanged.

### Faster Repeat Queries on Cayenne

Cayenne prepares a view of a table's current contents before it can answer a query. It now reuses that preparation until the data actually changes, instead of rebuilding it, so repeated queries against a table that is not being written return faster and stop querying the metastore entirely once warm.

Datasets fed by continuous change data capture — a `cdc:` or `debezium:` source, or `refresh_mode: changes` — reuse the preparation for up to one second so a burst of incoming changes can share it. Datasets that accept writes always see their own writes immediately. No configuration is required.

### Better Pruning for Decimal, Binary, and Time Columns

Cayenne did not record minimum and maximum values for decimal, binary, time, and 16-bit float columns, so queries filtering on them could not skip files and had to read more data than necessary. Those columns now carry the same statistics as every other type, and clustering on a decimal column works as intended.

After a change that widens a decimal column's scale, Cayenne discards the affected statistics and rebuilds them, so queries read a little more until that completes. Results are unaffected.

### Localpod Datasets Stay in Sync with Their Source

A [localpod](https://spiceai.org/docs/components/data-connectors/localpod) dataset reads through another dataset in the same Spicepod. When the source dataset was reloaded — because its configuration changed — the localpod dataset kept reading the replaced copy, so it answered with data the source no longer had, and both copies kept refreshing.

A localpod dataset now reloads whenever the dataset it reads through does, including through several levels of chaining, and its cached results are cleared at the same time. This also works when the source is named with its full path, such as `localpod:spice.public.parent`. Fixes [#3288](https://github.com/spiceai/spiceai/issues/3288).

### Catalog Tables No Longer Go Missing

When two datasets in the same schema loaded at the same time, one could be silently discarded, and every later query against it failed with `Table not found`. Both datasets reported that they had loaded successfully, and which one went missing varied between restarts. Datasets that share a schema now always both register.

### Cayenne Detects an Ambiguous Metastore at Startup

Cayenne datasets that each set a different `cayenne_file_path`, without a shared `cayenne_metadata_dir`, could open the wrong metadata directory after a restart. Cayenne then started up empty even though the data was still on disk, which looks like a total loss of accelerated data.

This configuration is now rejected at startup, naming each dataset and path involved and linking to the documentation. If a Spicepod uses several Cayenne data paths, set the same `cayenne_metadata_dir` on each dataset before upgrading.

### Distributed Queries over HTTP Datasets

An async distributed query submitted to `/v1/queries` failed if it read an unaccelerated HTTP dataset. These queries now run, with credentials, headers, and pagination behaving as they do for a non-distributed query. Fixes [#14104](https://github.com/spiceai/spiceai/issues/14104).

### Other Fixes

- **Partitioned datasets**: a full refresh of a partitioned dataset only replaced the partitions the new data reached, so rows deleted at the source stayed queryable, and a refresh that returned no rows changed nothing. Every partition is now replaced.
- **Iceberg write-through**: a write to a partitioned dataset could deadlock against a refresh running at the same time, leaving both waiting.
- **[BigQuery](https://spiceai.org/docs/components/data-connectors/bigquery)**: a case-insensitive `LIKE` could fail the query or return the wrong rows, because BigQuery has no `ILIKE`. Spice now evaluates it locally. Ordinary `LIKE` is unchanged.
- **[PostgreSQL](https://spiceai.org/docs/components/data-connectors/postgres)**: timestamps written back to PostgreSQL lost everything below the second. Microseconds are now preserved.

### Dependency Updates

No crate versions changed in this release. DataFusion remains at v54.1.0, Arrow remains at v58.3.0, and Vortex remains at v0.79.0.

Spice updates two fork revisions: `datafusion-table-providers` for the BigQuery and PostgreSQL fixes above, and `duckdb-rs` so the bundled DuckDB builds against the macOS 27 SDK.

## Contributors

- [@lukekim](https://github.com/lukekim)
- [@phillipleblanc](https://github.com/phillipleblanc)

## Breaking Changes

**`cayenne_datalake_clustering_columns` is replaced by `cayenne_cluster_by`.** Rename the parameter before upgrading. The new one groups data on every storage tier, not only the coldest:

```yaml
acceleration:
  engine: cayenne
  params:
    cayenne_cluster_by: 'tenant_id, event_time'
```

A dataset that sets both `cayenne_sort_columns` and a cluster key is now rejected when it loads. Remove `cayenne_sort_columns` to keep the cluster key.

**Cayenne datasets using several data paths must share a metastore.** If your Cayenne datasets set different `cayenne_file_path` values, set the same `cayenne_metadata_dir` on each one before upgrading. Spice now refuses to start on this configuration instead of risking an empty-looking acceleration:

```yaml
acceleration:
  engine: cayenne
  params:
    cayenne_file_path: /mnt/a/cayenne
    cayenne_metadata_dir: /mnt/shared/cayenne-metadata
```

## Cookbook Updates

No new cookbook recipes.

The [Spice Cookbook](https://spiceai.org/cookbook) includes more than 104 recipes to help you get started with Spice quickly and easily.

## Upgrading

To upgrade to v2.3.2, use one of the following methods:

**CLI**:

```console
spice upgrade
```

**Homebrew**:

```console
brew upgrade spiceai/spiceai/spice
```

**Docker**:

Pull the `spiceai/spiceai:2.3.2` image:

```console
docker pull spiceai/spiceai:2.3.2
```

For available tags, see [DockerHub](https://hub.docker.com/r/spiceai/spiceai/tags).

**Helm**:

```console
helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.3.2
```

**AWS Marketplace**:

Spice is available in the [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-jmf6jskjvnq7i).

## What's Changed

### Changelog

- fix(postgres): preserve microseconds in timestamp writeback by [@phillipleblanc](https://github.com/phillipleblanc) in [#13963](https://github.com/spiceai/spiceai/pull/13963)
- feat(cayenne): reuse ScanView until write, lag only for read-only CDC by [@lukekim](https://github.com/lukekim) in [#14055](https://github.com/spiceai/spiceai/pull/14055)
- perf: serve results-cache hits where the request arrives and cut per-hit overhead by [@lukekim](https://github.com/lukekim) in [#14103](https://github.com/spiceai/spiceai/pull/14103)
- fix(cluster): support distributed HTTP scans by [@phillipleblanc](https://github.com/phillipleblanc) in [#14108](https://github.com/spiceai/spiceai/pull/14108)
- fix(bigquery): keep ILIKE evaluation local by [@phillipleblanc](https://github.com/phillipleblanc) in [#14110](https://github.com/spiceai/spiceai/pull/14110)
- feat(cayenne): cluster warm and datalake tiers, and write full refreshes as key-range files by [@lukekim](https://github.com/lukekim) in [#14124](https://github.com/spiceai/spiceai/pull/14124)
- fix(cayenne): reject ambiguous metastore paths by [@phillipleblanc](https://github.com/phillipleblanc) in [#14130](https://github.com/spiceai/spiceai/pull/14130)
- fix(cayenne): round-trip decimal, binary, and time stats and drop them on scale change by [@lukekim](https://github.com/lukekim) in [#14139](https://github.com/spiceai/spiceai/pull/14139)
- feat(cayenne): build secondary indexes from `indexes` in file and memory mode by [@phillipleblanc](https://github.com/phillipleblanc) in [#14149](https://github.com/spiceai/spiceai/pull/14149)
- fix(catalog): keep the tables registered under an existing schema by [@phillipleblanc](https://github.com/phillipleblanc) in [#14193](https://github.com/spiceai/spiceai/pull/14193)
- ci: stop exporting empty AWS and DuckLake endpoints to the schema test by [@phillipleblanc](https://github.com/phillipleblanc) in [#14194](https://github.com/spiceai/spiceai/pull/14194)
- perf(cache): promote encoded SQL results to raw after the second decode by [@lukekim](https://github.com/lukekim) in [#14199](https://github.com/spiceai/spiceai/pull/14199)
- perf(cache): tighten the Raw SQL results-cache serve path by [@lukekim](https://github.com/lukekim) in [#14205](https://github.com/spiceai/spiceai/pull/14205)
- fix(runtime): reload a localpod dataset when the dataset it reads through is reloaded (fixes #3288) by [@claudespice](https://github.com/claudespice) in [#14208](https://github.com/spiceai/spiceai/pull/14208)

**Full Changelog**: [https://github.com/spiceai/spiceai/compare/v2.3.1...v2.3.2](https://github.com/spiceai/spiceai/compare/v2.3.1...v2.3.2)