v2.2.1

Spice v2.2.1 is a patch release that improves accelerated data management, query federation, and search. Spice Cayenne now reclaims the storage that upserts and compaction leave behind, and new metrics report what each maintenance pass did. Retention policies apply to more Cayenne datasets. Federated queries push more work down to BigQuery. DuckDB and HTTP datasets keep their memory bounded. Vector search with HuggingFace embedding models returns more relevant results.

## What's New in v2.2.1

### Cayenne Reclaims Storage Automatically

A [Spice Cayenne](https://spiceai.org/docs/components/data-accelerators/cayenne) acceleration now reclaims the space that a write or a compaction supersedes. Each maintenance pass removes four kinds of state that earlier releases kept:

- tombstones for superseded upsert rows
- snapshot directories that a later commit replaced
- manifest and catalog rows for snapshots that compaction merged away
- deletion vectors that a compaction orphaned

Cayenne also releases the metastore pages that a pass frees. On a table with frequent refreshes or CDC upserts, disk and metastore usage now stay bounded. No configuration change is required.

### Retention Policies Apply to More Cayenne Datasets

Cayenne now applies `retention_sql` on datasets with [`refresh_mode: full`](https://spiceai.org/docs/features/data-acceleration/refresh-modes/full) and on CDC-accelerated datasets with [`refresh_mode: changes`](https://spiceai.org/docs/features/data-acceleration/refresh-modes/changes). Before this release, only the append refresh path ran the retention pass. A full refresh reloaded every source row, and the rows that a policy had removed came back.

A `retention_sql` predicate that calls `now()` also works now. Cayenne prepared the predicate once when the table opened, at a point where DataFusion cannot evaluate `now()`, so the pass failed and removed nothing. Cayenne now resolves `now()` at the start of each pass. See [Data Refresh](https://spiceai.org/docs/features/data-acceleration/data-refresh) for the retention settings.

```yaml
datasets:
  - from: postgres:events
    name: events
    acceleration:
      engine: cayenne
      retention_sql: DELETE FROM events WHERE created_at < now() - INTERVAL '7 days'
```

Cayenne also accepts [`refresh_append_overlap`](https://spiceai.org/docs/features/data-acceleration/refresh-modes/append) now. It warns when a dataset sets `retention_sql` together with `mode: memory`, or sets `indexes`, because it cannot apply either setting.

### Cayenne Maintenance and Storage Metrics

Cayenne now reports its maintenance decisions and its storage footprint as metrics on the `cayenne` meter. The runtime serves them at `/metrics` when `spiced` starts with `--metrics <addr>`. An operator can now tell a pass that found nothing from a pass that declined. The metrics also show where disk and metastore space goes. See [Metrics](https://spiceai.org/docs/features/observability/component_metrics) for the metrics endpoint.

- `cayenne_compaction_outcome_total` and `cayenne_maintenance_outcome_total` count each pass by outcome, such as `committed`, `no_op`, or `declined_<reason>`.
- `cayenne_compaction_trigger_total` names the threshold that requested a pass.
- `cayenne_maintenance_reclaimed_files_total`, `cayenne_maintenance_reclaimed_bytes_total`, and `cayenne_maintenance_reclaimed_rows_total` count what each pass reclaimed.
- `cayenne_storage_files`, `cayenne_storage_bytes`, and `cayenne_storage_rows` report the live footprint by storage tier.
- `cayenne_data_dir_files`, `cayenne_data_dir_bytes`, and `cayenne_data_dir_snapshot_dirs` report what the data directory holds on disk.
- `cayenne_metastore_db_bytes`, `cayenne_metastore_wal_bytes`, and `cayenne_metastore_table_rows` report the size of the metastore.

### Query Federation and SQL Improvements

- **ADBC federation**: The [ADBC Data Connector](https://spiceai.org/docs/components/data-connectors/adbc) again keeps Spice-only functions such as `json_get_str` out of the SQL it sends to a source, and it honours [`query_federation: disabled`](https://spiceai.org/docs/features/query-federation). The v2.2.0 move of the connector into its own crate dropped both settings. A query with a Spice-only function against BigQuery then failed with an unknown-function error.
- **JSON functions in BigQuery**: BigQuery federation now translates [`json_get_int` and `json_get_float`](https://spiceai.org/docs/reference/sql/json) into BigQuery SQL. A filter or a join on a JSON value runs in BigQuery instead of in Spice.
- **Negative JSON numbers**: `json_get_int` and `json_get_float` now read a negative JSON number. Every earlier release returned `NULL` for one.
- **Joins between described datasets**: An `INNER` or `FULL` join between two datasets that both set a [`description`](https://spiceai.org/docs/reference/spicepod/datasets) now plans. DataFusion carries the description as schema metadata, and the join failed with a schema mismatch error. This release updates DataFusion with the fix.

### Cayenne Query and Delete Fixes

- **Predicate deletes on upsert tables**: A [`DELETE`](https://spiceai.org/docs/reference/sql/dml) with a non-primary-key predicate on a Cayenne upsert table now removes only the rows that match. The delete matched the predicate against superseded row versions as well as live rows. A key whose old version matched therefore lost its live row too. The delete now applies the same visibility rules as a query.
- **Date filters on Cayenne data files**: A pushed-down `CAST(date_col AS TIMESTAMP)` comparison failed the scan for a `Date32` column. For a `Date64` column, the same comparison skipped files that held matching rows, so the query returned fewer rows. The scan cast the file's date statistic through the wrong storage type. It now converts the statistic before it prunes on it.
- **Primary keys after an interrupted index rebuild**: A rebuild of the primary-key index that failed or was cancelled left the index disabled for the life of the process. A later upsert could then write a second live row for a declared primary key, and every write rebuilt the whole keyset. Cayenne now closes the rebuild window on every exit path.
- **Row counts after a commit**: A distributed `COUNT(*)` that Cayenne answered from its maintained row count could miss the rows of a commit that had just become visible. Cayenne now claims a commit's row-count delta before it publishes the rows.
- **Point lookups on CDC datasets**: A primary-key lookup with an `ORDER BY` on the same key failed to plan on a [CDC-accelerated](https://spiceai.org/docs/features/cdc) Cayenne dataset after its first transactional commit. An example is `WHERE id = 7 ORDER BY id`. The runtime rejected the query and returned an error. The scan now forwards its constant columns across the schema cast, and the query plans again.
- **Arrow `Map` columns in file mode**: A Cayenne accelerator with [`mode: file`](https://spiceai.org/docs/components/data-accelerators/cayenne) now stores Arrow `Map` columns, such as the `response_headers` column of an HTTP dataset. Every write of such a column failed, and the acceleration stayed empty.

### DuckDB and HTTP Datasets Keep Memory Bounded

**DuckDB**: The bundled [DuckDB](https://spiceai.org/docs/components/data-accelerators/duckdb) moves from v1.5.5 to v1.4.4. On v1.5.5, a DuckDB-accelerated dataset that refreshed through an `ON CONFLICT ... DO UPDATE` upsert grew its memory by about 6 MiB per refresh. The row count stayed steady while the memory grew, and the write eventually failed with an out-of-memory error. The DuckDB project tracks the regression as [duckdb/duckdb#25162](https://github.com/duckdb/duckdb/issues/25162), and it is present in every v1.5.x release. v1.4.4 holds a steady footprint. Queries that run during a file-mode DuckDB swap also now recover when the swap invalidates their statements. The configured database path stays attached across the swap.

**HTTP**: The [HTTP Data Connector](https://spiceai.org/docs/components/data-connectors/https) now bounds its response cache. The cache kept every fetched response for the life of the process. An API-proxy workload with request-shaped keys therefore grew by about 1 MB per distinct request.

The cache now holds at most `response_cache_max_size_bytes` per dataset, 64 MiB by default, and evicts the oldest entries first. It expires an entry by the origin's `max-age` and `Age` headers, and it refuses to store a response that sends `no-store` or `no-cache`. The new `response_cache_fallback_ttl` setting keeps a response from an origin that sends no `Cache-Control` header at all. Two new metrics, `response_cache_size_bytes` and `response_cache_items_count`, report occupancy per dataset.

```yaml
datasets:
  - from: https://api.example.com/v1/items
    name: items
    params:
      response_cache_max_size_bytes: 16777216
      response_cache_fallback_ttl: 5m
```

### More Relevant Vector Search with HuggingFace Embeddings

[Vector search](https://spiceai.org/docs/features/search/vector-search) with a [HuggingFace](https://spiceai.org/docs/components/embeddings/huggingface) sentence-transformer embedding model, such as `all-MiniLM-L6-v2`, now returns relevant results. The runtime kept the fixed padding that some models declare in `tokenizer.json`, so it padded every input to 128 tokens. The padding then dominated the vector of a short input. The runtime now clears that padding and lets the embedding layer own the attention mask. On the MTEB SciFact benchmark with `all-MiniLM-L6-v2`, nDCG@10 rises from 0.018 to 0.640 (published: 0.645). Full-text search and `model2vec` static models were not affected.

A search filter on a timezone-aware timestamp column now matches the exact instant when the runtime pushes it down to a [DuckDB-backed search index](https://spiceai.org/docs/components/vectors/duckdb). The filter compared whole milliseconds, so a filter inside a millisecond returned different rows than requested.

### Change Data Capture and Write-Back

- **PostgreSQL write-back echo suppression**: A CDC-accelerated dataset that writes back to its [PostgreSQL](https://spiceai.org/docs/features/cdc/postgres-replication) source now applies each write once. The logical-replication stream returns every write-back transaction as a change. Spice now records the transaction ID of each delivery and drops the matching changes from the stream. Spice drops a replay after a slot reconnect the same way.
- **Write-back for `BIGINT` primary keys**: [Durable write-back](https://spiceai.org/docs/components/data-accelerators/cayenne) now delivers for a table whose primary key is a single `BIGINT` column. Such a table acknowledged writes but never delivered them to the source.
- **MySQL rebuild after a purged binlog**: A [MySQL](https://spiceai.org/docs/features/cdc/mysql-replication) acceleration with `mysql_replication_invalid_checkpoint_behavior: restart` now rebuilds when the source has purged its binlog position, even when `mysql_replication_initial_snapshot: disabled` is set. It previously resumed from the current head and kept the rows that the source had deleted in the gap.
- **Append refresh on Cayenne**: An [append refresh](https://spiceai.org/docs/features/data-acceleration/refresh-modes/append) on a Cayenne dataset resumes correctly again. The runtime read the append watermark through a timestamp cast that the Cayenne scan could not evaluate. The read failed, and the dataset stopped appending. The runtime now reads the raw column and normalizes the value in memory.

### Other Improvements

- **Startup logs**: The log events that the runtime emits while it builds now reach the configured log output. These include the DuckDB memory budget and the query memory-limit defaults. Earlier releases dropped them at every log level.
- **Spice Cloud CLI**: The CLI now reconciles its stored credential state and active organization for [Spice Cloud](https://spiceai.org/docs/cli/reference/cloud) after login.
- **Cayenne write concurrency**: A Cayenne write that requires a single writer now runs serially even when [`cayenne_write_concurrency`](https://spiceai.org/docs/components/data-accelerators/cayenne) is raised.

## Contributors

- [@bjchambers](https://github.com/bjchambers)
- [@Jeadie](https://github.com/Jeadie)
- [@krinart](https://github.com/krinart)
- [@lukekim](https://github.com/lukekim)
- [@peasee](https://github.com/peasee)
- [@phillipleblanc](https://github.com/phillipleblanc)
- [@sgrebnov](https://github.com/sgrebnov)

## Breaking Changes

No breaking changes.

One behavior change to note before you upgrade:

- **[DuckDB](https://spiceai.org/docs/components/data-accelerators/duckdb) moves from v1.5.5 to v1.4.4** to keep the memory of DuckDB-accelerated datasets bounded under upsert refreshes ([duckdb/duckdb#25162](https://github.com/duckdb/duckdb/issues/25162)). SQL syntax and functions that DuckDB added in v1.5 are not available to DuckDB-accelerated datasets.

## Cookbook Updates

No new cookbook recipes.

The [Spice Cookbook](https://spiceai.org/cookbook) includes more than 104 recipes to help you get started with Spice quickly and easily.

## Upgrading

To upgrade to v2.2.1, use one of the following methods:

**CLI**:

```console
spice upgrade
```

**Homebrew**:

```console
brew upgrade spiceai/spiceai/spice
```

**Docker**:

Pull the `spiceai/spiceai:2.2.1` image:

```console
docker pull spiceai/spiceai:2.2.1
```

For available tags, see [DockerHub](https://hub.docker.com/r/spiceai/spiceai/tags).

**Helm**:

```console
helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.2.1
```

**AWS Marketplace**:

Spice is available in the [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-jmf6jskjvnq7i).

## What's Changed

### Changelog

- Make a pinned single partition actually mean one writer by [@lukekim](https://github.com/lukekim) in [#13325](https://github.com/spiceai/spiceai/pull/13325)
- feat(postgres): outstanding write-back xid registry for CDC echo suppression by [@Jeadie](https://github.com/Jeadie) in [#13350](https://github.com/spiceai/spiceai/pull/13350)
- fix(llms): clear tokenizer padding/truncation so TEI owns masking by [@Jeadie](https://github.com/Jeadie) in [#13416](https://github.com/spiceai/spiceai/pull/13416)
- fix(http): bound the connector's response cache and honour `no-store` (fixes #13460) by [@bjchambers](https://github.com/bjchambers) in [#13462](https://github.com/spiceai/spiceai/pull/13462)
- fix(search): render a DuckDB search filter against the table schema (fixes #13144) by [@claudespice](https://github.com/claudespice) in [#13489](https://github.com/spiceai/spiceai/pull/13489)
- fix(duckdb): bump datafusion-table-providers for the attachment-race fixes by [@sgrebnov](https://github.com/sgrebnov) in [#13491](https://github.com/spiceai/spiceai/pull/13491)
- fix(cli): reconcile Spice Cloud credential state by [@phillipleblanc](https://github.com/phillipleblanc) in [#13500](https://github.com/spiceai/spiceai/pull/13500)
- fix(mysql): rebuild a purged-position acceleration whatever the snapshot mode says (fixes #13024) by [@claudespice](https://github.com/claudespice) in [#13501](https://github.com/spiceai/spiceai/pull/13501)
- fix(cayenne): decode write-back markers for a single Int64 primary key by [@phillipleblanc](https://github.com/phillipleblanc) in [#13502](https://github.com/spiceai/spiceai/pull/13502)
- fix(deps): bump DataFusion to pick up the join schema-metadata fix by [@krinart](https://github.com/krinart) in [#13526](https://github.com/spiceai/spiceai/pull/13526)
- fix(cayenne): give each cayenne_catalog test a table root of its own (fixes #13527) by [@claudespice](https://github.com/claudespice) in [#13535](https://github.com/spiceai/spiceai/pull/13535)
- fix(acceleration): read the append watermark without casting the time column by [@sgrebnov](https://github.com/sgrebnov) in [#13539](https://github.com/spiceai/spiceai/pull/13539)
- test(postgres): cover CDC echo suppression through a real delivery transaction by [@phillipleblanc](https://github.com/phillipleblanc) in [#13553](https://github.com/spiceai/spiceai/pull/13553)
- test(federation): guard the derived-table output names a pin bump landed unguarded (refs #12751) by [@claudespice](https://github.com/claudespice) in [#13555](https://github.com/spiceai/spiceai/pull/13555)
- fix(runtime): carry a scan's constant columns across the schema cast by [@phillipleblanc](https://github.com/phillipleblanc) in [#13565](https://github.com/spiceai/spiceai/pull/13565)
- fix(cayenne): store Arrow Map columns in a file-mode accelerator (fixes #13524) by [@bjchambers](https://github.com/bjchambers) in [#13567](https://github.com/spiceai/spiceai/pull/13567)
- fix(cayenne): stop a predicate DELETE destroying live rows on an upsert table by [@lukekim](https://github.com/lukekim) in [#13574](https://github.com/spiceai/spiceai/pull/13574)
- fix(logging): deliver the runtime build's events to the installed subscriber by [@sgrebnov](https://github.com/sgrebnov) in [#13587](https://github.com/spiceai/spiceai/pull/13587)
- fix(adbc): restore the federation deny-list and query_federation wiring by [@peasee](https://github.com/peasee) in [#13590](https://github.com/spiceai/spiceai/pull/13590)
- feat(cayenne): observability for maintenance decisions and footprint growth by [@bjchambers](https://github.com/bjchambers) in [#13619](https://github.com/spiceai/spiceai/pull/13619)
- test(forks): guard the Spice patches carried on our upstream forks by [@bjchambers](https://github.com/bjchambers) in [#13623](https://github.com/spiceai/spiceai/pull/13623)
- test(cayenne): settle the tier before asserting why a subset pass declined by [@lukekim](https://github.com/lukekim) in [#13632](https://github.com/spiceai/spiceai/pull/13632)
- fix(cayenne): close the PK-index checkout window on every exit path (fixes #13267) by [@claudespice](https://github.com/claudespice) in [#13641](https://github.com/spiceai/spiceai/pull/13641)
- fix(cayenne): reclaim inline tombstones, and return the freed metastore pages by [@lukekim](https://github.com/lukekim) in [#13651](https://github.com/spiceai/spiceai/pull/13651)
- fix(cayenne): delete merged-away manifest rows at protected-snapshot compaction by [@Jeadie](https://github.com/Jeadie) in [#13658](https://github.com/spiceai/spiceai/pull/13658)
- fix(cayenne): prune old-snapshot catalog rows at subset small-file compaction by [@Jeadie](https://github.com/Jeadie) in [#13659](https://github.com/spiceai/spiceai/pull/13659)
- fix(cayenne): delete emptied protected snapshot manifest rows in retention cleanup by [@Jeadie](https://github.com/Jeadie) in [#13660](https://github.com/spiceai/spiceai/pull/13660)
- fix(cayenne): schedule the orphaned deletion-vector sweep from every publication that orphans one by [@phillipleblanc](https://github.com/phillipleblanc) in [#13663](https://github.com/spiceai/spiceai/pull/13663)
- fix(cayenne): reclaim superseded snapshot dirs against live state by [@peasee](https://github.com/peasee) in [#13670](https://github.com/spiceai/spiceai/pull/13670)
- Translate the JSON extraction functions into BigQuery SQL, and read negative JSON numbers by [@phillipleblanc](https://github.com/phillipleblanc) in [#13672](https://github.com/spiceai/spiceai/pull/13672)
- test(forks): guard the four BigQuery dialect unparser fixes the datafusion pin carries by [@lukekim](https://github.com/lukekim) in [#13691](https://github.com/spiceai/spiceai/pull/13691)
- fix(cayenne): claim a commit's live-row delta before it publishes the rows (fixes #13036) by [@claudespice](https://github.com/claudespice) in [#13720](https://github.com/spiceai/spiceai/pull/13720)
- Downgrade DuckDB to v1.4.4 by [@sgrebnov](https://github.com/sgrebnov) in [#13742](https://github.com/spiceai/spiceai/pull/13742)
- fix(vortex): convert a date statistic into a timestamp before pruning on it (fixes #13624) by [@lukekim](https://github.com/lukekim) in [#13745](https://github.com/spiceai/spiceai/pull/13745)
- fix(cayenne): resolve now() in retention predicates once per pass by [@lukekim](https://github.com/lukekim) in [#13769](https://github.com/spiceai/spiceai/pull/13769)

**Full Changelog**: <https://github.com/spiceai/spiceai/compare/v2.2.0...v2.2.1>