4.42

## What's Changed

## Master and topology

Heartbeat protocol, volume registration, and a long series of allocation/copy removals on the master's hot paths.

- topology: digest the volumes a master believes each node holds ([#10619](https://github.com/seaweedfs/seaweedfs/pull/10619))
- heartbeat: carry a volume digest and verify it ([#10627](https://github.com/seaweedfs/seaweedfs/pull/10627))
- heartbeat: send only the volumes that changed ([#10640](https://github.com/seaweedfs/seaweedfs/pull/10640))
- heartbeat: keep the master current through collection churn ([#10657](https://github.com/seaweedfs/seaweedfs/pull/10657))
- topology: follow a volume that moved between a server's disks ([#10628](https://github.com/seaweedfs/seaweedfs/pull/10628))
- master: keep disk_id when registering volumes from incremental heartbeats ([#10686](https://github.com/seaweedfs/seaweedfs/pull/10686))
- topology: provisional volume update must not erase the reported disk id ([#10687](https://github.com/seaweedfs/seaweedfs/pull/10687))
- topology: keep per-node volume state with the location it describes ([#10654](https://github.com/seaweedfs/seaweedfs/pull/10654))
- topology: mark a volume crowded only if it can take writes ([#10655](https://github.com/seaweedfs/seaweedfs/pull/10655))
- topology: track volume size only where writes can land ([#10653](https://github.com/seaweedfs/seaweedfs/pull/10653))
- topology: clamp the deleted-vs-total subtractions in volume stats ([#10633](https://github.com/seaweedfs/seaweedfs/pull/10633))
- master: keep a racing registration out of a dying collection ([#10677](https://github.com/seaweedfs/seaweedfs/pull/10677))
- master: align default volume size with EC rows ([#10761](https://github.com/seaweedfs/seaweedfs/pull/10761))
- master: let VolumeList ask for the volumes it wants ([#10674](https://github.com/seaweedfs/seaweedfs/pull/10674))
- master: stream volume listings ([#10676](https://github.com/seaweedfs/seaweedfs/pull/10676))
- master: stop copying the cluster to name it ([#10700](https://github.com/seaweedfs/seaweedfs/pull/10700))
- topology: keep the volume map's values out of its slots ([#10680](https://github.com/seaweedfs/seaweedfs/pull/10680))
- topology: build the volume list without copying the volume map first ([#10668](https://github.com/seaweedfs/seaweedfs/pull/10668))
- topology: diff a heartbeat without copying the volume map ([#10608](https://github.com/seaweedfs/seaweedfs/pull/10608))
- topology: preallocate the heartbeat volume conversion slice ([#10607](https://github.com/seaweedfs/seaweedfs/pull/10607))
- topology: preallocate the disk volume snapshot ([#10609](https://github.com/seaweedfs/seaweedfs/pull/10609))
- topology: size the new-volume list from the actual delta ([#10613](https://github.com/seaweedfs/seaweedfs/pull/10613))
- topology: preallocate the per-disk VolumeList payload ([#10614](https://github.com/seaweedfs/seaweedfs/pull/10614))
- topology: preallocate the client-facing topology snapshots ([#10615](https://github.com/seaweedfs/seaweedfs/pull/10615))
- topology: gather a node's volumes into one slice ([#10617](https://github.com/seaweedfs/seaweedfs/pull/10617))
- topology: log writable-state changes, not every check ([#10612](https://github.com/seaweedfs/seaweedfs/pull/10612))
- topology: drop the unused DataNode volume id listing ([#10618](https://github.com/seaweedfs/seaweedfs/pull/10618))

## Volume server and storage

- storage: fix corrupted leveldb detection in DoOffsetLoading ([#10650](https://github.com/seaweedfs/seaweedfs/pull/10650))
- storage: count a volume's needles in uint32 ([#10718](https://github.com/seaweedfs/seaweedfs/pull/10718))
- storage: share the volume strings a cluster repeats ([#10665](https://github.com/seaweedfs/seaweedfs/pull/10665))
- storage: stop keeping the remote storage key on the master ([#10672](https://github.com/seaweedfs/seaweedfs/pull/10672))
- storage: order VolumeInfo by alignment ([#10669](https://github.com/seaweedfs/seaweedfs/pull/10669))
- needle: intern the stored ttl values ([#10611](https://github.com/seaweedfs/seaweedfs/pull/10611))
- super_block: intern the byte-encoded replica placements ([#10610](https://github.com/seaweedfs/seaweedfs/pull/10610))
- throughput limits for replicate, EC shard, and worker-driven moves ([#10749](https://github.com/seaweedfs/seaweedfs/pull/10749))

## Erasure coding

Interrupted-operation cleanup is the theme: encode, decode, and shard moves now roll back or resume into a consistent state, and a chaos harness exercises it.

- refactor: extract EC orchestration into a shared weed/ec package ([#10760](https://github.com/seaweedfs/seaweedfs/pull/10760))
- refactor: share volume and EC shard move logic between shell and workers ([#10727](https://github.com/seaweedfs/seaweedfs/pull/10727))
- erasure_coding: share the EC shard teardown primitive ([#10740](https://github.com/seaweedfs/seaweedfs/pull/10740))
- ec.encode: roll back a failed encode instead of leaving readonly volumes and orphan shards ([#10741](https://github.com/seaweedfs/seaweedfs/pull/10741))
- ec.encode: require the shards to agree on size before deleting the volume ([#10769](https://github.com/seaweedfs/seaweedfs/pull/10769))
- ec.decode: finish the cleanup an interrupted decode left behind ([#10767](https://github.com/seaweedfs/seaweedfs/pull/10767))
- ec.decode: check the rebuilt .dat is complete before the shards can be deleted ([#10768](https://github.com/seaweedfs/seaweedfs/pull/10768))
- EC worker: clear stale/interrupted shards at task start and on failure ([#10738](https://github.com/seaweedfs/seaweedfs/pull/10738))
- ec: confirm a surviving copy before deleting a duplicate EC shard ([#10719](https://github.com/seaweedfs/seaweedfs/pull/10719))
- EC: handle zero-sized shard files uniformly across moves, rebuilds, and startup cleanup ([#10753](https://github.com/seaweedfs/seaweedfs/pull/10753))
- ec.balance: add a -volumeIds filter ([#10667](https://github.com/seaweedfs/seaweedfs/pull/10667))

## Filer

- filer: list directories without decoding chunk lists ([#10616](https://github.com/seaweedfs/seaweedfs/pull/10616))
- filer: do not sweep children when deleting a folder non-recursively ([#10782](https://github.com/seaweedfs/seaweedfs/pull/10782))
- filer: restore a folder that received an entry while it was deleted ([#10783](https://github.com/seaweedfs/seaweedfs/pull/10783))
- filer: rebuild peer metadata subscriptions after a master reconnect ([#10648](https://github.com/seaweedfs/seaweedfs/pull/10648))
- filer: retry and surface metadata replay failures from peers ([#10714](https://github.com/seaweedfs/seaweedfs/pull/10714))
- filer: drain pending log chunk refs when the metadata stream ends ([#10647](https://github.com/seaweedfs/seaweedfs/pull/10647))
- filer: add filer.meta.scan to audit one directory's change history ([#10645](https://github.com/seaweedfs/seaweedfs/pull/10645))
- reload filer config on local metadata updates ([#10622](https://github.com/seaweedfs/seaweedfs/pull/10622))
- log_buffer: end bounded reads that find the buffer empty ([#10750](https://github.com/seaweedfs/seaweedfs/pull/10750))

### TUS

- configurable TUS max upload size and session expiry ([#10638](https://github.com/seaweedfs/seaweedfs/pull/10638))
- TUS concatenation extension ([#10702](https://github.com/seaweedfs/seaweedfs/pull/10702))

### Filer stores (redis)

- redis2: remove orphaned directory index members on listing ([#10735](https://github.com/seaweedfs/seaweedfs/pull/10735))
- redis: remove orphaned directory index members on listing ([#10742](https://github.com/seaweedfs/seaweedfs/pull/10742))
- redis2: harden the orphaned index member cleanup ([#10743](https://github.com/seaweedfs/seaweedfs/pull/10743))
- redis2: expire entries without destroying a concurrent recreate ([#10744](https://github.com/seaweedfs/seaweedfs/pull/10744))
- redis2: orphan cleanup existence checks must not read replicas ([#10745](https://github.com/seaweedfs/seaweedfs/pull/10745))

## Mount

- mount: stop churning the inode table on every readdir ([#10606](https://github.com/seaweedfs/seaweedfs/pull/10606))
- mount: fix four readdir pagination bugs ([#10624](https://github.com/seaweedfs/seaweedfs/pull/10624))
- mount: read oversized directories through instead of caching them ([#10631](https://github.com/seaweedfs/seaweedfs/pull/10631))
- mount: let the kernel cache directory listings ([#10634](https://github.com/seaweedfs/seaweedfs/pull/10634))
- mount: invalidate hot directory listings by section ([#10712](https://github.com/seaweedfs/seaweedfs/pull/10712))
- windows mount: cache file data, resolved paths and attributes ([#10703](https://github.com/seaweedfs/seaweedfs/pull/10703))

## S3 API

- s3: add the RenameObject endpoint ([#10659](https://github.com/seaweedfs/seaweedfs/pull/10659))
- s3api: fix ListObjectsV2 dropping objects under a partial prefix ([#10656](https://github.com/seaweedfs/seaweedfs/pull/10656))
- s3: a key deleted after enabling versioning must leave the listing ([#10684](https://github.com/seaweedfs/seaweedfs/pull/10684))
- s3: take bucket sizes from the master's summary ([#10664](https://github.com/seaweedfs/seaweedfs/pull/10664))
- s3: option to disable bucket auto-creation on upload ([#10759](https://github.com/seaweedfs/seaweedfs/pull/10759))
- s3api/policy_engine: prune dead code ([#10599](https://github.com/seaweedfs/seaweedfs/pull/10599))

## Iceberg and S3 Tables

- iceberg: make a table commit a compare-and-swap ([#10775](https://github.com/seaweedfs/seaweedfs/pull/10775))
- iceberg: add view rename, scan-report and snapshots=refs to the catalog ([#10776](https://github.com/seaweedfs/seaweedfs/pull/10776))
- iceberg: vend table-scoped credentials to clients that ask for delegation ([#10777](https://github.com/seaweedfs/seaweedfs/pull/10777))
- iceberg: repair non-compliant manifests at commit ([#10641](https://github.com/seaweedfs/seaweedfs/pull/10641))
- iceberg: route unprefixed requests to the first table bucket ([#10675](https://github.com/seaweedfs/seaweedfs/pull/10675))
- iceberg maintenance: keep the snapshots that branches and tags pin ([#10774](https://github.com/seaweedfs/seaweedfs/pull/10774))
- iceberg: let table properties override the worker config ([#10772](https://github.com/seaweedfs/seaweedfs/pull/10772))
- iceberg: accept lowercase parquet file format when planning compaction ([#10751](https://github.com/seaweedfs/seaweedfs/pull/10751))
- s3tables: add the maintenance configuration APIs ([#10773](https://github.com/seaweedfs/seaweedfs/pull/10773))
- iceberg maintenance: fix the test build master merged broken ([#10780](https://github.com/seaweedfs/seaweedfs/pull/10780))

## Shell

- shell: volume.delete and volume.move accept a -timeout ([#10701](https://github.com/seaweedfs/seaweedfs/pull/10701))
- shell: volume.move cleans up when aborted after the copy phase ([#10704](https://github.com/seaweedfs/seaweedfs/pull/10704))
- shell: keep the source readonly when the incomplete target copy cannot be deleted ([#10705](https://github.com/seaweedfs/seaweedfs/pull/10705))
- shell: multi-target fs.mergeVolumes and volume.mark -readonlyCanDelete ([#10706](https://github.com/seaweedfs/seaweedfs/pull/10706))
- shell: parallelize volume balance moves ([#10737](https://github.com/seaweedfs/seaweedfs/pull/10737))
- shell: send read jwt when downloading chunks in fs.mergeVolumes and fs.distributeChunks ([#10717](https://github.com/seaweedfs/seaweedfs/pull/10717))

## Admin and workers

- admin: dashboard counts chunks, not files ([#10598](https://github.com/seaweedfs/seaweedfs/pull/10598))
- admin: show capacity per storage tier and stop counting remote-tiered bytes as local disk usage ([#10766](https://github.com/seaweedfs/seaweedfs/pull/10766))
- admin: make paths relative ([#10709](https://github.com/seaweedfs/seaweedfs/pull/10709))
- plugin scheduler: drain started jobs past the window close instead of killing them ([#10728](https://github.com/seaweedfs/seaweedfs/pull/10728))

## Remote storage and tiering

- read cold remote objects straight from the origin while caching ([#10731](https://github.com/seaweedfs/seaweedfs/pull/10731))
- remote_storage: build all S3-compatible clients through one constructor ([#10720](https://github.com/seaweedfs/seaweedfs/pull/10720))

## Security

- master: gate raft membership RPCs behind the admin whitelist ([#10649](https://github.com/seaweedfs/seaweedfs/pull/10649))
- volume: validate the file extension in CopyFile and ReceiveFile ([#10644](https://github.com/seaweedfs/seaweedfs/pull/10644))
- volume: decode IPv6 transition addresses in the remote-endpoint guard ([#10683](https://github.com/seaweedfs/seaweedfs/pull/10683))
- volume: extend the remote-endpoint guard to the azure backend ([#10754](https://github.com/seaweedfs/seaweedfs/pull/10754))
- volume: validate replica targets and restrict gcs credentials in FetchAndWriteNeedle ([#10755](https://github.com/seaweedfs/seaweedfs/pull/10755))
- sftp: url-encode the upload path ([#10758](https://github.com/seaweedfs/seaweedfs/pull/10758))
- grpc: optional client_cert/client_key for outgoing mTLS connections ([#10747](https://github.com/seaweedfs/seaweedfs/pull/10747))
- rust volume: mirror Go volume server logic to gate the admin RPCs ([#10748](https://github.com/seaweedfs/seaweedfs/pull/10748))

## Rust volume server

- rust: stop quarantining v2 volumes, load a disk's volumes concurrently ([#10602](https://github.com/seaweedfs/seaweedfs/pull/10602))
- rust volume: mirror the VolumeConsolidateIndex RPC from Go ([#10752](https://github.com/seaweedfs/seaweedfs/pull/10752))
- rust volume: mirror Go volume server logic to gate the admin RPCs ([#10748](https://github.com/seaweedfs/seaweedfs/pull/10748))

## Clients

- wdclient: keep the location of a volume reported added and removed at once ([#10635](https://github.com/seaweedfs/seaweedfs/pull/10635))
- clients: stream the volume listings that ask for everything ([#10679](https://github.com/seaweedfs/seaweedfs/pull/10679))

## Telemetry

- telemetry: version distribution over time ([#10625](https://github.com/seaweedfs/seaweedfs/pull/10625))
- telemetry: put the version pie back beside the stacked chart ([#10626](https://github.com/seaweedfs/seaweedfs/pull/10626))
- telemetry: count erasure-coded volumes in the reported totals ([#10632](https://github.com/seaweedfs/seaweedfs/pull/10632))

## Helm and deployment

- helm: mark filer db-init ConfigMap volume optional ([#10596](https://github.com/seaweedfs/seaweedfs/pull/10596))
- helm: enterprise license Secret, and a persistent-claim option for master data ([#10601](https://github.com/seaweedfs/seaweedfs/pull/10601))
- helm: add support for schedulerName ([#10716](https://github.com/seaweedfs/seaweedfs/pull/10716))

## Tests and CI

- test: EC lifecycle chaos harness, with four fixes it found ([#10763](https://github.com/seaweedfs/seaweedfs/pull/10763))
- test: systematic EC interruption verification, exhaustive model check plus deterministic kill matrix ([#10764](https://github.com/seaweedfs/seaweedfs/pull/10764))
- test: re-lock and retry every chaos command, not just the balance ([#10770](https://github.com/seaweedfs/seaweedfs/pull/10770))
- test: walk an EC volume through a multi-disk node's whole life ([#10721](https://github.com/seaweedfs/seaweedfs/pull/10721))
- test: drive ec.encode/balance/rebuild E2E with a byte-identical payload check ([#10722](https://github.com/seaweedfs/seaweedfs/pull/10722))
- test: assert EC shard identity and empty-view in multi-disk lifecycle tests ([#10723](https://github.com/seaweedfs/seaweedfs/pull/10723))
- s3tables: add ClickHouse iceberg catalog integration test ([#10637](https://github.com/seaweedfs/seaweedfs/pull/10637))
- clickhouse catalog test: cover latest ClickHouse and catalog-side CREATE TABLE ([#10707](https://github.com/seaweedfs/seaweedfs/pull/10707))
- ci: run the gated redis store tests ([#10746](https://github.com/seaweedfs/seaweedfs/pull/10746))
- test: let the vacuum shell session outlive the vacuum ([#10682](https://github.com/seaweedfs/seaweedfs/pull/10682))
- mount: absorb the WinFsp metadata cache window in the concurrent-reader test ([#10636](https://github.com/seaweedfs/seaweedfs/pull/10636))
- skiplist: fix TestFindGreaterOrEqual flake, compare against largest key, not value ([#10757](https://github.com/seaweedfs/seaweedfs/pull/10757))

## Docs

- docs: update example link ([#10725](https://github.com/seaweedfs/seaweedfs/pull/10725))

## Dependencies

- bump github/codeql-action from 4.37.4 to 4.37.6 ([#10699](https://github.com/seaweedfs/seaweedfs/pull/10699))
- bump github.com/aws/aws-sdk-go-v2/credentials from 1.19.32 to 1.19.34 ([#10698](https://github.com/seaweedfs/seaweedfs/pull/10698))
- bump cloud.google.com/go/kms from 1.31.0 to 1.33.0 ([#10697](https://github.com/seaweedfs/seaweedfs/pull/10697))
- bump modernc.org/sqlite from 1.53.0 to 1.56.0 ([#10696](https://github.com/seaweedfs/seaweedfs/pull/10696))
- bump github.com/pierrec/lz4/v4 from 4.1.27 to 4.1.28 ([#10695](https://github.com/seaweedfs/seaweedfs/pull/10695))
- bump github.com/ydb-platform/ydb-go-sdk/v3 from 3.146.3 to 3.147.1 ([#10694](https://github.com/seaweedfs/seaweedfs/pull/10694))

## New Contributors
* @andrei-brindas97 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/10596
* @hsdfat made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/10656
* @ruriww made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/10709
* @cavus700 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/10716
* @aleksandr4842 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/10725
* @rooseveltlai made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/10761

**Full Changelog**: https://github.com/seaweedfs/seaweedfs/compare/4.41...4.42