v1.0.0-rc1
juicedata/juicefsv1.0.0-rc1Jun 15, 2022by davies
AI Summary
The first release candidate for v1.0.0, featuring massive performance improvements in metadata dumping/loading and the introduction of SQL and TiKV as metadata engines.
Key Highlights
- Massive reduction in metadata dump memory usage (below 1/20)
- Stream loading of metadata to reduce memory spikes
- Improved SQL engine stability under heavy workload
- Added `juicefs objbench` command for object storage testing
- Added `juicefs version` command
New Features
- SQL and TiKV metadata engines
- Pyroscope profiling support
- Path finding for inodes via `juicefs info`
- Progress bar for `juicefs rmr` and `juicefs warmup`
- Object storage benchmarking tool
Full Release Notes
JuiceFS v1.0.0-rc1 is the first release candidate for v1.0. It has 184 commits from 17 contributors, thanks to @davies @zhoucheng361 @SandyXSD @zhijian-pro @sanwan @xiaogaozi @tangyoupeng @solracsf @showjason @rayw000 @AIXjing @helix-loop @Suave @zhouaoe @chnliyong @yuhr123 @liufuyang ! # Highlights - Dumping metadata from Redis has been improved, massively reducing the memory required to below 1/20 of that before. It also relieves the memory spike of a client when doing metadata backup. Dumping metadata from SQL and TiKV within a single transaction to ensure consistency. - Loading metadata to all engines has been improved as well. Instead of loading the whole dumped file in one step, JuiceFS will now read it in a stream, and simultaneously import metadata to the engine. This saves a lot of memory when the dumped file is huge. - Improved stability for SQL engine under heavy workload. - Added a new command `juicefs objbench` that can be used to run basic function tests and benchmarks on object storage, making sure it works as expected. # New - Supports using SQL databases, etcd as data storage (#2003, #2009) - Supports finding all paths of an inode with `juicefs info` command (#2058, #2161, #2193) - Supports using Pyroscope to record JuiceFS profiling (#1952) - Added progress bar for `juicefs rmr` and `juicefs warmup` commands (#2197) - Added a new command `juicefs objbench` to run basic benchmarks on object storage (#2055, #2091) - Added a new command `juicefs version` to print version, as an alternative to `--version` (#2229) # Changed - cmd: check the range of parameters (#2195) - cmd: eliminate panic which is triggered by missing argument (#2183) - cmd/mount: warn about the behavior of mounting the same directory multiple times (#2141) - cmd/warmup: support warmup from inside of a container (#2056) - meta: add delayed slice only when chunkid > 0 (#2231) - meta: speed up and reduce memory for loading metadata (#2142, #2148) - meta: add a pessimistic lock to reduce conflicted transactions in the database (#2111) - meta: limit the number of scanned files in cleanup (#2157) - meta: limit number of files when cleanup trash (#2061) - meta: limit the number of coroutines to delete file data (#2042) - meta: log last error if a transaction has been ever restarted (#2172) - meta: log session info when cleaning up a stale one (#2045) - meta: skip updating mtime/ctime of the parent if it's updated recently (#1960) - meta/redis: check config 'maxmemory-policy' (#2059) - meta/redis: Speedup dump for Redis and reduce memory usage (#2156) - meta/tkv: Speedup dump for kv storage (#2140) - meta/tkv: dump metadata using snapshot (#1961) - meta/tkv: use scanRange to get delayed slices (#2057) - meta/sql: dump metadata in a single transaction (#2131) - chunk/store: keep cache after uploading staging blocks (#2168) - object: reload the configuration for data storage (#1995) - object/sftp: load default private keys for sftp (#2014) - object/oss: add user agent for oss #1992 (#1993) - vfs: hide .control from readdir (#1998) - gateway: clean up expired temporary files (#2082) - SDK: package amd64 and arm64 libjfs (#2198) - SDK: don't reuse fd in Java SDK (#2122) - Dependency: upgrade coredns for CVE-2019-19794 (#2190) - Dependency: upgrade azblob sdk (#1962) - meta: keep valid utf8 in dumped JSON (#1973) - SDK: mvn shade some dependency to avoid class conflict (#2035) - meta: truncate trash entry name if it's too long (#2049) - meta/sql: use repeatable-read for transaction (#2128) # Bugfix - cmd: fix not showing arguments for commands without META-URL (#2158) - cmd/sync: fix sync lost file (#2106) - cmd/warmup: fix warmup on read-only mount point (#2108) - meta: skip updating sliceRef if id is 0 (#2096) - meta: fix update xattr with the same value (#2078) - meta/redis: handle lua result from Redis v7.0+ (#2221) - meta/sql: fix dump with more than 10000 files (#2134) - meta/sql: one transaction in SQLite at a time (#2126) - meta/sql: fix rename with Postgres with repeatable read (#2109) - meta/sql: fix deadlock in PG (#2104) - meta/sql: ignore error about duplicated index (#2087) - meta/sql: read database inside transaction (#2073, #2086) - meta/sql: retry transaction on duplicated entry and concurrent update (#2077) - meta/sql: fix the deadlock in rename (#2067) - meta/sql: retry transaction for duplicate key in PG (#2071) - meta/sql: fix update query in SQL transaction (#2024) - meta/tkv: fix value of delSliceKey (#2054) - meta/tkv: upgrade TiKV client to 2.0.1 to fix nil connection (#2050) - chunk/store: fix stats of cached space in writeback mode (#2227) - object: delete should be idempotent (#2034) - object/file: Head of file should return File (#2133) - object/s3: check prefix and marker with returned keys from S3 (#2040) - object/prefix: fix with prefix returning nil error for unsupported ops (#2021) - object/sftp: fix auth of sftp with multiple keys (#2186) - object/sftp: fix prefix of sftp, support ssh-agent (#1954) - vfs/backup: skip cleanup if list failed (#2044) - SDK: handle atomic rename exception (#2192)