v1.0.0-beta3

juicedata/juicefsv1.0.0-beta3May 5, 2022by SandyXSD

AI Summary

The third beta release for v1.0.0, introducing etcd support, Redis Cluster compatibility, and significant changes to session management and trash handling.

Key Highlights

  • Supports etcd as a new metadata engine
  • Supports Redis Cluster and compatible services (Amazon MemoryDB)
  • New session management format (old clients cannot detect sessions)
  • Compacted slices are kept for `trash-days` if trash is enabled
  • Improved `juicefs sync` command

Breaking Changes

  • Old clients are unable to detect sessions with version 1.0.0-beta3 or higher via `juicefs status` or `juicefs destroy` command

New Features

  • etcd metadata engine support
  • Redis Cluster support
  • Non-UTF8 file name handling for SQL engines
  • TLS support for TiKV and object storage
  • Oracle Object Storage support
  • Umask setting for S3 Gateway
  • Graphite metrics for Java SDK
  • Hashed prefix for objects
  • Customized storage type option

Full Release Notes

JuiceFS v1.0.0-beta3 is the third beta release for v1.0. It has 247 commits from 22 contributors, thanks to @SandyXSD @zhoucheng361 @davies @zhijian-pro @yuhr123 @sanwan @AIXjing @rayw000 @xiaogaozi @Suave @showjason @tangyoupeng @201341 @solracsf @guo-sj @chnliyong @DeanThompson @zwwhdls @wph95 @lidaohang @sjp00556 @DEvmIb !

# Highlights

- Supports [etcd](https://etcd.io/) as a new metadata engine. It can be a handy choice when you only need a small volume but cares more about the data availability and persistence.
- Supports Redis Cluster and other compatible services (Amazon MemoryDB for Redis) as metadata engines.
- When using SQL metadata engines, file names not encoded by UTF-8 can now be properly handled after manual modification to the table schema, see [details](https://juicefs.com/docs/community/release_notes/#upgrade-to-juicefs-v100-beta3).
- A new session management format is introduced. Old clients are unable to detect sessions with version 1.0.0-beta3 or higher via `juicefs status` or `juicefs destroy` command, see [details](https://juicefs.com/docs/community/release_notes/#upgrade-to-juicefs-v100-beta3).
- If trash is enabled, compacted slices are kept as well in case they are needed to recover file data. These slices will be cleaned up automatically after `trash-days`, and can be deleted manually via `juicefs gc` command.
- A lot of improvements have been made to `juicefs sync` command.
- A lot of protection checks against unintentional misuse have been added.

# New

- Supports etcd as metadata engine (#1638)
- Supports Redis in cluster mode while using one slot for each file system (#1696)
- Supports handling file names not encoded by UTF-8 for SQL metadata engines (#1762)
- Supports TLS when using TiKV as metadata engine or object storage (#1653, #1778)
- Supports Oracle Object Storage as data storage (#1516)
- Supports setting umask for S3 Gateway (#1537)
- Java SDK now supports pushing metrics to Graphite (#1586)
- Added a new option `--heartbeat` for the `mount` command to adjust heartbeat interval (#1591, #1865)
- Added many improvements for `sync` command to make it more handy (#1554, #1619, #1651, #1836, #1897, #1901)
- Added a new option `--hash-prefix` for the `format` command to add a hashed prefix for objects (#1657)
- Added a new client option `--storage` to allow customized storage type (#1912)

# Changed

- compacted slices will be kept for `trash-days` if trash is enabled (#1790)
- cmd: support using integer for duration flags (#1796)
- cmd: use homedir as default working directory for non-root users (#1869)
- cmd/format: create a uuid object in the target bucket (#1548)
- cmd/dump&load: dump load behavior supports non-ascii characters (#1691)
- cmd/dump: omit empty value in dumped JSON (#1676)
- cmd/dump: remove secret key (#1569)
- meta: encrypt the secret-key and encrypt-key in setting (#1562)
- meta: create subdir automatically (#1712)
- meta: specify the format field preventing update (#1776)
- meta: escape meta password from env (#1879)
- meta/redis: check redis version (#1584)
- meta/redis: use smaller retry backoff in sentinel mode (#1620)
- meta/redis: retry transaction for connection error or EXECABORT (#1637)
- meta/sql: retry transaction after too many connections (#1876)
- meta/sql: add primary key for all tables (#1913, #1919)
- meta&chunk: Set max retries of meta & chunk according to the config `io-retries` (#1713, #1800)
- chunk: limit number of upload goroutines (#1625)
- chunk/store: limit max retry for async upload as well (#1673)
- object/obs: Verify Etag from OBS (#1715)
- object/redis: implement listAll api for redis  (#1777)
- fuse: automatically add `ro` option if mount with `--read-only` (#1661)
- vfs/backup: reduce the limit for skipping backup (#1659)
- sync: reduce memory allocation when write into files (#1644)
- SDK: use uint32 for uid,gid (#1648)
- SDK: handle removeXAttr return code (#1775)
- Dependency: switch to Go 1.17 (#1594)
- Dependency: fix jwt replace (#1534)
- Dependency: upgrade golang-cross version to v1.17.8 (#1539)
- Dependency: upgrade tikv to v2.0.0 (#1643)
- Dependency: reduce dep from minio (#1645)
- Dependency: upgrade gjson to 1.9.3 (#1647)
- Dependency: upgrade sdk for object storage (#1665)
- Dependency: upgrade qiniu sdk (#1697)

# Bugfix

- cmd/format: fix setting quota (#1658)
- cmd/mount: fix parsing of cache dir (#1758)
- cmd/warmup: fix handling of relative paths (#1735)
- cmd/sync: fix sync command not working  when destination is webdav (#1636)
- cmd/gateway: fix s3 gateway DeleteObjects panic (#1527)
- meta: forbid empty name for dentry (#1687)
- meta: lock counters when loading entries (#1703)
- meta: fix snap not released if error occurs when dumping meta (#1669)
- meta: don't update parent attribute if it's a trash directory (#1580)
- meta/redis: fix loading large directory into Redis (#1858)
- meta/redis: update used space/inodes in memory whenever needed (#1573)
- meta/sql: use upsert to update xattr for PG (#1825)
- meta/sql: split insert batch (#1831)
- meta/sql: fix wrong result from scanning SQL database (#1854)
- chunk/cache: Fix read disk_cache not existed remove cache key (#1677)
- object: fallback to List only if ListAll is not supported (#1623)
- object/b2: check returned bucket from B2 (#1745)
- object/encrypt: fix parse rsa from pem (#1724)
- object/encrypt: Add JFS_RSA_PASSPHRASE environment variable prompt information (#1706)
- object/sharding: fix ListAll returning invalid objects (#1616)
- object/ceph: fix listAll hangs if there are many objects (#1891)
- vfs: write control file asynchronously (#1747)
- vfs: fix getlk in access log (#1788)
- sync: Fix copied and copiedBytes (#1801)
- utils: fix the problem that the progress bar loses the log (#1756)
- SDK: rename libjfs atomic (#1939)