v1.2.0

juicedata/juicefsv1.2.0Jun 19, 2024by SandyXSD

AI Summary

This is the third major release and an LTS version, introducing significant features like graceful restart support, POSIX ACLs, IAM integration, and support for new object storage backends like NFS and Dragonfly.

Key Highlights

  • Supports graceful restart for mount processes
  • Supports POSIX ACLs
  • Supports Identity and Access Management (IAM) and events notify for gateway
  • Supports NFS, Dragonfly, and Bunny as object storage
  • Adds a new `compact` command

New Features

  • New `compact` command
  • NFS object storage support
  • Dragonfly object storage support
  • Bunny object storage support
  • POSIX ACL support via `--enable-acl`
  • Various new mount options (e.g., `--skip-dir-mtime`, `--cache-expire`, `--custom-labels`)
  • Open File Description (OFD) lock support
  • Chunk cache consistent hash and bad disk detection

Full Release Notes

This is the third major release of JuiceFS, and is an LTS version as well that will be maintained for 24 months.
Starting from v1.1 it has 464 commits from 44 contributors, thanks to @zhijian-pro @davies @zhoucheng361 @SandyXSD @jiefenghuang @polyrabbit @CaitinChen @yuhr123 @Hexilee @tangyoupeng @YunhuiChen @eryugey @timfeirg @Suave @solracsf @kyungwan-nam @xiaogaozi @1ambd4 @stefanb @XDTD @occupyhabit @menwenjun @li1213987842 @testwill @fengleng @deferdeter @camcui @uvletter @T-TRz879 @yuchen-sun @TechQuery @reAsOn2010 @mtnk2005 @kanemathers @JoJossd @sjp00556 @CodingPoeta @abserari @AChangFeng @feeyman @l0wl3vel @chnliyong @JoyLiuc @prOOrc!

# Highlights

- Supports graceful restart for mount process (#4392, #4450, #4487, #4499, #4509, #4511, #4524, #4547, #4555, #4792, #4823, etc.)
- Supports POSIX ACL (#4421, #4437, #4443, #4490, #4503, #4518, #4522, #4523, #4516, #4551, #4640)
- Supports Identity and Access Management (IAM) and events notify for gateway (#4030, #4593, #4622, #4845, #4891)

# New

- Added a new command `compact` to trigger compaction on a specified path (#4337)
- Supports using NFS as object storage (#3823, #4101, #4105, #4111, #4147, #4163, #4718)
- Supports using [Dragonfly](https://github.com/dragonflyoss/Dragonfly2) as object storage (#4057)
- Supports using [Bunny](https://bunny.net/storage/) as object storage (#4719)
- cmd: add a new option `--log-id` to append user-specified ID in log (#4040)
- cmd/mount: supports restarting mount process gracefully (#4392, #4450, #4487, #4499, #4509, #4511, #4524, #4547, #4555, #4792, #4823, etc.)
- cmd/format: add a new option `--enable-acl` to enable POSIX ACL (#4421, #4437, #4443, #4490, #4503, #4518, #4522, #4523, #4516, #4551, #4640)
- cmd/mount: add a new option `--skip-dir-mtime` to eliminate burst mtime updates of a directory (#4189)
- cmd/mount: add a new option `--upload-hours` to limit the period for background uploads (#4250)
- cmd/mount: add a new option `--cache-expire` to set expired time for cached blocks (#4255, #4365, #4434)
- cmd/mount: add a new option `--custom-labels` to set labels for metrics (#4312, #4612)
- cmd/mount: add a new environment variable `JFS_MOUNT_TIMEOUT` to adjust timeout for checking mount point ready (#4219)
- cmd/mount: add a new option `max-stage-write` to enable concurrent write to both stage files and object storage in writeback mode (#4743)
- cmd/gateway: supports IAM and events notify (#4030, #4593, #4622, #4845, #4891)
- cmd/gateway&webdav: add a new option `--background` to support daemon running (#3993, #4631, #4708, #4915)
- cmd/gateway: add a new option `--domain` to support virtual-host-style requests (#4462)
- cmd/warmup: add a new option `--evict` to manually evict cached blocks (#4370)
- cmd/warmup: and a new option `--check` to check the ratio of already warmed blocks (#4370)
- cmd/dump: add a new option `--fast=false` to reduce memory usage with a slower dump process (#4253)
- cmd/dump: add a new option `--skip-trash` to skip the trash directory when backing up metadata (#4479, #4563)
- cmd/dump: add a new option `--threads` to adjust speed for dump (#4581)
- cmd/sync: add a new option `--inpace` to put file in-place, eliminating the rename operation (#4104)
- cmd/sync: supports two stars `**` and three stars `***` to match all files recursively (#4422, #4448, #4460, #4468, #4474)
- cmd/sync: add a new option `--match-full-path` to match the filter against full path (#4492)
- cmd/sync: add new options `max-size`, `min-size`, `max-age` and `min-age` to filter files based on size and age (#4912, 7491e33, #4913)
- cmd/sync: add p8s metrics to better monitor the sync progress (#4119)
- meta: supports Open File Description (OFD) lock (#4083)
- chunk: disk cache switch to consistent hash and add bad disk detection (#4543, #4749, #4780, #4784, #4871, #4848)
- object: add query string to disable md5 and crc verification when s3 compatible storage does not support them (#4606)
- object: add a new environment variable `JFS_S3_VHOST_STYLE=1` to enable vhost style (#4697)
- hadoop: add `push-labels` option to set custom labels for metrics (#4312)

# Changed

- cmd: support human friendly values for options (#4246, #4558, #4567, #4592, #4641, #4671)
- cmd/mount: only use `writeback_cache` to enable fuse writeback (#4632)
- cmd/gateway: judge loop symbolic links rather than hard-code deep limit (#4045)
- cmd/gateway: create MinioMetaBucket when it doesn't exist (#4383)
- cmd/gateway: create `.minio.sys` directory even if multi-buckets are not enabled (#4367)
- cmd/dump: use a temporary file during dumping (#4767)
- cmd/sync: limit the maximum memory size for multiple uploads (#4135)
- cmd/sync: speed up sync for huge file (#4456)
- cmd/sync: use larger io-size when writing to backends (#4546)
- cmd/sync: skip listing the destination in force mode (#4602)
- cmd/sync: use HEAD instead of LIST for a single object (#4623)
- cmd/sync: limit the total size of each batch task from the manager (#4886)
- meta: add mount time in session info (#3987)
- meta: export meta backup metrics (#4480)
- meta: force compaction when there are too many slices in one chunk (#4573)
- meta: remove tail zero slice when compacting a chunk (#4789)
- meta/redis: reduce the memory usage when dumping large directories (#4481)
- meta/tkv & meta/sql: speedup the dump process using multiple threads (#4501)
- meta/badger: upgrade to v4 (#4564)
- chunk: prefetch the whole block only on successful partial read (#4658)
- chunk: delete leaked objects if it's already deleted by other goroutines (#4748)
- vfs: fuse ops metric supports showing method label (#4554, #4560)
- vfs: add errno label to fuse ops metric (#4656)
- object/tos: adjust min part size from 5 MiB to 4 MiB (#4688)
- object/ceph: allow delete when ceph cluster is full (#4131)
- object/ceph: speedup list for ceph (#4218)
- object/ceph: set default path for admin socket and log file (#4608)
- hadoop: support human friendly values for options (#4655, #4813)
- hadoop: support using `;` to separate values for juicefs.users and juicefs.group (#4724)
- deps: upgrade golang.org/x/net to 0.23 (#4911)

# Bugfix

- cmd/gateway: fix ListMultipartUploads sorting rule and support delimiter (#4297)
- cmd/gateway: fix HEAD directory returns 200 but expect 404 (#4486)
- cmd/gateway: fix the issue that memory is not released immediately after object downloaded (#4507)
- cmd/gateway: fix empty bucket display exception (#4875)
- cmd/gateway: fix the size of JfsObjects.CompleteMultipartUpload (#4893)
- cmd/gateway: fix listing objects with symbolic links (#4919)
- cmd/sync: fix the issue that empty files are always synchronized (#4223)
- cmd/sync: execute chmod after chown to avoid setuid/setgid failure (#4418)
- cmd/sync: fix the issue that setuid/setgid/sticky bits are not set when using `jfs` protocol (#4419)
- cmd/sync: return the error of listing objects (#4879)
- cmd/sync: fix the issue that manager's IP address is never set in cluster mode (#4939)
- cmd/clone: set the attr should be done after checking permissions (#4745)
- cmd/clone: fix source mode check (#4856)
- cmd/debug: add a timeout when collecting the information (#4824)
- meta: fix the issue that `df` may see negative values (#4591)
- meta: fix the error message in quota that always shows inode 0 (#4741)
- meta: fix the protection check for compaction (#4901)
- meta: fix the error code of baseMeta.Open (#4920)
- meta/redis: fix some transactions that are not wrapped by multi/exec (#4801)
- meta/sql: use upsert in Write() to avoid possible deadlock (#4529)
- meta/postgreSQL: fix the address parse (#4731)
- vfs: fix the issue that file entry may still be returned if it's deleted during readdir (#4453)
- vfs: fix the issue that file length may not be updated immediately after fallocate & copyfilerange (#4427)
- vfs: fix the race of getting file length (#4876)
- vfs: fix the issue that plock may be unlocked by mistake (#4910)
- hadoop: make libjfs singleton to avoid possible jnr weakhashmap infinity loop (#4407)
- hadoop: fix guid lookupGroup (#4532)
- hadoop: use auto-increment fs handle to avoid possible conflict (#4411)
- hadoop: fix the issue that `dfsio` does not work because `mapreduce.output.fileoutputformat.compress` is true (#4646)