v2.4.18

milvus-io/milvusv2.4.18Dec 20, 2024by yanliang567

AI Summary

Milvus 2.4.18 introduces customizable privilege groups and an enhanced Grant/Revoke API to streamline permission management. This version includes performance improvements such as CPU optimization, faster collection loading, and sparse index enhancements (~10% improvement), along with critical bug fixes for system stability and synchronization.

Key Highlights

  • Customizable privilege groups (Cluster, Database, Collection levels) with built-in presets
  • Enhanced Grant/Revoke API that does not require specifying an ObjectType
  • Sparse index optimization and Knowhere version update for better performance
  • Critical bug fixes for crash scenarios and synchronization failures

New Features

  • Customizable privilege groups
  • Built-in privilege groups (ClusterReadOnly, DatabaseReadWrite, etc.)
  • New Grant/Revoke API version
  • Support for hyphens in partition names
  • Templates for expressions in RESTful APIs
  • Removal of load task limits per round
  • Support for modifying properties via alterindex and altercollection
  • Support for deleting properties via alterdatabase
  • Detailed replica counts for resource groups
  • Score-based balancing for channel policies
  • Metrics for non-zero values in sparse searches
  • Removal of RPC layer of coordinator in standalone/mixed mode
  • Mmap file usage metrics
  • Support for database requests in RESTful API
  • Rate limiting for RESTful V1
  • Collection ID to search request count metrics
  • Refine clustering compaction logs
  • Accelerate collection loading process
  • Improve compaction performance by removing ParamTable lookups
  • Retry searches when topk is reduced and results are insufficient

Full Release Notes

## v2.4.18

Release Date: December 20, 2024

| Milvus version | Python SDK version | Java SDK version | Node.js SDK version |
|----------------|--------------------|------------------|---------------------|
| 2.4.18         | 2.4.12              | 2.4.9           | 2.4.9               |

Milvus 2.4.18 introduces customizable privilege groups and an enhanced Grant/Revoke API, significantly streamlining permission management. This version also brings a suite of built-in privilege groups for common operational scenarios, as well as a host of performance and stability improvements—ranging from CPU optimization to faster collection loading and more efficient indexing. In addition, critical bug fixes ensure a more resilient system, addressing issues like crash scenarios and synchronization failures. We recommend you upgrade to 2.4.18 to take advantage of these key enhancements and improved overall reliability.

### Features

RBAC Enhancements:

- Introduces customizable privilege groups, allowing users to create, drop, list, and dynamically manage privileges (add/remove) within their defined groups.
- Includes a suite of built-in privilege groups for common operational scenarios:
  - Cluster-Level: *ClusterReadOnly*, *ClusterReadWrite*, *ClusterAdmin*
  - Database-Level: *DatabaseReadOnly*, *DatabaseReadWrite*, *DatabaseAdmin*
  - Collection-Level: *CollectionReadOnly*, *CollectionReadWrite*, *CollectionAdmin*
- Provides a new version of the Grant/Revoke API, enabling the use of these new interfaces without the need to specify an `ObjectType`.

### Improvements

- Allow hyphens in partition names ([#38474](https://github.com/milvus-io/milvus/pull/38474))
- Optimize CPU usage for health check requests ([#35595](https://github.com/milvus-io/milvus/pull/35595))
- Support templates for expressions in RESTful APIs ([#38161](https://github.com/milvus-io/milvus/pull/38161))
- Remove the limit on the number of load tasks per round ([#38497](https://github.com/milvus-io/milvus/pull/38497))
- `alterindex` & `altercollection` now support modifying properties ([#38361](https://github.com/milvus-io/milvus/pull/38361) [#38111](https://github.com/milvus-io/milvus/pull/38111) [#38421](https://github.com/milvus-io/milvus/pull/38421))
- `alterdatabase` supports deleting properties ([#38450](https://github.com/milvus-io/milvus/pull/38450))
- Add detailed replica counts for resource groups ([#38315](https://github.com/milvus-io/milvus/pull/38315))
- Support score-based balancing for channel policies  ([#38378](https://github.com/milvus-io/milvus/pull/38378))
- Add metrics to count the number of non-zero values/tokens in sparse searches ([#38328](https://github.com/milvus-io/milvus/pull/38328))
- Remove the RPC layer of the coordinator when running in standalone or mixed mode ([#38207](https://github.com/milvus-io/milvus/pull/38207))
- Add mmap file usage metrics ([#38211](https://github.com/milvus-io/milvus/pull/38211))
- Support database requests in RESTful API ([#38188](https://github.com/milvus-io/milvus/pull/38188))
- Enable rate limiting for RESTful V1 ([#38190](https://github.com/milvus-io/milvus/pull/38190))
- Add collection ID to search request count metrics ([#38144](https://github.com/milvus-io/milvus/pull/38144))
- Refine clustering compaction logs ([#38102](https://github.com/milvus-io/milvus/pull/38102))
- Accelerate the collection loading process ([#37841](https://github.com/milvus-io/milvus/pull/37841))
- Improve compaction performance by removing ParamTable lookups ([#37882](https://github.com/milvus-io/milvus/pull/37882))
- Support retrying searches when topk is reduced and results are insufficient ([#37093](https://github.com/milvus-io/milvus/pull/37093))
- Update Knowhere version ([#38277](https://github.com/milvus-io/milvus/pull/38277))
  - Optimize sparse index and get ~10% performance improvement

### Bug fixes

- Fixed a crash caused by growing-groupby ([#38553](https://github.com/milvus-io/milvus/pull/38553))
- Fixed an issue where the `SyncSegments` RPC would always fail ([#38032](https://github.com/milvus-io/milvus/pull/38032))
- Fixed an issue where sync tasks remained running after the DataNode had stopped ([#38441](https://github.com/milvus-io/milvus/pull/38441))
- Fixed inaccurate general counts ([#38525](https://github.com/milvus-io/milvus/pull/38525))
- Escaped prefixes before conducting searches in inverted indexes  ([#38425](https://github.com/milvus-io/milvus/pull/38425))
- Fixed an issue where roles could be dropped even though grants still existed ([#38369](https://github.com/milvus-io/milvus/pull/38369))
- Fixed empty import task results ([#38317](https://github.com/milvus-io/milvus/pull/38317))
- Fixed a DataNode issue where progress could stall at the writer buffer memory check ([#38287](https://github.com/milvus-io/milvus/pull/38287))
- Fixed an issue that prevented the permission grant on the `manualcompact` API ([#38168](https://github.com/milvus-io/milvus/pull/38168))
- Fixed inaccurate partition count metrics ([#38073](https://github.com/milvus-io/milvus/pull/38073))
- Accelerated flushing speed by optimizing lock usage ([#37897](https://github.com/milvus-io/milvus/pull/37897))
- Handled errors gracefully when the compaction queue is full ([#37990](https://github.com/milvus-io/milvus/pull/37990))
- Optimized loading speed by separating the pool for target observation and collection loading ([#37735](https://github.com/milvus-io/milvus/pull/37735))
- Fixed a crash caused by retrieving varchar data from a memory-mapped growing segment ([#37995](https://github.com/milvus-io/milvus/pull/37995))
- Fixed an issue where channels could be accidentally released after balancing ([#37940](https://github.com/milvus-io/milvus/pull/37940))