v2.9.0

Project-HAMi/HAMiv2.9.0May 19, 2026by github-actions[bot]

AI Summary

This release introduces HAMi-core mode for Ascend devices, optimizes performance, and declares HAMi-DRA (NVIDIA) ready for use. It includes extensive new features for debugging, Prometheus monitoring, and support for various hardware vendors like VastAI and Enflame.

Key Highlights

  • HAMi-core mode for Ascend devices
  • HAMi-DRA (NVIDIA) ready for use
  • Enhanced debugging skills and vGPUmonitor capabilities
  • Prometheus ServiceMonitor integration
  • Support for VastAI and Enflame GCU-share

New Features

  • HAMi-core mode for Ascend devices
  • HAMi-DRA support for NVIDIA
  • vGPUmonitor metrics flags and ServiceMonitors
  • k8s-debug-gpu-pod debugging skill
  • VastAI device support
  • Local-deploy target for minikube/kind
  • Multi-device requests for Ascend devices
  • DeepCopy functions for device usage
  • enableGetPreferredAllocation flag

Full Release Notes

<!-- Release notes generated using configuration in .github/release.yml at v2.9.0 -->

## What's Changed

## Core Features:

* **feat: add HAMi-core mode for Ascend devices**
* **feat: Optimize HAMi-core performance**
* **feat: HAMi-DRA(NVIDIA) is ready for use**
* **feat: Volcano-vgpu-device-plugin sync with version 0.19, and support CDI**
* **feat: HAMi-skills for debugging and developing**
* **Document: Add latest benchmark to value the performance of HAMi-core**
* **Fix: Initialization error when using tp on vllm > 0.18**

### ✨ New Features
* feat: add vGPUmonitor --metrics-bind-address flag  by @dongjiang1989 in https://github.com/Project-HAMi/HAMi/pull/1613
* refactor: extract checkUUID function for reuse by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1622
* feat: add promtheus serviceMonitor in helm-charts by @dongjiang1989 in https://github.com/Project-HAMi/HAMi/pull/1614
* feat: add serviceMonitor for device plugin by @dongjiang1989 in https://github.com/Project-HAMi/HAMi/pull/1633
* feat: check resource quota in webhook by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1605
* feat: support module-pair allocation for Ascend 910C devices in SuperPod environments by @ashergaga in https://github.com/Project-HAMi/HAMi/pull/1610
* feat(skill): Add k8s-debug-gpu-pod skill for HAMi GPU troubleshooting by @haitwang-cloud in https://github.com/Project-HAMi/HAMi/pull/1654
* feat: add support for vastai device by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1645
* feat(helm): add config namespaceSelector & objectSelector for webhook by @haitwang-cloud in https://github.com/Project-HAMi/HAMi/pull/1653
* feat(metrics): align prometheus metric and label names with best practices by @MyoungHaSong in https://github.com/Project-HAMi/HAMi/pull/1644
* feat(logging): optimize log verbosity and add unit tests by @haitwang-cloud in https://github.com/Project-HAMi/HAMi/pull/1710
* feat(workflow): stale issues and PRs older than one year by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1725
* feat: increase operations-per-run for stale action by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1756
* feat: add local-deploy target for deploying to minikube/kind clusters by @anandj91 in https://github.com/Project-HAMi/HAMi/pull/1760
* feat: add hami_vgpu_metrics_summarizer skill by @haitwang-cloud in https://github.com/Project-HAMi/HAMi/pull/1755
* feat: add Ascend ResourceCoreName to support hami-vnpu-core virtualization by @ashergaga in https://github.com/Project-HAMi/HAMi/pull/1771
* feat: add Ascendxxx-core resource by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1804
* feat: support node filtering based on hami-vnpu-core annotation by @ashergaga in https://github.com/Project-HAMi/HAMi/pull/1812
* refactor: filter device at nvml-manager by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1825
* feat: add DeepCopy function for DeviceUsage and its nested types by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1818
* feat: Supports multi-device requests with hami-vnpu-core enabled. by @ashergaga in https://github.com/Project-HAMi/HAMi/pull/1837
* feat: update skills and add CLAUDE.md for codebase guidance by @haitwang-cloud in https://github.com/Project-HAMi/HAMi/pull/1842
* feat: add enableGetPreferredAllocation flag by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1824
### 🐛 Bug Fixes
* fix device typos by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1608
* fix: precedence bug in schedulerName check by @hoteye in https://github.com/Project-HAMi/HAMi/pull/1627
* fix: add nil checks to prevent panics in leaderelection by @haitwang-cloud in https://github.com/Project-HAMi/HAMi/pull/1603
* fix: panic on nil resourceReqs in scheduler calcScore by @yxxhero in https://github.com/Project-HAMi/HAMi/pull/1626
* Fix the issue Iluvatar device scheduling policy binpack and spread are reversed by @qiangwei1983 in https://github.com/Project-HAMi/HAMi/pull/1631
* Fix contact link and add slack channel to README.md by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/1635
* fix: resolve cardinality explosion in Device_memory_desc_of_container by @maishivamhoo123 in https://github.com/Project-HAMi/HAMi/pull/1628
* fix: handle GetMemoryInfo ERROR_NOT_SUPPORTED for unified memory GPUs by @jsl9208 in https://github.com/Project-HAMi/HAMi/pull/1637
* fix typos by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1657
* Fix: optimize nodelock scalability with exponential backoff and listers by @maishivamhoo123 in https://github.com/Project-HAMi/HAMi/pull/1663
* fix apply kubescheduler config version by @CoderTH in https://github.com/Project-HAMi/HAMi/pull/1666
* fix: failing readiness probe when replica > 1 by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1677
* fix: ci deepth by @Atroxgod in https://github.com/Project-HAMi/HAMi/pull/1690
* fix(scheduler): correct slot usage prediction and add device type fil… by @maishivamhoo123 in https://github.com/Project-HAMi/HAMi/pull/1700
* fix vastai fit loop iteration direction by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1715
* fix: retain terminating pod in cache to prevent premature eviction by @maishivamhoo123 in https://github.com/Project-HAMi/HAMi/pull/1719
* fix(chart): derive ld.so.preload from devicePlugin.libPath to fix non-default path deployments by @ilia-medvedev in https://github.com/Project-HAMi/HAMi/pull/1714
* fix: support device allocation for multi-container  with init containers by @haitwang-cloud in https://github.com/Project-HAMi/HAMi/pull/1650
* fix: Reponse with correct content-type by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1604
* fix: suppress scheduler cleanup noise for unrelated vendors by @Yonsun-w in https://github.com/Project-HAMi/HAMi/pull/1749
* fix(scheduler): add missing return in ondelpod by @CFH2436 in https://github.com/Project-HAMi/HAMi/pull/1759
* fix: global image tag always covers per-component image tag by @FouoF in https://github.com/Project-HAMi/HAMi/pull/1774
* fix(device-plugin): align kubelet allocation with scheduler annotations (#1741) by @xrwang8 in https://github.com/Project-HAMi/HAMi/pull/1743
* fix(chart): sanitize managedNodeSelector keys for environment variables by @almazkhalikov in https://github.com/Project-HAMi/HAMi/pull/1783
* fix: filter device does not work by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1817
* Fix: Handle Kernel 6.17 handshake edge cases in NVIDIA health checks by @maishivamhoo123 in https://github.com/Project-HAMi/HAMi/pull/1810
* fix: git errors when building with latest hami-core by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1782
* fix: mig does not work by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1819
* fix(scheduler): guard against zero-value division in ComputeScore (#1… by @lin121291 in https://github.com/Project-HAMi/HAMi/pull/1820
* fix(ascend): check unmarshal error before iterating node devices by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/1831
* fix(device): parse handshake annotation timestamp in local timezone by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/1816
* fix: recover scheduling on nodes with stale Deleted_ handshake by @saiyam1814 in https://github.com/Project-HAMi/HAMi/pull/1843
* fix A100-80G template by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1847
* fix: allocation failed when using mig in CDI mode by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1826
### 📚 Documentation
* docs: remove stale tasklist.md, migrate content to separate issues by @ManishSharma1609 in https://github.com/Project-HAMi/HAMi/pull/1801
* docs: add CNCF copyright disclaimer to README files by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/1830
* docs: complete tasklist.md migration - add hardware entries, resource… by @maishivamhoo123 in https://github.com/Project-HAMi/HAMi/pull/1829
* docs: add v2.9.0 documentation audit report by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/1761
### 🔨 Other Changes
* add device type label in metrics by @xiyichan in https://github.com/Project-HAMi/HAMi/pull/1612
* security: add io.LimitReader to scheduler routes to prevent DoS #554 by @maishivamhoo123 in https://github.com/Project-HAMi/HAMi/pull/1620
* add fouof to approvers by @FouoF in https://github.com/Project-HAMi/HAMi/pull/1642
* Update Maintainers information by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/1646
* [Snyk] Security upgrade tensorflow/tensorflow from 2.20.0rc0-gpu to 2.21.0rc0-gpu by @wawa0210 in https://github.com/Project-HAMi/HAMi/pull/1652
* chore: remove deprecated scheduler policy configmap by @haitwang-cloud in https://github.com/Project-HAMi/HAMi/pull/1651
* [Snyk] Security upgrade tensorflow/tensorflow from 2.21.0rc0-gpu to 2.21.0rc1-gpu by @wawa0210 in https://github.com/Project-HAMi/HAMi/pull/1681
* vGPUmonitor: skip devices with invalid UTF-8 UUID during container init by @charford in https://github.com/Project-HAMi/HAMi/pull/1703
* Helm - Render nvidia.overwriteEnv from values with a default of false. by @jcustenborder in https://github.com/Project-HAMi/HAMi/pull/1706
* Update dashboard.md fix nvidia.com/gpucores comment by @Nov11 in https://github.com/Project-HAMi/HAMi/pull/1712
* update nvidia_dp and nvidia_container_runtime module by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/1731
* chore: resolve staticcheck and modernize linter warnings by @maishivamhoo123 in https://github.com/Project-HAMi/HAMi/pull/1728
* chore: using LeaderElectionConfiguration for kubernetes v1.23 and newer version by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1737
* Add general Technical review to documents by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/1752
* test: improve coverage for pkg/version/version.go by @Yonsun-w in https://github.com/Project-HAMi/HAMi/pull/1748
* Configure ASCEND_VISIBLE_DEVICES env for container and RuntimeClassName for pods by @peachest in https://github.com/Project-HAMi/HAMi/pull/1738
* test: improve nvidia device.go UT coverage to 98.5% by @kenwoodjw in https://github.com/Project-HAMi/HAMi/pull/1757
* security: upgrade golang for security issue by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1772
* Upgrade Go to 1.26.2 by @luohua13 in https://github.com/Project-HAMi/HAMi/pull/1791
* update benchmarks by @maverick123123 in https://github.com/Project-HAMi/HAMi/pull/1790
* add DSFans2014 to approvers by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1805
* Disable host network for device plugin by @luohua13 in https://github.com/Project-HAMi/HAMi/pull/1789
* bump HAMi-DRA version to v0.2.0 by @FouoF in https://github.com/Project-HAMi/HAMi/pull/1845
* Preparations to release v2.9.0 by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/1850

## New Contributors
* @maishivamhoo123 made their first contribution in https://github.com/Project-HAMi/HAMi/pull/1620
* @hoteye made their first contribution in https://github.com/Project-HAMi/HAMi/pull/1627
* @jsl9208 made their first contribution in https://github.com/Project-HAMi/HAMi/pull/1637
* @ashergaga made their first contribution in https://github.com/Project-HAMi/HAMi/pull/1610
* @Atroxgod made their first contribution in https://github.com/Project-HAMi/HAMi/pull/1690
* @MyoungHaSong made their first contribution in https://github.com/Project-HAMi/HAMi/pull/1644
* @charford made their first contribution in https://github.com/Project-HAMi/HAMi/pull/1703
* @jcustenborder made their first contribution in https://github.com/Project-HAMi/HAMi/pull/1706
* @Nov11 made their first contribution in https://github.com/Project-HAMi/HAMi/pull/1712
* @ilia-medvedev made their first contribution in https://github.com/Project-HAMi/HAMi/pull/1714
* @Yonsun-w made their first contribution in https://github.com/Project-HAMi/HAMi/pull/1749
* @CFH2436 made their first contribution in https://github.com/Project-HAMi/HAMi/pull/1759
* @kenwoodjw made their first contribution in https://github.com/Project-HAMi/HAMi/pull/1757
* @anandj91 made their first contribution in https://github.com/Project-HAMi/HAMi/pull/1760
* @ManishSharma1609 made their first contribution in https://github.com/Project-HAMi/HAMi/pull/1801
* @maverick123123 made their first contribution in https://github.com/Project-HAMi/HAMi/pull/1790
* @almazkhalikov made their first contribution in https://github.com/Project-HAMi/HAMi/pull/1783
* @lin121291 made their first contribution in https://github.com/Project-HAMi/HAMi/pull/1820

**Full Changelog**: https://github.com/Project-HAMi/HAMi/compare/v2.8.0...v2.9.0