v2.5.0
Project-HAMi/HAMiv2.5.0Feb 6, 2025by github-actions[bot]
AI Summary
Major update introducing Dynamic MIG support and centralizing all configurations into a single configMap, ensuring safety during reinstallations.
Key Highlights
- Support dynamic MIG feature
- All configurations in a central configMap
- Reinstall HAMi will not crash GPU tasks
New Features
- Dynamic MIG feature
- Centralized configMap
- Fix for cuMallocAsync issues
- Fix for high glibc images
Full Release Notes
<!-- Release notes generated using configuration in .github/release.yml at v2.5.0 --> Major features: 1. Support dynamic mig feature, please refer to this [document](https://github.com/Project-HAMi/HAMi/blob/master/docs/dynamic-mig-support.md) 2. Reinstall Hami will NOT crash GPU tasks 3. Put all configurations into a configMap, you can customize hami installation by modify its content: see [details](https://github.com/Project-HAMi/HAMi/blob/master/docs/config.md) Major bug fixes: 1. Fix an issue where hami-core will stuck on tasks using 'cuMallocAsync' 2. Fix hami-core stuck on high glib images, like 'tf-serving:latest' ## What's Changed ### ⬆️ Dependencies * Bump aquasecurity/trivy-action from 0.28.0 to 0.29.0 by @dependabot in https://github.com/Project-HAMi/HAMi/pull/631 * Bump nvidia/cuda from 12.4.1-base-ubuntu22.04 to 12.6.3-base-ubuntu22.04 in /docker by @dependabot in https://github.com/Project-HAMi/HAMi/pull/676 * Bump actions/upload-artifact from 4.4.3 to 4.5.0 by @dependabot in https://github.com/Project-HAMi/HAMi/pull/717 * Bump docker/build-push-action from 6.9.0 to 6.10.0 by @dependabot in https://github.com/Project-HAMi/HAMi/pull/644 * Bump docker/build-push-action from 6.10.0 to 6.11.0 by @dependabot in https://github.com/Project-HAMi/HAMi/pull/792 ### 🔨 Other Changes * Fix Kubernetes version string handling by stripping metadata by @Nimbus318 in https://github.com/Project-HAMi/HAMi/pull/623 * Update vGPUmonitor to add dynamic adjustment on core and memory limit by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/624 * feat: support device plugin daemonset update strategy by @devenami in https://github.com/Project-HAMi/HAMi/pull/628 * add ut about schedule policy by @yt-huang in https://github.com/Project-HAMi/HAMi/pull/638 * Fix: Refactor the license based on the approaches used in OpenSearch and ElasticSearch. by @haitwang-cloud in https://github.com/Project-HAMi/HAMi/pull/626 * add ut for the scheduler by @shijinye in https://github.com/Project-HAMi/HAMi/pull/645 * docs(issue-tmpl): add FAQ link to issue templates by @Nimbus318 in https://github.com/Project-HAMi/HAMi/pull/647 * fix: filter device registry to node by @lengrongfu in https://github.com/Project-HAMi/HAMi/pull/639 * Add self-hosted runner by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/659 * fix-example-yaml by @WQL782795 in https://github.com/Project-HAMi/HAMi/pull/667 * update docs by @yangshiqi in https://github.com/Project-HAMi/HAMi/pull/668 * add ut for ascend by @shijinye in https://github.com/Project-HAMi/HAMi/pull/664 * optimization map init in test by @lengrongfu in https://github.com/Project-HAMi/HAMi/pull/678 * Optimize monitor by @for800000 in https://github.com/Project-HAMi/HAMi/pull/683 * fix code lint faild by @lengrongfu in https://github.com/Project-HAMi/HAMi/pull/685 * fix(helm): Add NODE_NAME env var to the vgpu-monitor container from spec.nodeName by @Nimbus318 in https://github.com/Project-HAMi/HAMi/pull/687 * fix vGPUmonitor deviceidx is always 0 by @lengrongfu in https://github.com/Project-HAMi/HAMi/pull/684 * add ut for pkg/scheduler/event.go by @Penguin-zlh in https://github.com/Project-HAMi/HAMi/pull/688 * add ut for nodes by @shijinye in https://github.com/Project-HAMi/HAMi/pull/695 * add license for pkg/scheduler/event_test.go by @Penguin-zlh in https://github.com/Project-HAMi/HAMi/pull/706 * fix: exception happen when creating multiple ascend-gpu pods concurrently by @lijm87 in https://github.com/Project-HAMi/HAMi/pull/575 * add ut for device/nvidia by @shijinye in https://github.com/Project-HAMi/HAMi/pull/657 * add ut for pkg/monitor/nvidia/v0/spec.go by @yt-huang in https://github.com/Project-HAMi/HAMi/pull/670 * Enable Dynamic-mig feature for HAMi by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/708 * Fix chart can not be deployed properly by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/711 * Fix NodeLock issue by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/714 * fix example yaml by @lixd in https://github.com/Project-HAMi/HAMi/pull/709 * add ut for device/cambricon by @shijinye in https://github.com/Project-HAMi/HAMi/pull/712 * Update dynamic mig documents and examples by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/718 * random time may be zero by @shijinye in https://github.com/Project-HAMi/HAMi/pull/697 * fix grafana dashboard and clarify dashboard usage more clearly. by @jiangsanyin in https://github.com/Project-HAMi/HAMi/pull/543 * doc(README): add examples for GPU sharing and update-examples by @xiaoyao in https://github.com/Project-HAMi/HAMi/pull/665 * add ut for github.com/Project-HAMi/HAMi/pkg/scheduler/pod.go by @yt-huang in https://github.com/Project-HAMi/HAMi/pull/673 * Add design document to 'dynamic-mig' feature by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/725 * fix(doc): fix a typo and resolve markdown warnings in the tasklist by @elrondwong in https://github.com/Project-HAMi/HAMi/pull/724 * add ut for pkg/util/nodelock/nodelock.go by @learner0810 in https://github.com/Project-HAMi/HAMi/pull/719 * test: add ut for pkg/version/version.go by @Penguin-zlh in https://github.com/Project-HAMi/HAMi/pull/677 * Update on mig mode by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/726 * Update documents for config & config_cn by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/729 * set PASS_DEVICE_SPECS ENV to device-plugin by @jingzhe6414 in https://github.com/Project-HAMi/HAMi/pull/690 * fix device-plugin-version by @learner0810 in https://github.com/Project-HAMi/HAMi/pull/743 * feat: Return the nodes that failed to be scheduled back to the scheduler by @chaunceyjiang in https://github.com/Project-HAMi/HAMi/pull/746 * fix(log): fix missing log output in nvidiadeviceplugin server by @elrondwong in https://github.com/Project-HAMi/HAMi/pull/735 * support configuration resources limits and requests by @flpanbin in https://github.com/Project-HAMi/HAMi/pull/739 * feat(test): add TestMarshalNodeDevices scenarios by @elrondwong in https://github.com/Project-HAMi/HAMi/pull/747 * print flags for device-plugin and scheduler by @flpanbin in https://github.com/Project-HAMi/HAMi/pull/756 * Fix typos, add more contributors and maintainers. by @yangshiqi in https://github.com/Project-HAMi/HAMi/pull/765 * Add a mind map(Chinese and English) to help understand this project by @oceanweave in https://github.com/Project-HAMi/HAMi/pull/764 * [Docs] update config pages by @windsonsea in https://github.com/Project-HAMi/HAMi/pull/760 * add ut for device-map by @KubeKyrie in https://github.com/Project-HAMi/HAMi/pull/762 * refactor(ci): use go.mod file for Go version in workflows by @yxxhero in https://github.com/Project-HAMi/HAMi/pull/766 * support set log level for device plugin by @flpanbin in https://github.com/Project-HAMi/HAMi/pull/771 * feat: Restart/Upgrade device-plugin will not affect services. by @chaunceyjiang in https://github.com/Project-HAMi/HAMi/pull/767 * add ut nvml devices by @KubeKyrie in https://github.com/Project-HAMi/HAMi/pull/773 * add ut for device-map by @KubeKyrie in https://github.com/Project-HAMi/HAMi/pull/772 * Optimize the time format layout by @learner0810 in https://github.com/Project-HAMi/HAMi/pull/741 * fix: nvidia-device-plugin no version info by @chaunceyjiang in https://github.com/Project-HAMi/HAMi/pull/779 * HAMi supports e2e by @Rei1010 in https://github.com/Project-HAMi/HAMi/pull/775 * Proposal: enable E2E test by @Rei1010 in https://github.com/Project-HAMi/HAMi/pull/633 * add ut for device/iluvatar by @shijinye in https://github.com/Project-HAMi/HAMi/pull/795 * add ut for device/hygon by @shijinye in https://github.com/Project-HAMi/HAMi/pull/787 * add ut for pkg/monitor/nvidia/v1 by @shijinye in https://github.com/Project-HAMi/HAMi/pull/780 * refactor(logging): enhance log messages for device resource counting by @haitwang-cloud in https://github.com/Project-HAMi/HAMi/pull/778 * Enrich pod health check by @Rei1010 in https://github.com/Project-HAMi/HAMi/pull/801 * docs: fix broken link by @lixd in https://github.com/Project-HAMi/HAMi/pull/802 * Optimize the E2E execution logic by @Rei1010 in https://github.com/Project-HAMi/HAMi/pull/803 * optimize MetricsBindAddress to MetricsBindPort by @phoenixwu0229 in https://github.com/Project-HAMi/HAMi/pull/796 * fix: handle the node nil issue & E2E test failure by @haitwang-cloud in https://github.com/Project-HAMi/HAMi/pull/804 * add ut for device/mthreads by @shijinye in https://github.com/Project-HAMi/HAMi/pull/808 * fix: Resolve formatting issue in ConfigMap causing display anomalies by @lixd in https://github.com/Project-HAMi/HAMi/pull/814 * [docs] Update ascend910b-support.md by @windsonsea in https://github.com/Project-HAMi/HAMi/pull/816 * Refine metrics logs by @haitwang-cloud in https://github.com/Project-HAMi/HAMi/pull/817 * Update mig-related logics and refine logs by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/833 * Add 910B4 config to device-configmap for ascend by @lijm87 in https://github.com/Project-HAMi/HAMi/pull/828 * [docs] fix: glibc version requirement in README by @chinaran in https://github.com/Project-HAMi/HAMi/pull/826 * Update HAMi-core for v2.5.0 by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/834 * FIx multi-process device memory count issue by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/835 * bump version to v2.5.0 by @wawa0210 in https://github.com/Project-HAMi/HAMi/pull/836 * Fix CI by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/838 * Fix CI release by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/840 * Fix release ci by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/841 * Fix Dockerfile to make CI pass by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/846 * Fix E2E failure with pod status check by @Rei1010 in https://github.com/Project-HAMi/HAMi/pull/847 * Fix scheduler crash if a 'mig' task running accidentally on a 'hami-core' GPU by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/848 ## New Contributors * @yt-huang made their first contribution in https://github.com/Project-HAMi/HAMi/pull/638 * @shijinye made their first contribution in https://github.com/Project-HAMi/HAMi/pull/645 * @WQL782795 made their first contribution in https://github.com/Project-HAMi/HAMi/pull/667 * @yangshiqi made their first contribution in https://github.com/Project-HAMi/HAMi/pull/668 * @for800000 made their first contribution in https://github.com/Project-HAMi/HAMi/pull/683 * @Penguin-zlh made their first contribution in https://github.com/Project-HAMi/HAMi/pull/688 * @lixd made their first contribution in https://github.com/Project-HAMi/HAMi/pull/709 * @jiangsanyin made their first contribution in https://github.com/Project-HAMi/HAMi/pull/543 * @xiaoyao made their first contribution in https://github.com/Project-HAMi/HAMi/pull/665 * @elrondwong made their first contribution in https://github.com/Project-HAMi/HAMi/pull/724 * @learner0810 made their first contribution in https://github.com/Project-HAMi/HAMi/pull/719 * @jingzhe6414 made their first contribution in https://github.com/Project-HAMi/HAMi/pull/690 * @flpanbin made their first contribution in https://github.com/Project-HAMi/HAMi/pull/739 * @oceanweave made their first contribution in https://github.com/Project-HAMi/HAMi/pull/764 * @windsonsea made their first contribution in https://github.com/Project-HAMi/HAMi/pull/760 * @KubeKyrie made their first contribution in https://github.com/Project-HAMi/HAMi/pull/762 * @yxxhero made their first contribution in https://github.com/Project-HAMi/HAMi/pull/766 * @Rei1010 made their first contribution in https://github.com/Project-HAMi/HAMi/pull/775 * @phoenixwu0229 made their first contribution in https://github.com/Project-HAMi/HAMi/pull/796 * @chinaran made their first contribution in https://github.com/Project-HAMi/HAMi/pull/826 **Full Changelog**: https://github.com/Project-HAMi/HAMi/compare/v2.4.1...v2.5.0