Starstruk Logo MarkStarstruk

v0.9.3

skypilot-org/skypilotv0.9.3May 20, 2025by zpoint

AI Summary

SkyPilot v0.9.3 is a feature-rich release introducing a High Availability Service Controller, major Kubernetes improvements (GPU bin packing, CPU-only job scheduling, IPv6 SSH support), expanded cloud support (Nebius internal IP, RunPod credentials via Helm, GCP H200 GPUs), and various API server enhancements. The release also includes documentation updates, testing improvements, and backward compatibility fixes.

Key Highlights

  • Introduced High Availability Service Controller for improved reliability
  • Major Kubernetes enhancements: GPU bin packing via affinity, CPU-only jobs scheduling, IPv6 SSH support, and force terminate for misbehaving pods
  • Expanded cloud provider support: Nebius (internal IP, Docker support, config file), RunPod (Helm credentials, zone provisioning), GCP (hyperdisk-balanced for a3 series, H200 GPUs)
  • API server improvements: parallel sky check, controller name validation, AWS token reloading, log handling optimization
  • Documentation updates: API server tuning guide, RunPod/Nebius credential setup, AWS auth consolidation, EFA docs

New Features

  • High Availability Service Controller
  • Kubernetes GPU bin packing via affinity
  • Kubernetes CPU-only jobs prefer nodes without GPUs
  • Kubernetes IPv6 SSH support
  • Kubernetes force terminate misbehaving pods
  • Kubernetes show-gpus new UX with 0 GPU nodes support
  • Nebius cloud support: internal IP usage, Docker support, config file support
  • RunPod support: Helm credentials, zone-based provisioning, docker environment variables preservation
  • GCP support: hyperdisk-balanced for a3 series, H200 GPU catalog, GPU direct tcpx
  • AWS: consistent instance termination logic, default security group name script
  • Private docker registry support
  • API server parallel sky check
  • Allow specifying autostop in resources
  • Task envs to event_callback
  • Add optional version parameter to docker build pipeline
  • Remove upper limit on urllib3 version

Full Release Notes

## What's Changed
* [Core] Do not initialize conda for users if using docker image by @SeungjinYang in https://github.com/skypilot-org/skypilot/pull/5303
* [AWS] Use consistent list for instance termination logic by @SeungjinYang in https://github.com/skypilot-org/skypilot/pull/5316
* Fix: validate dag should not block uvicorn by @aylei in https://github.com/skypilot-org/skypilot/pull/5328
* [UX] Rename file to match import path by @DanielZhangQD in https://github.com/skypilot-org/skypilot/pull/5349
* update installation docs and Dockerfile to reflect omegaconf by @cg505 in https://github.com/skypilot-org/skypilot/pull/5356
* add typechecking for boto clients/resources by @cg505 in https://github.com/skypilot-org/skypilot/pull/5319
* [GCP] add helptext about API srv missing `gcloud` when installed using wget by @SeungjinYang in https://github.com/skypilot-org/skypilot/pull/5335
* support `all_users` in wait for job status in back compact tests by @zpoint in https://github.com/skypilot-org/skypilot/pull/5346
* fixes from #5335 by @SeungjinYang in https://github.com/skypilot-org/skypilot/pull/5361
* [k8s] Hints for querying stale kube current context by @kyuds in https://github.com/skypilot-org/skypilot/pull/5273
* [Docs] Remove meetup announcement banner by @romilbhardwaj in https://github.com/skypilot-org/skypilot/pull/5364
* API version schema check for release pipeline by @zpoint in https://github.com/skypilot-org/skypilot/pull/5309
* [GCP] Support hyperdisk-balanced for a3 series by @JiangJiaWei1103 in https://github.com/skypilot-org/skypilot/pull/5351
* [k8s] Better error message for stale jobs controller by @kyuds in https://github.com/skypilot-org/skypilot/pull/5274
* [GH] Fix release action by @Michaelvll in https://github.com/skypilot-org/skypilot/pull/5373
* [RunPod] Use zone to provision in a specific data center ID  by @Kovbo in https://github.com/skypilot-org/skypilot/pull/5166
* [UX] remove references to SKYPILOT_GLOBAL_CONFIG envvar in docs by @SeungjinYang in https://github.com/skypilot-org/skypilot/pull/5374
* [aws] fix logic to detect rule with all traffic from security group by @SeungjinYang in https://github.com/skypilot-org/skypilot/pull/5332
* [UX] Remove credentials from the dashboard URL and update the dashboard build hint by @DanielZhangQD in https://github.com/skypilot-org/skypilot/pull/5363
* supress warning by setting default value of `asyncio_default_fixture_loop_scope` by @zpoint in https://github.com/skypilot-org/skypilot/pull/5348
* [Docs] Fix migration guide version by @romilbhardwaj in https://github.com/skypilot-org/skypilot/pull/5369
* update AWS credential setup docs, consolidate cloud auth docs by @cg505 in https://github.com/skypilot-org/skypilot/pull/5122
* [k8s] Force terminate misbehaving pods by @romilbhardwaj in https://github.com/skypilot-org/skypilot/pull/5370
* Fix broken test `test_gcp_disk_tier`  by @zpoint in https://github.com/skypilot-org/skypilot/pull/5393
* Remove `pytest.ini` to remove test warning by @zpoint in https://github.com/skypilot-org/skypilot/pull/5379
* Refine API server deployment doc by @aylei in https://github.com/skypilot-org/skypilot/pull/5295
* [Docs] Add API server tuning guide by @aylei in https://github.com/skypilot-org/skypilot/pull/5176
* Introduce High Availability Service Controller by @andylizf in https://github.com/skypilot-org/skypilot/pull/4564
* [API server] Fix worker number for non-local low resource env by @aylei in https://github.com/skypilot-org/skypilot/pull/5409
* [k8s] Fix IPv6 SSH by @kyuds in https://github.com/skypilot-org/skypilot/pull/5413
* Fix terminating k8s cluster by @aylei in https://github.com/skypilot-org/skypilot/pull/5412
* [UX] `api info`: display dashboard on last line by @concretevitamin in https://github.com/skypilot-org/skypilot/pull/5417
* [UX] Minor fix. by @concretevitamin in https://github.com/skypilot-org/skypilot/pull/5420
* [config] remove omegaconf as dependency by @SeungjinYang in https://github.com/skypilot-org/skypilot/pull/5375
* [k8s] idea: allow an accelerator to map to multiple label values by @SeungjinYang in https://github.com/skypilot-org/skypilot/pull/5343
* [Nebius] Don't cache session across multiple requests by @SalikovAlex in https://github.com/skypilot-org/skypilot/pull/5347
* [Nebius] Add Docker support for Nebius cloud by @SalikovAlex in https://github.com/skypilot-org/skypilot/pull/5334
* Qwen3 235b example  by @Michaelvll in https://github.com/skypilot-org/skypilot/pull/5425
* [UX][k8s] show-gpus for all allowed contexts by @kyuds in https://github.com/skypilot-org/skypilot/pull/5362
* [API server] make server config conherent by @aylei in https://github.com/skypilot-org/skypilot/pull/5414
* [Catalog] use v7 for latest runpod by @Michaelvll in https://github.com/skypilot-org/skypilot/pull/5422
* [UX] Update dashboard favicon with transparent background by @DanielZhangQD in https://github.com/skypilot-org/skypilot/pull/5426
* [Core][RunPod] Show error for RunPod multi-node by @kyuds in https://github.com/skypilot-org/skypilot/pull/5368
* Support SDK backward compatibility test by @zpoint in https://github.com/skypilot-org/skypilot/pull/5398
* Add helm support for RunPod credentials by @funkypenguin in https://github.com/skypilot-org/skypilot/pull/5214
* [aws] script to get default security group name for aws by @SeungjinYang in https://github.com/skypilot-org/skypilot/pull/5427
* Update pypi description by @Michaelvll in https://github.com/skypilot-org/skypilot/pull/5444
* avoid using removed LEGACY_SINGLETON_REGION constant by @cg505 in https://github.com/skypilot-org/skypilot/pull/5441
* [Docs] Fix DWS/Kueue title and URL by @Michaelvll in https://github.com/skypilot-org/skypilot/pull/5443
* release pipeline trigger filter based on name by @zpoint in https://github.com/skypilot-org/skypilot/pull/5367
* [Doc] Add runpod credentials setup for API server by @aylei in https://github.com/skypilot-org/skypilot/pull/5433
* Fix flaky of `test_multi_echo` -- change sshd config to support large number of jobs by @zpoint in https://github.com/skypilot-org/skypilot/pull/5323
* Support launch controller and jobs on different cloud for smoke test by @zpoint in https://github.com/skypilot-org/skypilot/pull/5435
* (Helm chart) Add configurable ingress host by @turtlebasket in https://github.com/skypilot-org/skypilot/pull/5452
* [Example] AWS EFA Example  by @KeplerC in https://github.com/skypilot-org/skypilot/pull/5318
* [runpod] preserve docker configured environment variables by @SeungjinYang in https://github.com/skypilot-org/skypilot/pull/5451
* [Docs] Clarify Nebius credential setup by @Michaelvll in https://github.com/skypilot-org/skypilot/pull/5298
* [k8s] gpu bin packing via affinity by @SeungjinYang in https://github.com/skypilot-org/skypilot/pull/5423
* [docs] leave in instructions to deal with omegaconf until next stable release by @SeungjinYang in https://github.com/skypilot-org/skypilot/pull/5460
* [k8s] CPU only jobs to prefer nodes without GPUs by @SeungjinYang in https://github.com/skypilot-org/skypilot/pull/5357
* Fix failure of `test_kubernetes_context_failover` by @zpoint in https://github.com/skypilot-org/skypilot/pull/5455
* Fix flaky of `test_cancel_launch_and_exec_async` by @zpoint in https://github.com/skypilot-org/skypilot/pull/5456
* [GCP] Remap series-specific disk types by @JiangJiaWei1103 in https://github.com/skypilot-org/skypilot/pull/5457
* add task envs to event_callback by @ggilley in https://github.com/skypilot-org/skypilot/pull/5474
* [Nebius] Conditionally mount AWS credential files for Nebius profile by @SalikovAlex in https://github.com/skypilot-org/skypilot/pull/5464
* Remove upper limit on urllib3 version by @vnavkal in https://github.com/skypilot-org/skypilot/pull/5469
* fix controller cluster name breaking by @cg505 in https://github.com/skypilot-org/skypilot/pull/5482
* [UX][k8s] backwards compatibility for k8s show-gpus by @kyuds in https://github.com/skypilot-org/skypilot/pull/5488
* [UX] Make `sky check` parallel by @kyuds in https://github.com/skypilot-org/skypilot/pull/5483
* reload AWS_SESSION_TOKEN and KUBECONFIG on local API server by @cg505 in https://github.com/skypilot-org/skypilot/pull/5478
* [jobs/serve] validate controller name before updating value by @cg505 in https://github.com/skypilot-org/skypilot/pull/5486
* [Nebius] Add support config file and remove hardcode by @SalikovAlex in https://github.com/skypilot-org/skypilot/pull/5463
* [k8s] do not consider nodes with exact cpu/mem requirements by @SeungjinYang in https://github.com/skypilot-org/skypilot/pull/5481
* [docs] snippet on multi node jobs in k8s by @SeungjinYang in https://github.com/skypilot-org/skypilot/pull/5495
* [k8s] fix helm chart deployment of API server by @SeungjinYang in https://github.com/skypilot-org/skypilot/pull/5507
* Release pipeline refactor - automated release by @zpoint in https://github.com/skypilot-org/skypilot/pull/5470
* Use more specific header name by @colinjc in https://github.com/skypilot-org/skypilot/pull/5515
* remove API version bump, add bw compatibility code by @SeungjinYang in https://github.com/skypilot-org/skypilot/pull/5522
* chore: minor fix to api server documentation by @SeungjinYang in https://github.com/skypilot-org/skypilot/pull/5512
* Reload AWS default profile for local API server by @aylei in https://github.com/skypilot-org/skypilot/pull/5511
* [Examples] Llama 3.1 lora finetuning torch version pin by @romilbhardwaj in https://github.com/skypilot-org/skypilot/pull/5531
* [Docker] Add private docker registry by @Michaelvll in https://github.com/skypilot-org/skypilot/pull/5526
* Add optional version parameter to docker build pipeline to prevent version mismatch by @zpoint in https://github.com/skypilot-org/skypilot/pull/5525
* [GCP] Correctly delete cpu mig instance by @Michaelvll in https://github.com/skypilot-org/skypilot/pull/5524
* [Docs] move `ordered` to a section by @zpoint in https://github.com/skypilot-org/skypilot/pull/5540
* [Docs] Add a few tutorial links. by @concretevitamin in https://github.com/skypilot-org/skypilot/pull/5546
* [Docs] Minor fix to the container tabs by @Michaelvll in https://github.com/skypilot-org/skypilot/pull/5551
* [k8s] New UX for`show-gpus`, and add back the 0 GPU nodes by @Michaelvll in https://github.com/skypilot-org/skypilot/pull/5490
* [Docs] Fix quote for URL by @Michaelvll in https://github.com/skypilot-org/skypilot/pull/5516
* [LLM] Update gemma example by @Michaelvll in https://github.com/skypilot-org/skypilot/pull/5071
* [Dashboard] Add vscode connection snippet by @romilbhardwaj in https://github.com/skypilot-org/skypilot/pull/5552
* [Nebius] Add support for internal IP usage in Nebius configurations by @SalikovAlex in https://github.com/skypilot-org/skypilot/pull/5513
* [docs] add docs for setting up TCP IAP tunneling by @cg505 in https://github.com/skypilot-org/skypilot/pull/5549
* Optimize the throughput of websocket proxy by @aylei in https://github.com/skypilot-org/skypilot/pull/5539
* [API server] handle logs request in coroutine by @aylei in https://github.com/skypilot-org/skypilot/pull/5366
* [Doc] Add link to local kind deployment doc by @DanielZhangQD in https://github.com/skypilot-org/skypilot/pull/5322
* [Doc] Add helm values spec reference by @aylei in https://github.com/skypilot-org/skypilot/pull/5415
* [Core] Support gcp gpu direct tcpx by @DanielZhangQD in https://github.com/skypilot-org/skypilot/pull/5553
* [docker] update docker login creds when relaunching cluster by @cg505 in https://github.com/skypilot-org/skypilot/pull/5559
* [Nebius] add ~/.ssh/sky-cluster-key by @SalikovAlex in https://github.com/skypilot-org/skypilot/pull/5564
* [GCP] Add H200 to catalog by @Michaelvll in https://github.com/skypilot-org/skypilot/pull/5101
* Revert "[API server] handle logs request in coroutine" by @aylei in https://github.com/skypilot-org/skypilot/pull/5580
* [Doc] Update docs for efa by @DanielZhangQD in https://github.com/skypilot-org/skypilot/pull/5471
* Add helm release in release pipeline by @aylei in https://github.com/skypilot-org/skypilot/pull/5568
* [optimizer] chore: only check clouds again if there is a cloud to check by @SeungjinYang in https://github.com/skypilot-org/skypilot/pull/5541
* [API server] handle logs request in coroutine by @aylei in https://github.com/skypilot-org/skypilot/pull/5582
* remove mypy exclusion by @zpoint in https://github.com/skypilot-org/skypilot/pull/5528
* [UX] Update dashboard routes and verify test package before releasing by @DanielZhangQD in https://github.com/skypilot-org/skypilot/pull/5394
* [Core] Add backward compatibility for instance naming in k8s by @DanielZhangQD in https://github.com/skypilot-org/skypilot/pull/5581
* allow specifying autostop in resources by @cg505 in https://github.com/skypilot-org/skypilot/pull/5577
* Bump image tag in helm default values by @aylei in https://github.com/skypilot-org/skypilot/pull/5584
* [Build] Fix check script in build actions by @DanielZhangQD in https://github.com/skypilot-org/skypilot/pull/5594
* Fix `test_gcp_disk_tier` by @zpoint in https://github.com/skypilot-org/skypilot/pull/5592
* Release branch name fix by @zpoint in https://github.com/skypilot-org/skypilot/pull/5595
* Release 0.9.3 by @github-actions in https://github.com/skypilot-org/skypilot/pull/5598

## New Contributors
* @turtlebasket made their first contribution in https://github.com/skypilot-org/skypilot/pull/5452
* @ggilley made their first contribution in https://github.com/skypilot-org/skypilot/pull/5474
* @vnavkal made their first contribution in https://github.com/skypilot-org/skypilot/pull/5469

**Full Changelog**: https://github.com/skypilot-org/skypilot/compare/v0.9.2...v0.9.3