v0.10.0

# SkyPilot v0.10.0: Enterprise-readiness with SSO, dashboard, external PostgreSQL, workspaces and more

We are excited to announce [SkyPilot 0.10](https://github.com/skypilot-org/skypilot/releases/tag/v0.10.0)! This release is the largest release by far, bringing enterprise-ready features including API server deployment in production with SSO, feature-rich dashboard, external PostgreSQL, workspace isolation and graceful upgrade, automatic network setup, and SSH Node Pools.

Get it now:

```bash
pip install -U skypilot
```

- [Highlights](#highlights)
  - [API server deployment in production](#api-server-deployment-in-production)
    - [Single Sign-On (SSO) and service account (Okta, Google Workspace, etc.)](#single-sign-on-sso-and-service-account-okta-google-workspace-etc)
    - [Feature-rich Dashboard](#feature-rich-dashboard)
    - [External PostgreSQL for API server in production](#external-postgresql-for-api-server-in-production)
    - [Workspaces: isolation and declarative configuration for teams](#workspaces-isolation-and-declarative-configuration-for-teams)
    - [Graceful upgrade of API server](#graceful-upgrade-of-api-server)
  - [Automatic high performance network setup (`network_tier: best`)](#automatic-high-performance-network-setup-network_tier-best)
  - [SSH Node Pools: bring your own machines](#ssh-node-pools-bring-your-own-machines)
  - [New clouds](#new-clouds)
- [What's new](#whats-new)
  - [CLI \& Core interfaces](#cli--core-interfaces)
  - [Authentication \& Security](#authentication--security)
  - [Dashboard](#dashboard)
  - [API Server](#api-server)
  - [Enterprise-ready Kubernetes cluster support](#enterprise-ready-kubernetes-cluster-support)
  - [SSH Node Pools](#ssh-node-pools)
  - [Managed Jobs](#managed-jobs)
  - [Backend](#backend)
  - [Cloud Support](#cloud-support)
  - [Documentation and Examples](#documentation-and-examples)
- [⚠️ Deprecations and removals](#️-deprecations-and-removals)
  - [Deprecated](#deprecated)
  - [Removed](#removed)
- [Migration guide](#migration-guide)
  - [CLI Changes](#cli-changes)
  - [API server deployment](#api-server-deployment)
  - [Jobs controller](#jobs-controller)
- [Get started today](#get-started-today)

## Highlights

### API server deployment in production


#### [Single Sign-On (SSO) and service account](https://docs.skypilot.co/en/latest/reference/auth.html#sso-recommended) (Okta, Google Workspace, etc.)

SkyPilot now integrates with enterprise SSO providers like [Okta](https://skypilot.readthedocs.io/en/latest/reference/api-server/examples/api-server-auth-proxy.html#oauth2-proxy-okta), [Google Workspace](https://skypilot.readthedocs.io/en/latest/reference/api-server/examples/api-server-auth-proxy.html#oauth2-proxy-google), enabling secure authentication with automatic account creation and access control.

<img width="1675" height="1067" alt="image" src="https://github.com/user-attachments/assets/6bb3b043-c5b9-4054-af39-7468f1ccf0b6" />

Log in to the API server with SSO enabled:

```console
$ sky api login -e https://skypilot.example.com
A web browser has been opened to http://skypilot.example.com/token. Please continue the login in the web browser.
To manually copy the token, press ctrl+c.

Logged into SkyPilot API server at: http://skypilot.example.com
└── Dashboard: http://skypilot.example.com/dashboard
```

Users authenticate via their organization's SSO provider, and their identities are automatically tracked across all SkyPilot resources.


#### [Feature-rich Dashboard](https://docs.skypilot.co/en/latest/getting-started/quickstart.html#access-the-dashboard)

SkyPilot dashboard now includes significant amount of new features:
* See all infrastructure available in one page
* Edit your SkyPilot config in dashboard
* See and manage all your users in an organization
* Find your GPU metrics in dashboard
* Check your YAML/entrypoint/git commit hash for jobs
* Find more features in [Dashboard](#dashboard) section below.

<img width="1675" height="1067" alt="image" src="https://github.com/user-attachments/assets/c98dcbc6-a6c9-4c6e-806d-9694087e89b5" />

<img width="2948" height="1779" alt="infra-gpu" src="https://github.com/user-attachments/assets/0d2cfcd4-78ac-4304-831f-30ba01967310" />


#### [External PostgreSQL for API server in production](https://docs.skypilot.co/en/latest/reference/api-server/api-server-admin-deploy.html#api-server-persistence-db)

SkyPilot 0.10 adds support for persisting API server state to an external PostgreSQL database, enabling high availability and disaster recovery for production deployments.

Configure your deployment to use a managed database service (e.g., AWS RDS, Cloud SQL) to ensure your cluster and job state survive API server restarts or migrations.

```yaml
db: postgresql://myusername:mypassword@hostname:5432/database
```


#### [Workspaces: isolation and declarative configuration for teams](https://docs.skypilot.co/en/latest/admin/workspaces.html)

Workspaces provides a declarative way to define isolated environments with custom cloud configurations for different teams or projects.

Configure workspaces to control which teams can access which infrastructure:

```yaml
# API server config
workspaces:
  research-private:
    private: true
    allowed_users:
      - alice@skypilot.co
      - mike@skypilot.co
    gcp:
      project_id: skypilot-research-private
    aws:
      disabled: true
  ml-team:
    gcp:
      project_id: skypilot-ml-team-prod

```

Teams simply set their active workspace to use their workspace configuration:

```yaml
# In team's .sky.yaml
active_workspace: ml-team
```

<img width="1675" height="1067" alt="image" src="https://github.com/user-attachments/assets/76fb347f-bfe5-41f4-a890-8b9b1a1ee021" />


#### [Graceful upgrade of API server](https://docs.skypilot.co/en/latest/reference/api-server/api-server-upgrade.html#graceful-upgrade)

SkyPilot 0.10 introduces robust graceful upgrade of API server:

* Clients automatically wait for an upgrade and retries
* Future compatibility across minor/major versions

<p align="center">
  <img src="https://i.imgur.com/jUjXu0J.gif" alt="Graceful upgrade demo" />
</p>


### [Automatic high performance network setup](https://docs.skypilot.co/en/latest/reference/training-guide.html#use-high-performance-networking) (`network_tier: best`)

SkyPilot v0.10.0 can now automatically configure high-performance network with a single `network_tier: best` config. Supported infra:

* Nebius VMs
* Nebius managed Kubernetes service
* GCP VMs
* Google Kubernetes Engine (GPUDirect-TCPX, GPUDirect-TCPXO, GCPDirect-RDMA for H100, and H200)


### [SSH Node Pools: bring your own machines](https://docs.skypilot.co/en/latest/reservations/existing-machines.html)

Turn your existing machines — on-premises servers, cloud reserved instances or even your personal workstation — into [SSH Node Pools](https://skypilot.readthedocs.io/en/latest/reservations/existing-machines.html) to run SkyPilot clusters and jobs on them.

Configure your machines in `~/.sky/ssh_node_pools.yaml`:

```yaml
# ~/.sky/ssh_node_pools.yaml
my-node-pool:
  hosts:
    - 1.2.3.4
    - 1.2.3.5
```

Deploy SkyPilot on them with a single command:

```bash
$ sky ssh up
$ sky launch --infra ssh/my-node-pool -- python train.py
```

Your machines now appear as infra choices alongside cloud providers, complete with GPU availability tracking and resource management.


### New clouds

* **Hyperbolic cloud** integration for cost-effective AI workloads (#5517)
* **Samsung Cloud Platform (SCP)** support for new provisioner interface in SkyPilot (#5587)


## What's new

### CLI & Core interfaces

* New `--infra` option to specify infrastructure instead of separate `--cloud/--region/--zone` flags (#5602, #5656, #5695, #5703)
  * Supports cloud providers: `--infra aws/us-west-2/us-west-2a`, `--infra aws/*/us-west-2b`
  * Kubernetes contexts: `--infra k8s/my-k8s-context`
  * SSH node pools: `--infra ssh/my-ssh-pool`
* **Secrets**: define secrets in your SkyPilot YAML for secure environment variable injection (#6015, #6025)
* GPU selection by memory size: `--gpus 80GB+` (#5948)
* Support units for disk, memory, and autostop specification (#5952, #6026)
* Improved error handling and stacktrace display with `SKYPILOT_DEBUG=1` (#6121)
* `sky cancel` now supports glob patterns for cluster names (#5933)
* Centralized log collection for tasks (#5992)
* CLI fixes and improvements (#5915, #5811, #6213, #5798, #5871, #5729, #5880, #5590, #6228, #6030)
* Core stability improvements (#6019, #5985, #5698, #5699, #5754, #5768, #5776, #5787, #5786, #5833, #5838, #5863, #5882, #6088, #6113)

### Authentication & Security

* **Single Sign-On (SSO) integration** with OAuth2 providers (#5640, #5641, #5651, #5684, #5717, #5721, #5758, #5759, #5781, #5817, #6164)
* **Service account support** for programmatic access (#6077)
* **Role-Based Access Control (RBAC)** (#5872, #5938)
* RESTful **Admin policies** for policy enforcement (#5927, #6089, #5940)
* Enhanced authentication security (#5816, #6119, #6037, #6141, #6152, #6120, #5877)
 
### Dashboard

* **User management and filtering** (#5936, #6078, #5708, #5722, #5997, #6183, #5935)
* **Cluster and jobs history tracking** (#6041, #5944)
* **New Infra page** (#5623, #5788)
* **Configuration editing** (#5748, #5770)
* **GPU and API server metrics** (#5907, #6116, #6092, #6139)
* **YAML/entrypoint/git commit hash display for clusters and jobs** (#5813, #5906, #5900)
* Real-time log streaming for managed jobs (#5808)
* Loading speed improvements (#5777, #5825)
* API server version display (#5784, #5827)
* Enhanced search and filtering capabilities (#5997, #6183)
* Display improvements (#5904, #5822, #6193)

### API Server

* **Workspace support** for team isolation (#5660, #5711, #5732, #5741, #5736, #5744, #5925)
  * Multi-tenant environments with workspace-specific configurations
  * Dashboard CRUD operations for workspace management
  * Workspace-aware cloud and infrastructure filtering
  * URL-based dashboard access with embedded credentials (#5925)
  * Support for setting private workspaces and user management (#5903)
* **PostgreSQL** for production deployments (#5707, #6052)
  * Database URL configuration via environment variables (#6052)
  * Connection pooling with NullPools for better performance (#6180)
* **API compatibility** between minor versions (#6105, #6174)
* **Performance optimizations**:
  * Uvloop integration for faster async operations (#6175)
  * Request cleanup and retention policies (#6157, #6173)
  * Database connection pooling improvements (#6180)
  * Increased file limits to system hard limits (#6131)
  * Improved process management with current Python path (#6106)
  * Log streaming buffer improvements (#5702)
  * Enhanced log streaming with buffering and test output (#5864)
  * Worker process retry optimization (#5556)
* **Enhanced API functionality**:
  * Jobs/queue filtering capabilities (#5848)
* **API server deployment**:
  * Multiple deployment environments support (#5563)
  * **Graceful API server upgrades** with automatic client retry (#6048)
  * CloudFlare proxy support (#5643)
  * Helm chart updates and improvements (#5710, #5713, #5717, #5721, #5754, #5758, #5759, #5760)
  * Persistent kubeconfig support for HA deployments (#5742)
  * Extra init containers support for Helm deployments (#5918)
  * Server-side configuration storage in database (#5897)
  * **Customizable ingress class** for flexible deployment (#6055)
  * API version management and compatibility (#5866)
  * Test coverage and robustness improvements (#5924, #6187)
* **Admin policies** enhancements (#5625, #5634)

### Enterprise-ready Kubernetes cluster support

SkyPilot 0.10 brings major enhancements to Kubernetes support:

* **Multi-cluster Kubernetes configuration** allowing simultaneous management of multiple K8s clusters (#6083)
* **Native support for Kueue** (#5498, #6023, #6114)
* **Enhanced GPU and resource management**:
  * GPU isolation fixes when no GPUs are requested (#5823)
  * Better GPU label formatting for various providers (#5926)
  * **Improved TPU name handling on GKE for non-k8s TPU types** (e.g. tpu-v6e-8) (#5480)
  * PVC volume support for persistent storage (#6046, #6087, #6097)
  * Exec-based authentication support (#5453)
  * **Robust authentication** with exec-based auth and improved kubeconfig handling (#5453, #6071, #5975)
* Improved cluster reliability with better EKS support and setup processes (#5780, #5812)
* **Robust authentication** with exec-based auth and improved kubeconfig handling (#5453, #6071, #5975)
* **Performance optimizations** including image pull policy updates and deployment improvements (#6154, #5977)
* Test coverage and stability improvements (#5818, #5922, #5934, #5939)


### SSH Node Pools

* **SSH node pool implementation** (#5548, #5692, #5735, #5687, #5692, #5697, #5693, #5728, #5705, #5739)
  * Bring your own infrastructure with existing machines
  * GPU discovery and resource tracking
  * Volume mounting support (#5892)
  * Dashboard integration with UX improvements (#5881)
* **SSH infrastructure management**:
  * Automatic SSH key distribution and management (#5782)
  * TCP forwarding enablement (#5683, #5685)
  * Improved error handling and logging (#5735)
  * Support for custom SSH configurations and identity files (#6005, #6003)
  * Enhanced host configuration change handling (#5901)
  * Fixed SSH cloud dependency installation on controllers (#5870)

### Managed Jobs

* **High Availability (HA) job controller** (#5820, #5835)
  * Failure recovery and automatic restart capabilities
  * Distributed job controller deployment
* **Job consolidation mode** (#6016, #6068)
  * Centralized job management for improved efficiency
  * Shared job controller across multiple users
  * File mount optimization for consolidation (#6132)
  * SQLAlchemy support for managed job state (#6068)
  * HA recovery script database storage (#6090)
  * Protection against local execution with consolidation enabled (#6208)
* **Enhanced job management**:
  * User filtering and workspace-aware job listing (#4787)
  * Improved job status tracking and error handling
  * Better log collection and streaming capabilities (#5992)
  * Support for cancelling jobs for all users (#6222)
* **SkyServe improvements** with enhanced exception handling (#5836)

### Backend

* **Database architecture improvements**:
  * SQLAlchemy migration for better database management (#5634, #5730)
  * Lazy database initialization for better performance (#5853, #5947, #5950, #5957, #5963, #5964)
  * **Standardized database initialization** for consistency (#5964)
  * Database migration fixes and improvements (#5831)
  * Race condition prevention in database operations (#5965, #6212, #6216, #6221)
* **Code structure improvements**:
  * Service catalog reorganization (#5840, #5868)
  * CLI code reorganization and client separation (#6028)
  * Configuration system improvements (#5869, #5878)
  * Global user state independence (#5874)
  * Token service optimization (#6225, #6227)
  * **Ephemeral configuration engine** for improved performance (#6135)
  * **Cluster management refactoring** for better code organization (#6156)
* **Dependency management**:
  * Passlib dependency management for authentication (#6024)
  * Dependencies updates (#5597, #5734, #5785, #5790)
  * PyOpenSSL version pinning for security (#5785, #5790)
  * JWT dependency fixes (#6123)

### Cloud Support

* **New cloud providers**:
  * **Hyperbolic cloud** integration for cost-effective AI workloads (#5517)
  * **Samsung Cloud Platform (SCP)** support for enterprise deployments (#5587)
* **Enhanced cloud features**:
  * High-speed network configuration automation (#5688)
  * Improved credential management and validation
  * Robust cloud provisioning (#5633)
* **Cloud-specific improvements**:
  * **GCP**: VPC support across projects, newer instance types, fetcher improvements (#5787)
  * **GCP**: DWS flex-start support (#6125, #6190, #6198, #6202)
  * **AWS**: Enhanced security group management, performance improvements (#6161)
  * **Nebius**: Credential support, filesystem integration, network optimization (#5576, #5658, #5772, #5752)
  * **GKE**: H200 GPU support, scale-to-zero autoscaling, label formatter fixes (#6198)
  * **Cudo**: Fixed integration and catalog update frequency (#6130, #6178)
* **Infrastructure management**:
  * Network infrastructure cleanup for GPU Direct configurations (#5565)
  * Multi-node cluster fixes for GKE DWS (#6202)

### Documentation and Examples

SkyPilot 0.10 includes comprehensive documentation improvements across all major features:

* **Documentation** enhancements (#5646, #5713, #5498, #6023, #5646, #5789, #5723, #5706, #5740, #5765, #5778, #5783, #5752, #5797, #5829, #5845, #5859, #5941, #5834, #6043, #6034, #6026, #6025, #6031, #6038, #5774, #6164, #6143, #6153, #6086, #6098, #6099, #6093, #6089, #6190, #6094, #6155, #6177, #6165)
* **Examples and guides**:
  * **Llama 4 finetuning example** (#5960, #6058)
  * High-performance networking examples for AI workloads (#6034)
  * **Falcon H1 model example** (#6042)

## ⚠️ Deprecations and removals

### Deprecated

* `--cloud/--region/--zone` flags have been deprecated in favor of `--infra`
  * Use `--infra aws/us-west-2` instead of `--cloud aws --region us-west-2`
  * The old flags will continue to work till 0.13.0 but are discouraged

### Removed

* Legacy benchmark code has been removed from the codebase (#6000)
* Some internal API compatibility with very old versions
* Non-functional CLI arguments have been removed (#5912)
* Legacy jobs controller dashboard functionality (#5917)

## Migration guide

SkyPilot 0.10.0 maintains backward compatibility with existing clusters and jobs. However, there are a few considerations:


### CLI Changes

* Consider migrating from `--cloud/--region/--zone` to the new `--infra` flag

### API server deployment

When upgrading from 0.9.x to 0.10.0, both API server and client need to be upgraded, i.e. API server 0.10.0 does not support 0.9.x clients.

### Jobs controller

* Upgrade your jobs controller with the new SkyPilot version: 

```bash
CONTROLLER_NAME=$(sky status | grep "sky-jobs-controller" | awk '{print $1}')
sky start -f $CONTROLLER_NAME
```


## Get started today

```bash
pip install -U skypilot
```

Or, upgrade your existing API server to 0.10.0 (see [upgrade guide](https://docs.skypilot.co/en/latest/reference/api-server/api-server-upgrade.html)):

```bash
NAMESPACE=skypilot
RELEASE_NAME=skypilot
VERSION=0.10.0

helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
  --set apiService.image=berkeleyskypilot/skypilot:$VERSION \
  --version $VERSION --devel --reuse-values
```

_Note: ` --set apiService.image=berkeleyskypilot/skypilot:$VERSION` is needed in case your previous API server was using the nightly build._

## Thanks to all contributors!

This release includes contributions from many new and returning contributors. Special thanks to everyone who helped make SkyPilot 0.10 possible!

New contributors: @lucamanolache, @ykocaogullar, @omahs, @kilavvy, @vtjl10, @mundaym, @leopardracer, @dhiaEddineRhaiem, @crStiv, @zhenjiasun, @AngadSethi, @davidknittel728, @bikramnehra, @hyoxt121, @turtlebasket, @clayrosenthal, @kevinmingtarja, and many others from the community who contributed through issues, discussions, and feedback.

Thanks to all contributors: @SeungjinYang, @zpoint, @Michaelvll, @aylei, @cg505, @romilbhardwaj, @DanielZhangQD, @kyuds, @concretevitamin, @cblmemo, @rohansonecha, @lucamanolache, @Maknee, @SalikovAlex, @kevinmingtarja, @turtlebasket, @JiangJiaWei1103, @AngadSethi, @zhenjiasun, @ykocaogullar, @vtjl10, @vnavkal, @omahs, @mundaym, @leopardracer, @kilavvy, @hyoxt121, @greendev0127, @ggilley, @funkypenguin, @dhiaEddineRhaiem, @crStiv, @colinjc, @clayrosenthal, @bikramnehra, @andylizf, @Kovbo, @KeplerC

---

_To stay updated, star and watch our [GitHub repo](https://github.com/skypilot-org/skypilot/), follow [@skypilot_org](https://twitter.com/skypilot_org), or join our [community Slack](https://slack.skypilot.co/)._ 

**Full Changelog**: https://github.com/skypilot-org/skypilot/commits/v0.10.0