v0.10.2

# SkyPilot v0.10.2: Enhanced observability, programmatic SDK, performance, and more

SkyPilot v0.10.2 brings values in cluster management, improved Kubernetes support, programmatic SDK, and numerous stability and performance enhancements for production use from teams with a large number of workloads.

Get it now with:
```
uv pip install "skypilot>=0.10.2"
```

## Highlights


### Cluster observability improvements

Get provisioning logs with the new `--provision` option for `sky logs` (#6638):

```bash
sky logs --provision <cluster-name>
```

Find the detailed reason for your cluster failure, e.g., OOM, with the cluster events (#6590, #6593, #6615, #6620, #6621, #6667, #6658, #6609, #6617):

![](https://i.imgur.com/uxdxZ8L.png)


### SDK: Process your logs while streaming

SkyPilot introduces a new `preload_content` option for `tail_logs` to enable processing logs while streaming.

```python
logs = sky.tail_logs(cluster_name, job_id, follow=True, preload_content=False)
for line in logs:
    if line is not None:
        if 'needle in the haystack' in line:
            print("found it!")
            break
logs.close()
```


### Performance Improvements

- **Job dashboard page loading speed improvement for 10k+ jobs** (#6714, #6652)
- **Volume mounting on Kubernetes from 2mins to seconds** (#6679)
- Increase the connection limit from a single client: 10 to 100 (#6782)
- **Speed up `sky down` for AWS clusters with exposed ports by 4x** (#6629, #6663, #6720)
- Fix the OOM issue with large file_mounts (#6865)
- **Enhanced SSH connection robustness** (#6715)



## What's New


### Infrastructure-specific Improvements

- **Kubernetes**:
  - **Label support for volumes** (#6696)
  - Better authentication handling for in-cluster services (#6600)
  - Proxy setup guide for Kubernetes (#6618)
  - Improved error handling: Enhanced failover error messages (#6613)
  - Enable write access to conda base environment (#6766)
  - Better GPU label detection with empty labels (#6859)
- **AWS**:
  - Customizable SSH users (#6625)
  - Root device name detection (#6644)
  - Support Rocky Linux (#6711)
  - Fix fractional GPU quantity for P5.48xlarge (#6722)
- **Nebius**:
  - **Memory configuration support** (#6832): Users can now properly specify memory requirements (e.g., 64GB instances)
  - More robust cluster status fetching (#6693, #6672)
  - Enhanced VRAM calculation (#6628)
  - Improved infinite waiting fixes (#6674)
- **RunPod**:
  - Docker image handling with any_of config (#6728)
  - Docker name conflict resolution (#6769)
- **Lambda**: Support B200 on Lambda (#6816)
- R2 mount issue resolution (#6662)


### Dashboard Enhancements

- **Managed Jobs**:
  - Download buttons for logs (#6689, #6757, #6758)
  - Fixed pipeline job ordering (#6703)
- **User Management**:
  - Allow editing allowed_users and private settings for workspaces (#6566)
- **Miscs**:
  - Fixed tour auto-start behavior (#6654)

### API Server Enhancements

- Robust jobs/serve controller against upgrades (#6779)
- Prometheus metrics in deployment mode (#6712)
- Docs for using Cloud SQL for states (#6587)
- **State management improvements**:
  - Unique constraint violation detection for PostgreSQL (#6773)
  - SQLAlchemy warning suppression (#6796)
  - Reduce DB creation operations (#6594)
- **Admin Policy & Permissions & RBAC**:
  - Apply admin policy for volumes (#6668, #6781, #6749)
  - Enhanced restful admin policy in testing (#6353, #6626)
  - Backward compatibility handling for admin policies (#6692)
  - Optimized permission module logging (#6805)



### UX/API Improvements

- **CLI Enhancements**:
  - Better managed job log tailing behavior (#6719)
  - Improved `--config` option documentation (#6794)
  - Better logs organization for downloading (#6795)
  - Warning for disk size specifications on Kubernetes (#6637, #6684)
  - Client version and commit info in `sky api info` (#6748)
  - Improved cluster event messaging and wording (#6686, #6688)
  - Better user feedback with spinner for delay messages (#6575)
  - Enhanced pod configuration validation (#6825)
- **SDK Improvements**:
  - **Improved transient failure handling** (#6808, #6807): More intelligent retry mechanisms
  - **Simpler log streaming utility** (#6750): More intuitive API for processing logs in real-time with iterator-based approach
  - Better response typing (#6659, #6718, #6527)
  - Exposed `sky.endpoints` function (#6599)
  - Enhanced decoder backward compatibility (#6810)
  - Comprehensive endpoints documentation (#6815)
- **Robustness and Performance Improvements**:
  - Fixed authentication with simplejson (#6698)
  - Reduced import statement overhead (#6641, #6645, #6648)
  - Optimized directory utilities (#6646)
  - Resolved API logging issues (#6619)
  - Enhanced retry logic for non-transient errors (#6844)
  - Better handling of missing pods with termination filtering (#6697)
  - Fixed various cluster name querying issues (#6616, #6624)
  - Improved job import handling in sky init (#6623)


### Testing and Infrastructure

- **Test Improvements**:
  - Automatic retry for core tests (#6710)
  - Expanded nightly build coverage (#6695)
  - Staggered test execution (#6732)
  - Remote server test support (#6819, #6823)
  - New smoke tests for provisioning logs (#6790, #6818)
  - Fixed endpoint comparison in unittests (#6578)
  - Fixed TPU test example failure (#6673)
  - Added Kubernetes volume merging unit tests (#6813)
  - Added support for custom cloud config files in smoke tests (#6851)
  - Enhanced serve status checks after termination (#6713)
  - Include Nebius VMs tests in CI/CD (#6835)
- **CI/CD Enhancements**:
  - PAT token for release publishing (#6612)
  - Helm lint in pre-commit hooks (#6798, #6803)
  - Fixed nightly build triggers (#6610)
- **Helm Charts & Deployment**:
  - Fixed RBAC template rendering issues (#6737, #6742)
  - Resolved ingress configuration for oauth in grafana (#6799)
  - Enhanced values.yaml documentation (#6800)
  - Better patch strategy for merging Kubernetes configs (#6653)

### Other feature improvements

- **SSH Node Pool**: region-specific launches (#6767)
- **SkyServe: GPU-aware Load Balancing** (#6147)
  SkyServe now intelligently scale based on GPU types. Set different QPS targets across heterogeneous GPUs:
  ```yaml
  load_balancing_policy: instance_aware_least_load
  replica_policy:
    target_qps_per_replica:
      "H100:1": 2.5    # H100 can handle 2.5 QPS
      "A100:1": 1.25   # A100 can handle 1.25 QPS
      "A10:1": 0.5     # A10 can handle 0.5 QPS
  ```

### Backend

- Alpha support for replacing skylet with grpc (#6771, #6754, #6788, #6574, #6640, #6643, #6649, #6850)
- Alpha support for Managed Jobs Pool (#6665, #6571, #6614, #6676, #6580, #6682, #6675, #6785, #6591, #6666)
- Enhanced event querying by cluster hash (#6617)
- Resolved unpickle issues with class location changes (#6731)
- Volume resolution before DAG application (#)
- Fixed log cancellation issues (#6801)

### Examples and Documentation

- RAG example improvements for embeddings (#6639)
- Kimi K2 multi-node serving example (#6706, #6723)
- Enhanced autostop documentation (#6577)
- Two-hop file transfer guide for managed jobs (#6576)
- SDK examples in Quick Start (#6561, #6597)
- Polished landing page design (#6756)
- Miscs (#6741, #6827, #6828, #6586, #6774)




## Contributors

We thank all contributors who made this release possible!

**New Contributors**: @miltava, @tomzx, @webconn, @hongsu-moreh, @ibpark-moreh, @nathan-liner

**All Contributors**: @DanielZhangQD, @cblmemo, @kyuds, @SeungjinYang, @lloyd-brown, @romilbhardwaj, @rohansonecha, @zpoint, @aylei, @SalikovAlex, @Michaelvll, @kevinmingtarja, @miltava, @Maknee, @cg505, @tomzx, @concretevitamin, @webconn, @sethkimmel3, @andylizf, @hongsu-moreh, @ibpark-moreh, @lucamanolache, @nathan-liner


Special thanks to the community for bug reports, feature requests, and pull requests that helped improve SkyPilot!

## Full Changelog

For a complete list of changes, see the [commit history](https://github.com/skypilot-org/skypilot/compare/v0.10.1...v0.10.2).