v0.14.0

# SkyPilot v0.14.0: Dynamic Job Groups, Multi-User Slurm, AWS Neuron, Read-Only Workspaces, and More

SkyPilot v0.14.0 introduces **dynamic job groups**, letting a running job group launch more managed jobs that are tracked and cancelled with it, and makes **Slurm a multi-user backend**: jobs are submitted as the authenticated user, with per-user QoS/account settings and stop/start for container clusters. This release also adds **AWS Trainium/Inferentia (Neuron)** on Kubernetes, **read-only visibility for private workspaces**, and a **conda-free default image**.

Get it now with:

```bash
uv pip install "skypilot[all]==0.14.0"
```

Or, upgrade your team SkyPilot API server:

```bash
NAMESPACE=skypilot
RELEASE_NAME=skypilot
VERSION=0.14.0

helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
  --set apiService.image=berkeleyskypilot/skypilot:$VERSION \
  --version $VERSION --devel --reuse-values
```

**Deprecations & Breaking Changes**:

- **Conda is no longer installed by default** ([#10132](https://github.com/skypilot-org/skypilot/pull/10132)). `provision.install_conda` now defaults to `false`, and conda has been removed from the default Kubernetes images, which instead ship system Python plus a default user venv (`~/sky-user-env`). Set `provision.install_conda: true` to opt back in; the opt-in install now uses a current Miniforge with conda-forge as the only channel.
- **Job Groups enforce in-group networking by default** ([#10297](https://github.com/skypilot-org/skypilot/pull/10297), [#10330](https://github.com/skypilot-org/skypilot/pull/10330)). Groups whose tasks cannot be co-located now fail at submission instead of being silently spread across clusters, and networking setup failures fail the job. Set `inter_connection: false` on the group to allow cross-cluster placement.
- **Tighter API authorization**: request listing and tracking endpoints are scoped to the calling user ([#10274](https://github.com/skypilot-org/skypilot/pull/10274), [#10278](https://github.com/skypilot-org/skypilot/pull/10278)), `sky debug-dump` is admin-only ([#10277](https://github.com/skypilot-org/skypilot/pull/10277)), download paths are authorized against the user's own roots ([#10683](https://github.com/skypilot-org/skypilot/pull/10683)), principals with an unrecognized role are denied ([#10501](https://github.com/skypilot-org/skypilot/pull/10501)), and non-admins can no longer grant the admin role via service accounts ([#9932](https://github.com/skypilot-org/skypilot/pull/9932), [#10482](https://github.com/skypilot-org/skypilot/pull/10482)).
- **`rbac.restrict_config_to_admins` now defaults to `true`** ([#10042](https://github.com/skypilot-org/skypilot/pull/10042)): users with the `user` role can no longer read the API server config. Set it to `false` to restore the previous behavior.
- Kubernetes: use `disk_size` instead of the `ephemeral_storage` field ([#9273](https://github.com/skypilot-org/skypilot/pull/9273)).
- Async request submission no longer prints the API stream link ([#10081](https://github.com/skypilot-org/skypilot/pull/10081)).

## Highlights

### [New] [Dynamic Job Groups](https://docs.skypilot.co/en/latest/examples/job-groups.html)

A task in a job group can now launch further managed jobs with `sky jobs launch` or `sky.jobs.launch`, and they **attach to the group automatically** as *dynamic tasks* ([#10698](https://github.com/skypilot-org/skypilot/pull/10698), [#10701](https://github.com/skypilot-org/skypilot/pull/10701), [#10702](https://github.com/skypilot-org/skypilot/pull/10702), [#10725](https://github.com/skypilot-org/skypilot/pull/10725), [#10734](https://github.com/skypilot-org/skypilot/pull/10734), [#10736](https://github.com/skypilot-org/skypilot/pull/10736), [#10732](https://github.com/skypilot-org/skypilot/pull/10732)). Dynamic tasks are listed under the group in `sky jobs queue` and the dashboard, numbered after the group's own tasks, addressed like any other task (`sky jobs logs 122 3`, `sky jobs cancel 122 --task 3`), and cancelled together with the group. A common use is an eval watcher that launches one evaluation job per checkpoint while the trainer keeps training:

```yaml
---
name: train-and-eval
execution: parallel
primary_tasks: [trainer, eval-watcher]
---
name: trainer
resources:
  accelerators: H100:8
run: python train.py --checkpoint-dir /checkpoints
---
name: eval-watcher
resources:
  cpus: 2
run: |
  for ckpt in $(python watch_checkpoints.py /checkpoints); do
    sky jobs launch -y -d -n "eval-$ckpt" eval.yaml --env CKPT=$ckpt
  done
```

```console
$ sky jobs queue
ID    TASK  NAME              ...  STATUS
122   -     train-and-eval    ...  RUNNING
 ↳    0     trainer [P]       ...  RUNNING
 ↳    1     eval-watcher [P]  ...  RUNNING
 ↳    2     eval-step-1       ...  SUCCEEDED
 ↳    3     eval-step-2       ...  RUNNING
 ↳    4     eval-step-3       ...  RUNNING
```

Job groups also gain an `inter_connection: true | false` knob ([#10297](https://github.com/skypilot-org/skypilot/pull/10297)): networking is a real guarantee when enabled, and `false` skips the networking machinery entirely so tasks can be placed across clusters or clouds.

### [New] Multi-User Slurm

Slurm clusters shared through one API server can now run each user's work **under their own Unix identity** ([#10294](https://github.com/skypilot-org/skypilot/pull/10294), [#10477](https://github.com/skypilot-org/skypilot/pull/10477), [#10685](https://github.com/skypilot-org/skypilot/pull/10685), [#10686](https://github.com/skypilot-org/skypilot/pull/10686)). With `slurm.submit_as_user`, provisioning, `srun`, rsync, cancellation, and SSH all run as the mapped user, so Slurm accounting, fairshare, and file permissions apply per person. Service accounts default to their creator's identity. A new `slurm.quota` block maps to `sbatch --qos` / `--account` and resolves per workspace ([#10680](https://github.com/skypilot-org/skypilot/pull/10680), [#10631](https://github.com/skypilot-org/skypilot/pull/10631)).

```yaml
slurm:
  submit_as_user: true
  username_map:
    jane.doe@example.com: jdoe
    inference-prod: inference-svc
  quota:
    queue: normal           # -> #SBATCH --qos=normal
    account: pre-training   # -> #SBATCH --account=pre-training
```

Slurm container (Pyxis) clusters also support **`sky stop` / `sky start`**, snapshotting each rank's filesystem to shared storage and restoring it into a new allocation ([#10544](https://github.com/skypilot-org/skypilot/pull/10544)), plus **autostop** ([#10645](https://github.com/skypilot-org/skypilot/pull/10645)) and **host path volume mounts** ([#10438](https://github.com/skypilot-org/skypilot/pull/10438)).

### [New] AWS Trainium & Inferentia on Kubernetes

SkyPilot now schedules **AWS Neuron accelerators** (Trainium and Inferentia) on Kubernetes ([#10169](https://github.com/skypilot-org/skypilot/pull/10169)). It recognizes the `aws.amazon.com/neuron` resource, detects accelerator type and count from node labels (including Karpenter), adds the Neuron taint toleration, and lists Neuron devices in `sky show-gpus`.

```bash
sky launch --gpus Trainium2:16 --infra k8s
```

Other new hardware in this release: **GCP RTX PRO 6000 Blackwell (G4)** ([#9634](https://github.com/skypilot-org/skypilot/pull/9634)), **InfiniBand on Nebius B200/B300** ([#10369](https://github.com/skypilot-org/skypilot/pull/10369)), **OCI GB200/GB300 NCCL profiles** for `network_tier: best` ([#10204](https://github.com/skypilot-org/skypilot/pull/10204)), and **SR-IOV RDMA delivery** on Kubernetes via `kubernetes.rdma` ([#10554](https://github.com/skypilot-org/skypilot/pull/10554)).

### [New] [Read-Only Visibility for Private Workspaces](https://docs.skypilot.co/en/latest/admin/workspaces.html)

A private workspace can now let non-members **see it and its clusters and managed jobs (including logs) without being able to modify anything** ([#10228](https://github.com/skypilot-org/skypilot/pull/10228)). Members and admins keep full access. The same change hardens per-resource write isolation so mutating an existing cluster is always gated by the cluster's own workspace.

```yaml
workspaces:
  private-ws:
    private: true
    read_access: all        # default: allowed_users
    allowed_users:
      - alice@example.com
```

Set `workspace_config.read_access: all` to make this the org-wide default.

### Why Is My Job Not Running?

Managed jobs now explain themselves. `sky jobs queue -v` shows the latest launch-progress event for a waiting job, such as a Slurm pending reason or a Kubernetes image pull ([#10692](https://github.com/skypilot-org/skypilot/pull/10692)), and, with API server metrics enabled, SkyPilot exports **where a job's start-up time went** as metrics broken down by stage (controller queue, provisioning, node start-up, setup) ([#10638](https://github.com/skypilot-org/skypilot/pull/10638)). Event logs also record who requested a cluster stop/termination or job cancellation ([#10621](https://github.com/skypilot-org/skypilot/pull/10621), [#10619](https://github.com/skypilot-org/skypilot/pull/10619), [#10661](https://github.com/skypilot-org/skypilot/pull/10661)).

```console
$ sky jobs queue -v
ID     NAME   STATUS    ...  DETAILS
56266  train  STARTING  ...  Launching (pending: Resources; partition: dev)
```

---

## What's new: compared to v0.13.0

### API Server

- **Run multiple isolated local API servers on one machine** via `SKYPILOT_API_SERVER_LOCAL_PORT` / `sky api start --port` and `SKY_RUNTIME_DIR` ([#10213](https://github.com/skypilot-org/skypilot/pull/10213)); `SKYPILOT_DISABLE_LOCAL_API_SERVER` blocks local server autostart ([#10093](https://github.com/skypilot-org/skypilot/pull/10093)).
- **The service account token API accepts a role** for new tokens ([#10115](https://github.com/skypilot-org/skypilot/pull/10115)); throttled `last_used_at` writes ([#10391](https://github.com/skypilot-org/skypilot/pull/10391)).
- Grant private-workspace access to users added to `allowed_users` before their first login ([#10079](https://github.com/skypilot-org/skypilot/pull/10079)); surface the resolved `active_workspace` to admin policies ([#10241](https://github.com/skypilot-org/skypilot/pull/10241)).
- Refuse uploads that cannot fit on the server's disk ([#10714](https://github.com/skypilot-org/skypilot/pull/10714)); process client task YAMLs in memory ([#10435](https://github.com/skypilot-org/skypilot/pull/10435)); move chunked-upload work off the event loop ([#10556](https://github.com/skypilot-org/skypilot/pull/10556)).
- Request lifecycle fixes: mark a request FAILED when enqueue fails ([#10617](https://github.com/skypilot-org/skypilot/pull/10617)), fail `/api/get` for requests interrupted by a restart ([#10257](https://github.com/skypilot-org/skypilot/pull/10257)), and reclaim orphaned request logs ([#10675](https://github.com/skypilot-org/skypilot/pull/10675), [#10657](https://github.com/skypilot-org/skypilot/pull/10657)).
- Opt-in executor pool sizing that accounts for server and controller memory (`SKYPILOT_MEMORY_AWARE_WORKER_SIZING`) ([#10550](https://github.com/skypilot-org/skypilot/pull/10550)); fixed retention-daemon cadence ([#10538](https://github.com/skypilot-org/skypilot/pull/10538)) and hourly requests GC ([#10555](https://github.com/skypilot-org/skypilot/pull/10555)).
- Security: never regenerate the JWT signing secret on a DB read failure ([#10566](https://github.com/skypilot-org/skypilot/pull/10566)); redact credentials from config debug logs ([#10511](https://github.com/skypilot-org/skypilot/pull/10511)) and basic-auth passwords in printed URLs ([#10286](https://github.com/skypilot-org/skypilot/pull/10286)); parameterize request-status queries ([#10273](https://github.com/skypilot-org/skypilot/pull/10273)).
- Helm chart: `storage.existingClaim` to attach a pre-provisioned PVC ([#10085](https://github.com/skypilot-org/skypilot/pull/10085), [#10086](https://github.com/skypilot-org/skypilot/pull/10086)), `apiService.host` to set the bind interface ([#10050](https://github.com/skypilot-org/skypilot/pull/10050)), configurable clients-dir subPath ([#10101](https://github.com/skypilot-org/skypilot/pull/10101)).
- Launches blocked on a cluster lock held by another operation release their executor worker while they wait, and the client shows what they are waiting on ([#10210](https://github.com/skypilot-org/skypilot/pull/10210), [#10291](https://github.com/skypilot-org/skypilot/pull/10291), [#10013](https://github.com/skypilot-org/skypilot/pull/10013), [#10534](https://github.com/skypilot-org/skypilot/pull/10534)).
- Auth runs on its own executor with bounded database lookups ([#10313](https://github.com/skypilot-org/skypilot/pull/10313), [#10373](https://github.com/skypilot-org/skypilot/pull/10373), [#10185](https://github.com/skypilot-org/skypilot/pull/10185)); idle-in-transaction time is bounded on PostgreSQL ([#10687](https://github.com/skypilot-org/skypilot/pull/10687)).
- Configurable state-DB connection pool size ([#10791](https://github.com/skypilot-org/skypilot/pull/10791)).
- Optional `SO_REUSEPORT` for spreading connections across API server workers ([#10016](https://github.com/skypilot-org/skypilot/pull/10016)).

### Observability & Metrics

- **New provisioning, pending-time, and availability metrics** with matching Grafana panels and 30-day availability tracking ([#10625](https://github.com/skypilot-org/skypilot/pull/10625)).
- Managed-job recovery gauges ([#10186](https://github.com/skypilot-org/skypilot/pull/10186)), `/api/get` latency by request name ([#10349](https://github.com/skypilot-org/skypilot/pull/10349)), client-side DB instrumentation ([#10525](https://github.com/skypilot-org/skypilot/pull/10525)), SQLite size and local disk metrics ([#10363](https://github.com/skypilot-org/skypilot/pull/10363), [#10674](https://github.com/skypilot-org/skypilot/pull/10674)), SSH round-trip timing ([#10656](https://github.com/skypilot-org/skypilot/pull/10656)), and event-loop stall attribution ([#10523](https://github.com/skypilot-org/skypilot/pull/10523)).
- `/metrics` runs on its own event loop, stays responsive when a collector hangs, and caches the multiprocess merge ([#10503](https://github.com/skypilot-org/skypilot/pull/10503), [#10377](https://github.com/skypilot-org/skypilot/pull/10377), [#10573](https://github.com/skypilot-org/skypilot/pull/10573), [#10575](https://github.com/skypilot-org/skypilot/pull/10575)).
- Slurm GPU metrics federated via the login node, with `/federate` scoping for a shared Prometheus ([#10572](https://github.com/skypilot-org/skypilot/pull/10572), [#10603](https://github.com/skypilot-org/skypilot/pull/10603)).

### Kubernetes

- **AWS Neuron (Trainium/Inferentia) support** ([#10169](https://github.com/skypilot-org/skypilot/pull/10169)) and **SR-IOV RDMA** via `kubernetes.rdma` ([#10554](https://github.com/skypilot-org/skypilot/pull/10554)), with a `pod_config` opt-out from OCI RoCE host networking ([#10546](https://github.com/skypilot-org/skypilot/pull/10546)).
- **EFA improvements**: co-locate multi-node EFA jobs in one AZ ([#10054](https://github.com/skypilot-org/skypilot/pull/10054)) and derive EFA count on scale-from-zero clusters ([#10053](https://github.com/skypilot-org/skypilot/pull/10053)).
- **Kueue**: don't count admission wait against `provision_timeout` and show when a launch is waiting for queue admission ([#10252](https://github.com/skypilot-org/skypilot/pull/10252)), and pools now work under Kueue quota ([#10718](https://github.com/skypilot-org/skypilot/pull/10718)).
- **Clearer failures**: fail on persistent volume mount failures instead of hanging ([#10218](https://github.com/skypilot-org/skypilot/pull/10218)), detect PVCs that will not bind ([#10440](https://github.com/skypilot-org/skypilot/pull/10440)), bound pod wait loops and explain missing pods ([#10460](https://github.com/skypilot-org/skypilot/pull/10460), [#10272](https://github.com/skypilot-org/skypilot/pull/10272)), surface which node a terminated cluster lost ([#10233](https://github.com/skypilot-org/skypilot/pull/10233)), and recommend `set_pod_resource_limits` after an OOM ([#10551](https://github.com/skypilot-org/skypilot/pull/10551)).
- `sky local up --num-nodes` for multi-node local clusters ([#10138](https://github.com/skypilot-org/skypilot/pull/10138)); package-manager-agnostic bootstrap and graceful Docker CLI skip on non-Debian images ([#10077](https://github.com/skypilot-org/skypilot/pull/10077), [#10382](https://github.com/skypilot-org/skypilot/pull/10382)).
- Report node GPU total from capacity with allocatable alongside ([#10395](https://github.com/skypilot-org/skypilot/pull/10395)); display actual pod requests when they diverge from the spec ([#10214](https://github.com/skypilot-org/skypilot/pull/10214)); check ephemeral storage when fitting instances ([#10131](https://github.com/skypilot-org/skypilot/pull/10131)).
- Let pods with DRA claims terminate gracefully ([#10751](https://github.com/skypilot-org/skypilot/pull/10751)); request gzip-compressed API responses ([#10153](https://github.com/skypilot-org/skypilot/pull/10153)); fixed a logging fork deadlock on Python < 3.13 ([#10104](https://github.com/skypilot-org/skypilot/pull/10104)) and multi-node rank crashes on multi-NIC / hostNetwork nodes ([#9924](https://github.com/skypilot-org/skypilot/pull/9924)).

### Volumes

- **Scoped auto-mounts**: `kubernetes.auto_mounts` entries take `scope: personal | workspace | global` ([#10411](https://github.com/skypilot-org/skypilot/pull/10411)), and the volume listing is filtered to the caller's workspaces ([#10531](https://github.com/skypilot-org/skypilot/pull/10531)).
- Refresh a volume's size from the cloud and show pending resizes ([#10549](https://github.com/skypilot-org/skypilot/pull/10549), [#10553](https://github.com/skypilot-org/skypilot/pull/10553)); show why a volume is not ready ([#10486](https://github.com/skypilot-org/skypilot/pull/10486)); fail early when an auto-mounted volume is not ready ([#10402](https://github.com/skypilot-org/skypilot/pull/10402)).
- Volume page filtering ([#10316](https://github.com/skypilot-org/skypilot/pull/10316)); don't fail launches when tagging AWS volumes is denied ([#10627](https://github.com/skypilot-org/skypilot/pull/10627)); validate volumes at creation ([#10513](https://github.com/skypilot-org/skypilot/pull/10513)).

### Managed Jobs & Serve

- **Dynamic job groups** and the **`inter_connection`** knob (see Highlights).
- **Server-side infra filtering** for `sky jobs queue` ([#10613](https://github.com/skypilot-org/skypilot/pull/10613)) and smaller payloads for field-filtered queries ([#10176](https://github.com/skypilot-org/skypilot/pull/10176)).
- Controller resilience: emergency recovery for unexpected controller errors ([#9868](https://github.com/skypilot-org/skypilot/pull/9868), [#10416](https://github.com/skypilot-org/skypilot/pull/10416)), retry transient DB and cluster-status errors ([#10655](https://github.com/skypilot-org/skypilot/pull/10655), [#10040](https://github.com/skypilot-org/skypilot/pull/10040)), fail deterministic task-prep errors fast ([#10326](https://github.com/skypilot-org/skypilot/pull/10326)), and fail jobs whose pool no longer exists ([#10203](https://github.com/skypilot-org/skypilot/pull/10203)).
- Exit code and log pointer in RECOVERING events ([#10256](https://github.com/skypilot-org/skypilot/pull/10256)); `--num-jobs` without a pool ([#10552](https://github.com/skypilot-org/skypilot/pull/10552)); fixed partial state writes on Job Group sync failure and parallelized group teardown ([#10415](https://github.com/skypilot-org/skypilot/pull/10415)).
- **Serve**: fixed `least_load` replica decrement and rotation ([#10445](https://github.com/skypilot-org/skypilot/pull/10445)).

### Core & Backend

- Launch failover stops trying further zones when a provisioner reports a request it can never serve, such as starting a legacy Nebius preemptible VM ([#10697](https://github.com/skypilot-org/skypilot/pull/10697)).
- **Region-free cost estimates** priced with the joint regional minimum ([#10199](https://github.com/skypilot-org/skypilot/pull/10199)).
- Extract external links (e.g. W&B) from job logs server-side ([#9934](https://github.com/skypilot-org/skypilot/pull/9934)); concise health-probe failures in unhealthy-cluster events ([#10201](https://github.com/skypilot-org/skypilot/pull/10201)).
- `sky debug-dump`: fast-fail unreachable kube contexts, parallel per-cluster collection, and bounded controller steps so a stuck controller cannot hang the dump ([#10041](https://github.com/skypilot-org/skypilot/pull/10041), [#10161](https://github.com/skypilot-org/skypilot/pull/10161), [#10068](https://github.com/skypilot-org/skypilot/pull/10068)).
- More robust runtime installs: fall back when an apt repository or distro mirror is unusable ([#10673](https://github.com/skypilot-org/skypilot/pull/10673), [#10512](https://github.com/skypilot-org/skypilot/pull/10512)).
- Fixed Ray job submit failures from a dashboard-agent port conflict ([#9856](https://github.com/skypilot-org/skypilot/pull/9856)); don't abort graceful teardown when command runners are unavailable ([#10604](https://github.com/skypilot-org/skypilot/pull/10604)); guard against a missing `~/.bashrc` and missing Git ([#10245](https://github.com/skypilot-org/skypilot/pull/10245), [#10230](https://github.com/skypilot-org/skypilot/pull/10230)).

### Infrastructure & Cloud Integrations

**Slurm**

- **Submit as authenticated users**, `username_map`, and `quota.queue` / `quota.account` (see Highlights).
- **Container cluster stop/start and autostop** ([#10544](https://github.com/skypilot-org/skypilot/pull/10544), [#10645](https://github.com/skypilot-org/skypilot/pull/10645), [#10652](https://github.com/skypilot-org/skypilot/pull/10652)) and **host path volume mounts** ([#10438](https://github.com/skypilot-org/skypilot/pull/10438)).
- GPU availability from node `AllocTRES`, CPU/memory allocation and usage in the Slurm node-info API, and batched inventory queries over one SSH connection ([#10285](https://github.com/skypilot-org/skypilot/pull/10285), [#10293](https://github.com/skypilot-org/skypilot/pull/10293), [#10425](https://github.com/skypilot-org/skypilot/pull/10425)); best-effort multi-cluster aggregation ([#10284](https://github.com/skypilot-org/skypilot/pull/10284), [#10289](https://github.com/skypilot-org/skypilot/pull/10289)).
- Run skylet under an allocation-owned keeper step ([#10516](https://github.com/skypilot-org/skypilot/pull/10516)); respect custom task CPU demands ([#10447](https://github.com/skypilot-org/skypilot/pull/10447)); fail fast when a task exits nonzero ([#10428](https://github.com/skypilot-org/skypilot/pull/10428)).
- Fixed leaked allocations on terminate ([#10296](https://github.com/skypilot-org/skypilot/pull/10296)), multi-node run-done barrier failures ([#10321](https://github.com/skypilot-org/skypilot/pull/10321)), step-scoped `SLURM_*` env leaking into user scripts ([#10309](https://github.com/skypilot-org/skypilot/pull/10309)), and containerized runs broken by an exported `which` function ([#10716](https://github.com/skypilot-org/skypilot/pull/10716)).

**AWS**

- Skip default security group pre-creation when the user specifies security groups ([#10178](https://github.com/skypilot-org/skypilot/pull/10178)).

**GCP**

- **RTX PRO 6000 Blackwell (G4)** support ([#9634](https://github.com/skypilot-org/skypilot/pull/9634)); federated credentials for GCS COPY ([#10212](https://github.com/skypilot-org/skypilot/pull/10212)).

**Azure**

- **Boot from private Shared Image Gallery images** ([#9843](https://github.com/skypilot-org/skypilot/pull/9843)).

**Nebius**

- **Personalized (per-tenant) pricing** ([#9107](https://github.com/skypilot-org/skypilot/pull/9107), [#10222](https://github.com/skypilot-org/skypilot/pull/10222)), **InfiniBand on B200/B300** ([#10369](https://github.com/skypilot-org/skypilot/pull/10369)), and optional **boot disk encryption** ([#10172](https://github.com/skypilot-org/skypilot/pull/10172)).
- Catalog fetcher switched to billing v1 ([#9829](https://github.com/skypilot-org/skypilot/pull/9829), [#10400](https://github.com/skypilot-org/skypilot/pull/10400)); wait for instance termination before deleting the security group ([#10220](https://github.com/skypilot-org/skypilot/pull/10220)).

**OCI / RunPod**

- OCI: GB200/GB300 NCCL profiles for `network_tier: best` ([#10204](https://github.com/skypilot-org/skypilot/pull/10204)).
- RunPod: corrected region to data center mapping ([#10072](https://github.com/skypilot-org/skypilot/pull/10072)); label account SSH keys with the SkyPilot user ([#9975](https://github.com/skypilot-org/skypilot/pull/9975)).

### User Experience

**Dashboard**

- **Reworked Infra page**: unified table, per-GPU-type summary, and responsive layout ([#10067](https://github.com/skypilot-org/skypilot/pull/10067)), with Slurm clusters broken down by partition, CPU-only and unreachable clusters listed, and progressive per-cluster loading ([#10317](https://github.com/skypilot-org/skypilot/pull/10317), [#10318](https://github.com/skypilot-org/skypilot/pull/10318), [#10329](https://github.com/skypilot-org/skypilot/pull/10329), [#10629](https://github.com/skypilot-org/skypilot/pull/10629)); node table pagination ([#10480](https://github.com/skypilot-org/skypilot/pull/10480)).
- **Clusters page defaults to "My Clusters"** with owner and activity toggles ([#9884](https://github.com/skypilot-org/skypilot/pull/9884), [#10089](https://github.com/skypilot-org/skypilot/pull/10089)).
- **Filters are named in the URL** for clusters, managed jobs, users, volumes, and infra, so filtered views can be bookmarked and shared ([#10455](https://github.com/skypilot-org/skypilot/pull/10455), [#10459](https://github.com/skypilot-org/skypilot/pull/10459), [#10505](https://github.com/skypilot-org/skypilot/pull/10505)).
- Managed jobs: PENDING reason in the status tooltip ([#10202](https://github.com/skypilot-org/skypilot/pull/10202)), recovery count, Started, and Partition on the detail page ([#10004](https://github.com/skypilot-org/skypilot/pull/10004), [#10647](https://github.com/skypilot-org/skypilot/pull/10647)), and summed job-group durations ([#10678](https://github.com/skypilot-org/skypilot/pull/10678)).
- URL-template and scoped external links ([#10090](https://github.com/skypilot-org/skypilot/pull/10090), [#10194](https://github.com/skypilot-org/skypilot/pull/10194)); fenced code blocks rendered in error messages ([#10088](https://github.com/skypilot-org/skypilot/pull/10088)); owning user shown for pools ([#10653](https://github.com/skypilot-org/skypilot/pull/10653)).
- New SkyPilot branding: logo and favicon ([#10181](https://github.com/skypilot-org/skypilot/pull/10181), [#10215](https://github.com/skypilot-org/skypilot/pull/10215), [#10237](https://github.com/skypilot-org/skypilot/pull/10237)).

**CLI**

- `sky api login` works when `SKYPILOT_API_SERVER_ENDPOINT` is set and follows HTTP to HTTPS redirects ([#10404](https://github.com/skypilot-org/skypilot/pull/10404), [#10624](https://github.com/skypilot-org/skypilot/pull/10624)).
- `sky jobs logs` / `sky jobs cancel` address job-group tasks by index ([#10734](https://github.com/skypilot-org/skypilot/pull/10734)); fixed `sky api logs` traceback on 4xx ([#10341](https://github.com/skypilot-org/skypilot/pull/10341)).

### Dependencies

- Lifted the `typing_extensions` cap and dropped `pydantic<2.12` ([#10649](https://github.com/skypilot-org/skypilot/pull/10649)).
- Unpinned the Nebius SDK now that 0.4.3 is loop-aware ([#10320](https://github.com/skypilot-org/skypilot/pull/10320)).

### Documentation & Examples

- **Dynamic job group docs** ([#10732](https://github.com/skypilot-org/skypilot/pull/10732)), **Hugging Face Hub storage guide** ([#10117](https://github.com/skypilot-org/skypilot/pull/10117)), refreshed Sandboxes ([#10109](https://github.com/skypilot-org/skypilot/pull/10109)) and Sky Batch pages ([#10453](https://github.com/skypilot-org/skypilot/pull/10453)).
- **New examples**: Hugging Face storage batch transcription ([#10558](https://github.com/skypilot-org/skypilot/pull/10558)); slime agentic coding-agent RL on Job Groups with Sandboxes ([#10200](https://github.com/skypilot-org/skypilot/pull/10200)), and resilient Ray training ([#10602](https://github.com/skypilot-org/skypilot/pull/10602)) and GB300 NVL72 Ray recovery ([#10644](https://github.com/skypilot-org/skypilot/pull/10644)) with Dynamic Node Set recovery. Sandboxes and Dynamic Node Set recovery are offered in [SkyPilot Platform](https://skypilot.ai/).
- Updated logos and brand assets across docs and README ([#10238](https://github.com/skypilot-org/skypilot/pull/10238), [#10179](https://github.com/skypilot-org/skypilot/pull/10179)); restyled Ask AI widget with dark mode ([#10180](https://github.com/skypilot-org/skypilot/pull/10180)).

### Testing & CI/CD

- Dashboard tests now run in CI ([#10510](https://github.com/skypilot-org/skypilot/pull/10510)); golden snapshot tests for the Kubernetes Ray template ([#10226](https://github.com/skypilot-org/skypilot/pull/10226)).
- Buildkite: global per-test serialization via `concurrency_group` ([#10150](https://github.com/skypilot-org/skypilot/pull/10150)), no auto-retry for exclusive steps ([#10385](https://github.com/skypilot-org/skypilot/pull/10385)), and cancellation mirrored from GitHub ([#10760](https://github.com/skypilot-org/skypilot/pull/10760)); nightly tagging and Slack notifications ([#10192](https://github.com/skypilot-org/skypilot/pull/10192), [#10719](https://github.com/skypilot-org/skypilot/pull/10719)).
- Moved tests off retired AWS p3/V100 instances ([#10098](https://github.com/skypilot-org/skypilot/pull/10098), [#10107](https://github.com/skypilot-org/skypilot/pull/10107), [#10113](https://github.com/skypilot-org/skypilot/pull/10113)); numerous test reliability and flakiness fixes.

## Contributors

Thank you to all contributors who made this release possible!

@DanielZhangQD, @aylei, @zpoint, @kevinmingtarja, @ishankaul1, @SeungjinYang, @lloyd-brown, @rohansonecha, @cg505, @stephenkevn, @Michaelvll, @MadhavJivrajani, @trevanth24, @hopechong, @ryxli, @Anai-Guo, @liuxu623, @romilbhardwaj, @ciprianf, @hentt30, @ahluwalij, @alex000kim, @complynx, @concretevitamin, @davanstrien, @daw3rd, @edmsimplicio, @himanshu1573, @intagliated, @JGSweets, @Jobarion, @Khas-Erdene-Tsogtsaikhan, @klenovda, @kyuds, @nicosuter, @pauldonzier, @pavneeta, @samuelv8, @spiffaz, @typesafeainate, @yash170106065

## New Contributors
* @ciprianf made their first contribution in https://github.com/skypilot-org/skypilot/pull/10072
* @nicosuter made their first contribution in https://github.com/skypilot-org/skypilot/pull/10050
* @spiffaz made their first contribution in https://github.com/skypilot-org/skypilot/pull/10156
* @edmsimplicio made their first contribution in https://github.com/skypilot-org/skypilot/pull/10230
* @yash170106065 made their first contribution in https://github.com/skypilot-org/skypilot/pull/9924
* @pavneeta made their first contribution in https://github.com/skypilot-org/skypilot/pull/10260
* @liuxu623 made their first contribution in https://github.com/skypilot-org/skypilot/pull/10284
* @complynx made their first contribution in https://github.com/skypilot-org/skypilot/pull/10345
* @Anai-Guo made their first contribution in https://github.com/skypilot-org/skypilot/pull/10356
* @ryxli made their first contribution in https://github.com/skypilot-org/skypilot/pull/10436
* @intagliated made their first contribution in https://github.com/skypilot-org/skypilot/pull/9932
* @klenovda made their first contribution in https://github.com/skypilot-org/skypilot/pull/9829
* @himanshu1573 made their first contribution in https://github.com/skypilot-org/skypilot/pull/10369
* @typesafeainate made their first contribution in https://github.com/skypilot-org/skypilot/pull/10172
* @davanstrien made their first contribution in https://github.com/skypilot-org/skypilot/pull/10558
* @daw3rd made their first contribution in https://github.com/skypilot-org/skypilot/pull/10716

## Full Changelog

For a complete list of changes, see the [commit history](https://github.com/skypilot-org/skypilot/compare/v0.13.0...v0.14.0).