v0.5.4

# 🚀 Announcing Agent Sandbox v0.5.4!

We're excited to announce the release of Agent Sandbox v0.5.4! This release brings significant improvements in core stability, performance, and introduces powerful new features for the Python SDK and router. Key areas of focus include hardening sandbox lifecycle management, optimizing API server interactions, and expanding integration capabilities.

### ⚠️ Breaking Changes / Action Required

*   **Sandbox `Suspended` condition changed to comply with Kubernetes convention (#1150):**
    *   The `Suspended` status condition is now always present on `Sandbox` resources after initial reconciliation. Clients should inspect its boolean `status` value (e.g., `status: "False"`) rather than checking for its presence or absence.
    *   The legacy `PodNotTerminated` reason string is deprecated in favor of `PodTerminating`. Update any client-side logic that explicitly checks for `PodNotTerminated`.

### Key Highlights

#### Core Stability & Lifecycle Management
*   **Persistent Suspended Conditions for Sandbox (#1150):** The Sandbox `Suspended` status condition is now consistently present and transitions its status (`True` or `False`) instead of appearing or disappearing, enabling standard `kubectl wait` commands and preserving history.
*   **Robust Adoption Assignment & Optimistic Locks (#1277):** Prevents assignment flips, orphaned sandboxes, and duplicate adoptions under load by applying optimistic locks on adoption patches and resolving conflicts against authoritative reads. Benign adoption contention now surfaces as an `AdoptionConflict` Ready reason.
*   **Optimistic-Locked Status Writes (#1256):** Eliminates duplicate startup-latency histogram observations and stale status overwrites by using optimistic-locked status patches, ensuring metrics are recorded exactly once and status reflects the authoritative state.
*   **Reduced Redundant Reconciles & Status Patching (#1254):** Improves controller efficiency by using non-optimistic merge patches for Sandbox status writes (eliminating 409 conflicts) and adding predicates to only reconcile owning `SandboxClaim` on relevant `Sandbox` field changes.
*   **Warm Pool Over-Creation Mitigations (#1266):** Prevents `SandboxWarmPool` from over-creating replicas due to informer-cache lag. Sandbox creations are now gated by a ReplicaSet-style expectations tracker, and terminating sandboxes count against the target. Introduces `WarmPoolNotProgressing` events for capacity shortfalls.
*   **Efficient Pod Cache Transformation (#1272):** Strips `Pod` finalizers from the informer cache, reducing memory usage and event decoding costs as they are not used by Agent Sandbox controllers.
*   **Semantic Comparison for K8s API Objects (#1278):** Replaces `reflect.DeepEqual` with `apiequality.Semantic.DeepEqual` for Kubernetes API object comparisons, preventing unnecessary status updates and reconciliation due to non-semantic differences.
*   **Stricter API Group Verification (#945):** Enhances warm-pool and sandbox ownership/identity validation by consistently verifying API Group in addition to Kind for controller and owner references.
*   **Improved RBAC for Events (#1080):** Grants core API group (`""`) events permission for the leader election event recorder, fixing `events is forbidden` errors.
*   **Exposed Sandbox Service Ports (#1258):** Generated headless `Sandbox` Services now automatically include ports derived from declared container ports, improving compatibility with service meshes like Istio.
*   **Accurate Startup Latency Metrics (#1087):** Fixes `SandboxClaim` controller startup latency histograms to record exactly once per claim lifetime, preventing overcounts due to readiness probe flapping.
*   **Go Client Handle Registry Race Fix (#998):** Prevents leaking orphaned sandbox handles in the Go client by making `GetSandbox`/`CreateSandbox` race-safe and ensuring proper disconnection of redundant handles.
*   **Enhanced Adoption Conflict Resolution (#1304):** Improves adoption handling for optimistic-lock contention and refines user-facing conflict messaging to avoid exposing low-level internal errors.

#### Performance & Scalability
*   **API Connection Sharding (#1240):** Introduces opt-in controller flags (`--separate-watch-connection` and `--api-connections`) to use dedicated HTTP/2 connections for informer watches and shard non-watch traffic, significantly improving API concurrency and reducing watch starvation.
*   **Optimized Metadata Writes (#1250):** Implements direct merge patches for hot-path metadata writes on `SandboxClaim` controllers, reducing CPU and memory allocations by building targeted patch payloads instead of full-object diffs.
*   **Stress Test Configuration Improvements (#1283):** Increases Kubelet API QPS limits and caps Kubelet event spam during stress tests, allowing for higher churn rates and reducing API server load.
*   **KWOK Scalability Presubmit Test (#1269):** Adds a fast, lightweight presubmit test using KWOK (Kubernetes WithOut Kubelet) to automatically benchmark performance and catch scalability regressions on every Pull Request.
*   **Optimized Pod Counting for Circuit Breaker (#1232):** The circuit breaker now efficiently counts pods using `limit=1` and `remainingItemCount` to reduce API server and memory load at large scales.
*   **Infrastructure Tuning for Stress Tests (#1275):** Configures benchmark worker node root volumes on `pd-ssd` and control plane nodes on `c3-standard-22`, addressing disk I/O and control plane CPU bottlenecks for improved stress test performance.
*   **Beta API Serving Optimization (#1234):** Stress tests now serve only the beta Sandbox APIs to reduce conversion traffic and improve API server efficiency.

#### Router Enhancements
*   **Scoped-Token Authorizer (#1243):** Introduces a new `--authz-mode=scoped-token` for `sandbox-router`, allowing local verification of signed tokens bound to a single (namespace, name) without requiring a kube-apiserver round-trip.
*   **Improved Warm-Pool Routing (#1239):** `sandbox-router` now routes requests carrying `X-Sandbox-Id` via the Pod-IP cache's namespace/name index, fixing 502 (NXDOMAIN) errors for warm-pool sandboxes without a dedicated Service.

#### SDK & Integrations
*   **Agent Sandbox Recycling (Python SDK) (#1232):** Introduces sandbox recycling (`reuse_git_restore_sandbox`) in the `agent-sandbox-rl` SDK, allowing reuse of claimed sandboxes across tasks (e.g., RL rollouts) to reduce claim latency and API load. Includes contamination guards, persistent exec sessions, and warm-pool over-creation mitigations.
*   **Optimized Python SDK `wait_for_claim_ready` (#1241):** The Python SDK's `wait_for_claim_ready()` now uses a single, watch-based approach on the claim itself, significantly reducing latency compared to the previous two-sequential-watch method. Also, faster dev port-forward polling.
*   **AsyncSandbox Functionality Alignment (#999):** Aligns `AsyncSandbox` functionality with `Sandbox` in the Python SDK, ensuring consistent behavior and expanding API capabilities.
*   **Langchain-Deepagents Integration (#1144):** Adds a new Python package for integrating with Langchain-Deepagents, enabling sandbox lifecycle management, command execution, and file transfer.
*   **MCP Server Introduction (#1141):** Adds an Agent Sandbox MCP (Multi-Cluster Proxy) server with basic implementation for listing sandboxes and tools for create/delete, command execution, and file operations.

#### Documentation & Examples
*   **Expanded Security Threat Model (#1299):** The security threat model documentation has been expanded to include architectural overview, trust boundaries, threat analysis with mitigations, and `SandboxTemplate` enforcement capabilities.
*   **Clarified Repository Scope (#1298):** The `README.md` now explicitly clarifies that Agent Sandbox is a sandbox orchestrator, delegating low-level container isolation to secure runtimes like gVisor or Kata Containers.
*   **Hermes Agents-as-a-Service Example (#1271):** Adds a comprehensive example demonstrating the multi-user "agents as a service" platform pattern, featuring warm-pool claims, suspend/resume for cost management, PVC state survival, and a minimal gateway.
*   **Firecracker Sandbox Example (#1238):** Introduces an example demonstrating how to run Agent Sandboxes on Kata Containers with the Firecracker VMM, providing microVM isolation and fast boot times.
*   **APF Insulation Overlay Example (#1270):** Adds an opt-in API Priority and Fairness (APF) insulation overlay and operator guide to prioritize critical controller API traffic and isolate bulk workloads.
*   **Runtime-Class-Aware Benchmarks (#1262):** Adds e2e tests and benchmarks for warm pool subsystem across runc, gVisor, and Kata runtimes, measuring cold start latency, warm pool claim speed, and burst recovery.
*   **Enhanced Stress Testing Phases (#1287):** Refactors tests to include a phase for testing with large numbers of pods (e.g., up to 80% capacity) and adds new capacity-related test assertions.
*   **Improved Stress Test Reports (#1284):** Enhances stress test reports with a "Limiter Regime by Component" table to identify client-go rate limiter behavior (queueing vs. pacing) and estimate implied QPS limits.

### Installation

#### Standard Install (Core + Extensions)
Recommended for most users and GitOps engines (Argo CD, Config Sync, kustomize):
```bash
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/v0.5.4/sandbox-with-extensions.yaml
```

#### Selective Install
Install components separately:
```bash
# Core only:
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/v0.5.4/sandbox.yaml

# Extensions (opt-in):
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/v0.5.4/extensions.yaml
```

#### Python SDK
```bash
pip install k8s-agent-sandbox==0.5.4
```

--------
### Contributors
We extend our sincere thanks to all contributors to this release:
@ArthurKamalov, @ChristianZaccaria, @Geun-Oh, @YahiaBadr, @aditya-shantanu, @aegeiger, @dongjiang1989, @gruebel, @hsinhoyeh, @janetkuo, @justinsb, @matiasinsaurralde, @ronaknnathani, @shrutiyam-glitch, @tomergee, @vvoronko

### New Contributors
* @matiasinsaurralde made their first contribution in https://github.com/kubernetes-sigs/agent-sandbox/pull/998
* @ronaknnathani made their first contribution in https://github.com/kubernetes-sigs/agent-sandbox/pull/1258
* @YahiaBadr made their first contribution in https://github.com/kubernetes-sigs/agent-sandbox/pull/1269
* @aegeiger made their first contribution in https://github.com/kubernetes-sigs/agent-sandbox/pull/1087
* @gruebel made their first contribution in https://github.com/kubernetes-sigs/agent-sandbox/pull/999
* @vvoronko made their first contribution in https://github.com/kubernetes-sigs/agent-sandbox/pull/1262
* @ChristianZaccaria made their first contribution in https://github.com/kubernetes-sigs/agent-sandbox/pull/1080
* @Geun-Oh made their first contribution in https://github.com/kubernetes-sigs/agent-sandbox/pull/1239

**Full Changelog**: https://github.com/kubernetes-sigs/agent-sandbox/compare/v0.5.3...v0.5.4