v2.19.7

<img width="1537" height="730" alt="image" src="https://github.com/user-attachments/assets/bb0ef600-70cf-4943-8247-debb59e438a7" />

# Semaphore UI v2.19

## Highlights

- **Workflows** — build multi-template pipelines with a graphical editor
- **Docker and Kubernetes executors** — run tasks in containers/pods instead of on the server (Pro/Enterprise)
- **JWT / OIDC ID tokens for tasks** — keyless auth to Vault, AWS, GCP, Azure
- **Encryption key rotation** with a labelled keyring
- **Real server-side pagination** for task history — projects with millions of tasks no longer choke
- **BoltDB removed** — SQLite/MySQL/Postgres only
- A large batch of **security hardening** across the API


## Workflows

A workflow is a graph of task templates that run as one unit.

- Graphical editor at `/workflows/new` and `/workflows/:id/edit` (Drawflow-based), with palette
  drag-and-drop, edge condition selectors, live cycle/self-edge guards, a validation problems panel,
  and auto-layout for position-less workflows. Node positions are persisted.
- Node kinds: **task** (runs a template), **approval** (gates the run, with timeout and message), and
  **note** (free-form annotation, never executed).
- Per-node task parameters (`task_params_id` on workflow nodes).
- API: `/project/{id}/workflows` (CRUD), `/{workflow_id}/run`, `/runs`,
  `/runs/{run_id}/{stop,artifacts,approvals}`, `POST /runs/{run_id}/approvals/{node_id}`.

## Executors: Docker and Kubernetes (Pro/Enterprise)

Runners can now execute tasks in a container or a pod instead of directly on the runner host.

- **Kubernetes** (`runner.executor.k8s`): `kubeconfig`, `namespace` (default `semaphore`), `image`,
  `helper_image`, `service_account`, `pull_secrets`, `poll_interval_seconds`, `cleanup_grace_seconds`.
- **Docker** (`runner.executor.docker`): `host`, `tls_verify`, `cert_path`, `image`, `helper_image`,
  `network`, `pull_policy`, `cpu_limit`, `memory_limit`, `privileged` (off by default), poll/cleanup
  timings.
- Each option also has a `SEMAPHORE_RUNNER_K8S_*` / `SEMAPHORE_RUNNER_DOCKER_*` environment variable.
- New `semaphoreui/job` and `semaphoreui/helper` images are built and published by CI; the job image
  ships Terraform/OpenTofu/Terragrunt and `paramiko`.
- **Per-template executor image** — a template can override the container image used for its tasks
  (`project__template.executor_image`).

## Task JWT / OIDC ID tokens

Semaphore can now act as an OIDC provider for running tasks, so jobs authenticate to external systems
without long-lived credentials.

- Short-lived ECDSA-signed JWTs issued per task, published via `GET /.well-known/jwks.json`.
- Per-template JWT options (multiple audiences, per-token TTL) configured in the template form; claims
  carry IDs only.
- Config block `jwt`: `enabled`, `issuer`, `default_ttl` (1h), `max_ttl` (24h), with `SEMAPHORE_JWT_*`
  env vars.
- Template JWT params persisted in `project__template.jwt_params`.

## Secrets & encryption

- **Encryption key rotation.** New `encryption` config block with a labelled keyring: inline `keys`
  (value or file), or a `keys_folder` where each file is a key named by its filename, plus
  `active.secret_key` / `active.option_key` pointers. Ciphertext now carries a key ID, so keys can be
  rotated without a big-bang re-encrypt. `keys_file` + `keys_poll_interval` (default `15s`) allow hot
  reload. The legacy flat `access_key_encryption` still works and is used when `encryption` is unset.
- **`option_encryption`** — separate key for encrypting DB-stored options.
- **Survey secret variables now work on remote runners and in HA.** Previously a survey `secret` value
  lived only in the memory of the node that accepted the task, so it arrived empty on remote runners
  and was lost across restarts. Secrets are now persisted as task-bound access keys
  (`access_key.task_id`, `access_key.expire_at`, cascade-deleted with the task); TTL is derived from
  `MaxTaskDurationSec` (+1h queue allowance, 24h when unlimited).
- **OpenBao** secret storage type (routed through the Vault provider), with its own icon in the UI.
- TLS-skip-verify checkbox for Vault/OpenBao storages.
- Synchronized and read-only secret fields are no longer wiped on update.

## Runners

- **Online/offline status** shown on the Runners page, derived from heartbeat liveness (webhook-driven
  runners are always dispatch candidates).
- **Hung-task recovery.** Runners report their process start time (`X-Runner-Started-At`, stored in
  `runner.started_at`), which lets the server detect a runner that restarted and silently lost its
  in-memory job pool. `starting` tasks are reassigned to a healthy runner; `running` tasks get a
  recovery window and are then failed with a clear message. Tunables in the new `runners` config
  block: `offline_timeout_sec` (120), `task_fail_timeout_sec` (420), `reconcile_interval_sec` (30).
- Tasks reassigned away from a runner are terminated on the old runner.
- **Per-runner RSA encryption keys removed** — secrets are protected by TLS on the wire instead. This
  removes ~380 lines of key-exchange code from the runner protocol.
- Runner registration tokens are stored hashed, with an expiry; invalid registration tokens are
  rejected with `400`, and token prefixes are validated.
- Fixed a TCP connection leak in the runner client.
- Runner options moved into a dedicated `runners` config struct (old flat options still read).
- The `active` flag was dropped from runner registration; only registered runners are used.

## Performance & scale

- **Keyset pagination for task history.** The history page used to fetch the 200 newest tasks and page
  through them client-side. The backend now returns one page at a time via a `before` cursor +
  `count` (legacy `limit` still accepted), with no `COUNT(*)` and no `OFFSET` — so page depth no
  longer degrades on projects with millions of tasks. Applies to project history, template task lists
  (`/templates/{id}/tasks`, `/tasks/last`, `/stats`) and the dashboard.
- Task lists reload at most once every 5 seconds; several redundant UI requests removed.
- **Git operations are serialized per repository directory** (`KeyLock`). Templates with
  `AllowParallelTasks=true` shared one working copy, and concurrent `git pull` + `git checkout` could
  corrupt it. `updateRepository()` + `checkoutRepository()` is now one critical section, covering both
  local and runner execution. Inventory repo operations are serialized the same way.
- **BoltDB removed.** SQLite replaces it everywhere, including session storage; the
  permanent-connection flag is gone.
- Stale HA pool state is released on duplicate finalize.

## Templates, tasks & UI

- **Dynamic playbook picker** — the template form lists actual playbook files from the repository
  (`GET /repositories/{repository_id}/playbooks`) instead of requiring a hand-typed path. Playbooks
  reset when the branch changes, and branch-load failures no longer block the playbook list.
- **Survey variable target** — a survey var can now be delivered as a process environment variable
  (`target: "env"`) instead of the app-specific CLI way (`--extra-vars` / `-var` / CLI arg). The env
  var name is the variable name verbatim, so `TF_VAR_foo` works. Stored in the existing `survey_vars`
  JSON — no migration.
- **New survey variable types**: `int`, `text` (multiline), and restyled `enum`.
- **Typed variables in variable groups** (including `int`).
- **Skip Ansible Galaxy install** — per-template and per-task option to skip role/collection
  requirement installation.
- Dropdown cards for JWT and schedule sections in the template form; new `DropdownCard` /
  `HighlightedCard` components.
- **Czech translation** added.
- Copy-to-clipboard icon is visible in light mode; running-task spinners fixed; template form bottom
  padding fixed.
- Schedules are validated with the server-side cron parser (client and server no longer disagree).
- Integration variable extraction preserves JSON objects and arrays instead of stringifying them.

## Observability

- **Prometheus metrics.** New `metrics` config block (`enabled`, `username`, `password`, plus
  `SEMAPHORE_METRICS_*`) exposing `/api/metrics` with Go/process collectors,
  `semaphore_tasks_running` (gauge) and `semaphore_tasks_total{status}` (counter). The endpoint is off
  by default and can be protected with basic auth.
- **Namespaced debug logging.** A Node.js `debug`-style filter via `--debug-filter` /
  `SEMAPHORE_DEBUG_FILTER` lets you turn on verbose tracing for one subsystem (runners, LDAP,
  schedules, git) without the noise from everything else. Applies to syslog hooks too.
  `SEMAPHORE_LOG_LEVEL` / `--log-level` behave exactly as before.
- Many new contextual debug statements across runners, tasks and auth.
- **SSH host key checking is configurable**: `ssh.known_hosts_file`, `ssh.config_path`,
  `ssh.strict_host_key_checking` (`no` / `yes` / `accept-new`). With no known-hosts file configured,
  Semaphore uses a persistent trust-on-first-use file under `TmpPath` — first connection pinned, later
  host-key changes rejected.

## Security

- **Changing a password or managing 2FA/TOTP now requires the current password** (CWE-620 —
  unverified password change).
- **Origin/Referer validation** on state-changing requests (CSRF hardening).
- Session cookies marked `Secure` over HTTPS.
- Custom role creation now checks the caller's permissions.
- Task branch override is only allowed when the template sets `AllowOverrideBranchInTask`.
- Git URL validation; `--end-of-options` passed to git so a crafted ref cannot be read as a flag;
  commit hashes format-checked; branches validated before repository browsing; playbook paths
  validated.
- Access key payloads validated; template `app` validated.
- Project/integration ID ownership verified on integration API requests; `template_id` verified on
  vault update.
- Runner tokens are no longer written into project backups.
- API returns after a write error instead of continuing with a partially written response.
- Runner per-runner encryption keys removed in favour of transport TLS.
- Fixed a nil-pointer panic when closing an uninitialized DB.
- CodeQL: potential integer overflow in allocation size computation.
- Security SLA published in `SECURITY.md`; release artifacts signed with the
  `security@semaphoreui.com` GPG key.

---

## Upgrade notes

### Breaking / behavioural changes

1. **BoltDB is gone.** If you are still running a BoltDB installation, migrate to SQLite (or
   MySQL/Postgres) before upgrading. `bolt` is no longer a valid `dialect`.
2. **Runner encryption keys removed.** Runners and server must both be on 2.19; the key-exchange step
   no longer exists. Make sure runner↔server traffic is TLS-protected.
3. **Task list APIs are paginated.** `GET /project/{id}/tasks` and the template task endpoints return
   one page and take a `before` cursor plus `count`. `limit` is still accepted for backward
   compatibility, but clients relying on getting the newest 200 tasks in one response must page.
4. **Runner `active` flag removed** from registration.
5. Project backups no longer contain runner tokens — re-register runners rather than expecting a
   restore to carry credentials.

### New configuration

`encryption`, `jwt`, `metrics`, `runners`, `ssh`, `runner.executor.{docker,k8s}`,
`ldap_tls_skip_verify`, `option_encryption`, `external_auth_email_matching`.

All are optional; existing configs keep working. `use_remote_runner` and the flat runner options are
deprecated in favour of `runners`. The JSON schema (`config.schema.yaml`) and the config-option
reference docs have been regenerated.

## Documentation

Docs are now a git submodule and are embedded in the binary, so they ship offline. New/updated pages:
Docker & Kubernetes executors, encryption and key rotation, JWT auth during task execution, OpenBao
secret storage, IdP-initiated OIDC, PingFederate TLS, license activation, dev setup, logging, runner
registration, and regenerated CLI reference.

## Dependencies & build

Go 1.26.5; Debian 13 and Alpine 3.24 base images; `go-git` 5.19.2, `go-oidc` 3.20.0,
`golang.org/x/crypto` 0.54.0, `go-ldap` 3.4.14, `modernc.org/sqlite` 1.54.0; frontend `axios` 1.19,
`cron-parser` 5.7, `nanoid` 6, `shell-quote` 1.10. MariaDB 12.1 migration compatibility fixed.
`THIRD-PARTY-LICENSES.md` regenerated.

## Known gaps

Two plans in `AGENTS/plans/2_19` did **not** ship in this line and are still open:
`password-hash-argon2id` (bcrypt → Argon2id) and `runner-token-hash` (hashing the runner bearer token
itself — only *registration* tokens are hashed today). The `runner-version-platform-uptime` plan
shipped partially: uptime and online/offline status landed, but runner version/OS/arch columns did
not.