v0.8

# Oumi v0.8 Release Notes

This release lands the new `oumi deploy` CLI for shipping models to dedicated inference endpoints, an `oumi-mcp` server for MCP-capable assistants, batch-API parity across hosted providers, and a major dependency push to `Transformers``v5 / `TRL` 0.30+ / `vLLM` 0.20+. 

## Highlights

### `oumi deploy` — new CLI for dedicated inference endpoints

A first-class deployment CLI that uploads a model and stands up a dedicated endpoint on a managed inference provider. **Fireworks.ai** lands first, **Parasail** ships in this release, and the architecture (`base_client.py`, typed exception hierarchy) is built so additional providers can plug in.

What it does in one go: validates the model directory → uploads weights (full model or LoRA adapter) → creates a dedicated endpoint with the requested hardware → polls until live → optionally fires test prompts.

```bash
# Single-command deploy from a YAML config
oumi deploy up --config configs/examples/deploy/fireworks_deploy.yaml

# Or assemble the deploy on the CLI
oumi deploy up \
  --model-path /path/to/my-finetuned-model/ \
  --provider fireworks \
  --hardware nvidia_h100_80gb \
  --gpu-count 2 \
  --min-replicas 1 \
  --max-replicas 4

# Lifecycle commands
oumi deploy upload     --model-path ... --provider fireworks --wait
oumi deploy create-endpoint --model-id ... --provider fireworks --wait
oumi deploy status     --endpoint-id ep-123 --provider fireworks --watch
oumi deploy list       --provider fireworks
oumi deploy list-models --provider fireworks
oumi deploy list-hardware --provider fireworks
```

Example `fireworks_deploy.yaml`:

```yaml
model_source: /path/to/my-finetuned-model/
provider: fireworks
model_name: my-finetuned-model-v1
model_type: full          # or "adapter" for LoRA + base_model: ...
hardware:
  accelerator: nvidia_h100_80gb
  count: 2
autoscaling:
  min_replicas: 1
  max_replicas: 4
test_prompts:
  - "Hello, how are you?"
```

### `oumi-mcp` — Model Context Protocol server

Oumi now ships an MCP server so MCP-capable assistants (Claude Desktop, Claude Code, Cursor, …) can browse the ~500 bundled YAML configs, launch and monitor training / eval / inference jobs locally or on cloud, and read built-in workflow guidance — all from chat.

```bash
pip install "oumi[mcp]"
# Two equivalent entry points, both stdio:
oumi-mcp
python -m oumi.mcp
```

Wire it up in your client (Claude Code shown):

```bash
claude mcp add oumi oumi-mcp
```

Or in `claude_desktop_config.json` / Cursor / `~/.claude.json`:

```json
{
  "mcpServers": {
    "oumi": { "command": "oumi-mcp" }
  }
}
```

Built-in prompts cover get-started, train, infer, eval, synth, analyze, post-training, cloud-launch, and an end-to-end `mle_workflow`. Full guide: `docs/user_guides/mcp.md`.


### Batch API support across hosted inference providers

Hosted batch-inference parity. `oumi infer` and the engines now expose batch endpoints for **Anthropic**, **Fireworks**, and **Together**, plus job control (cancel, partial retry) and progress tracking.

```python
from oumi.core.configs import InferenceConfig
from oumi.inference import AnthropicInferenceEngine

config = InferenceConfig.from_yaml("infer.yaml")
engine = AnthropicInferenceEngine(model_params=config.model)

# Submit, poll, fetch results — engine handles the batch lifecycle
results = engine.infer_batch(input=conversations, inference_config=config)
```

```yaml
# Inside an inference YAML, opt in to the batch API
remote_params:
  use_batch_api: true
  batch_completion_window: "24h"
```

### RPM / TPM rate limiting on `RemoteInferenceEngine`

Sliding-window rate limiting baked into every remote engine so you can pin API budgets without reaching for an external proxy. Tracks RPM, input TPM, and output TPM independently from each provider response. `politeness_policy` is now deprecated in favor of `requests_per_minute`.

```yaml
# infer.yaml
model:
  model_name: claude-opus-4-7

engine: ANTHROPIC

remote_params:
  num_workers: 16
  requests_per_minute: 4000
  input_tokens_per_minute: 400_000
  output_tokens_per_minute: 80_000
```


### Cerebras inference engine

New engine targeting Cerebras-hosted models, registered with the standard `InferenceEngineType` factory.

```yaml
model:
  model_name: llama-3.3-70b
engine: CEREBRAS
remote_params:
  api_key_env_varname: CEREBRAS_API_KEY
```

```python
from oumi.inference import CerebrasInferenceEngine
engine = CerebrasInferenceEngine(model_params=...)
```

Related work:
- Cerebras Inference Support (#2231)

### Multi-turn conversation synthesis

`oumi synth` learned to chain conversation synthesizers into multi-turn dialogues — useful for distilling assistant/customer-support style training data.

```bash
oumi synth -c oumi://configs/examples/synthesis/multiturn_conversation_synth.yaml
```

The bundled config generates 5 customer-support conversations with structured action blocks (`CLARIFY`, `LOOKUP_ORDER`, `INITIATE_RETURN`, …); use it as a starting point for your own scenarios. See `docs/user_guides/synth.md` for the synthesizer composition model.

Related work:
- Feature/multiturn synth (#2172)
- Add token usage accumulation to `AttributeSynthesizer` (#2201)
- Treat empty arrays in synthesis config as unset (#2246)

### Judge framework upgrades

The Judge gained batch inference, token-usage accounting, and the ability to take pre-built `Conversation` objects (handy when you already constructed multi-turn context elsewhere).

```python
from oumi.judge import judge_dataset

results = judge_dataset(
    judge_config="oumi://configs/judges/helpfulness.yaml",
    dataset=[
        {"question": "What is 2+2?", "answer": "4"},
        {"question": "How to cook?", "answer": "I don't know"},
    ],
    output_file="judgments.jsonl",
)
for r in results:
    print(r.field_values, r.field_scores)
```

### Inference engine quality of life

Smaller but high-leverage improvements that show up everywhere remote inference is used:

- `list_models()` API on every engine — discover what a provider exposes from code (#2333).
- `finish_reason` is now surfaced on conversation outputs across engines (#2249).
- Quota errors are a typed, catchable error class instead of opaque HTTP failures (#2222).
- Transient HTTP 400s are retried with backoff (#2357).
- `api_input` is attached to `APIStatusError` for easier debugging (#2355).
- Empty Anthropic responses raise an explicit error (#2346); reasoning-model `content: null` is handled (#2354).
- Anthropic prompt caching enabled; cache token usage reported for Anthropic and Together (#2324, #2245).
- vLLM engine accepts arbitrary kwargs through `vllm_config_overrides` for late-breaking server flags (#2314).
- Step is now included in metrics-logger callback (#2244).
- Temperature constraints handled for all OpenAI reasoning models (#2212).
- Lambda inference engine deprecated (#2332).

```python
engine.list_models(chat_only=True)  # ['accounts/.../models/llama-v3p1-70b-instruct', ...]

result = engine.infer(...)
result[0].messages[-1].finish_reason  # 'stop' | 'length' | 'tool_calls' | ...
```

### Config + CLI error handling

Configuration errors used to surface as raw `OmegaConf` stack traces. They now flow through a typed exception hierarchy with explicit messages:

- `OumiConfigParsingError` covers all `OmegaConf` exceptions (#2323).
- Config-specific exception hierarchy + CLI error handling (#2319).
- `DatasetParams.finalize_and_validate` checks `dataset_path` exists (#2320).

```python
from oumi.core.configs import TrainingConfig
from oumi.exceptions import OumiConfigParsingError

try:
    config = TrainingConfig.from_yaml("train.yaml")
except OumiConfigParsingError as e:
    # Clean, user-facing error — not an OmegaConf traceback
    print(e)
```

### `oumi launcher` cost & schedule fields

The launcher base cluster now exposes `start_at`, `end_at`, and `cost_per_hour` — useful for time-boxed runs and pricing-aware cluster selection on SkyPilot clouds.

```python
from oumi import launcher

cluster, job = launcher.up(job_config, cluster_name="train-run")
print(cluster.start_at, cluster.end_at, cluster.cost_per_hour)
```

### New & updated model configs

- **Qwen3.5 0.8B** — full-finetune, LoRA, and inference (HF + vLLM) recipes (#2285, #2305).
- **Qwen3-VL 2B / 4B / 8B / 30B-A3B** vision-language configs and MoE support (#2286, #2288, #2102).
- **GPT-OSS 120B LoRA** multi-GPU training config (#2186).
- **Qwen3 MoE** (235B, 30B-A3B, 80B-A3B-Instruct) LoRA training configs (#2211).
- **Llama 4 Scout** LoRA config refresh (#2184).

```bash
oumi train -c configs/recipes/qwen3_5/sft/0.8b_lora/train.yaml
oumi infer -c configs/recipes/qwen3_5/inference/0.8b_vllm_infer.yaml
oumi train -c configs/recipes/gpt_oss/sft/120b_lora_multi_gpu_train.yaml
oumi train -c configs/recipes/qwen3/sft/235b_lora/train.yaml
```

### Tool-use training: correct tool-result masking

`DataCollatorForCompletionOnlyLM` was extended to correctly mask tool-result tokens during completion-only training, so SFT on tool-use traces actually trains the assistant turn and not the (already-known) tool output (#2369).

---

## Dependency upgrades

The big one: **Transformers v5**. Several small breakages were absorbed in this release so users don't have to.

- **Transformers** → `>=4.57,<5.7` (#2317, #2344, #2394)
- **TRL** → `>=0.24,<1.4` (#2298, #2362, #2403)
- **vLLM** → `>=0.14,<0.21`, including 0.12 support (#2275, #2295, #2352, #2402)
- **veRL** → `>=0.5,<0.8` (#2297, #2316, #2322)
- **PEFT** → `>=0.17,<0.20` (#2379)
- **Pydantic** → `>=2.11,<2.14` (#2381)
- **SkyPilot** → `>=0.11.1,<0.13` (#1675e8e6, #2175, #2191, #2349)
- **datasets**, **torchvision**, **torch**, **wandb**, **liger-kernel**, **kernels**, **typer**, **uvicorn**, **pillow**, **responses**, **nvidia-ml-py** all bumped (see commit list).
- TFv5 fallout absorbed: `KTO`/`GKD` import paths updated, DPO preprocessing updated, `warmup_ratio` deprecation handled, VL processor message-to-dict transform, tokenizer/test fixes (#2266–#2279, #2287–#2290).

---

## New Contributors
* @idoudali made their first contribution in https://github.com/oumi-ai/oumi/pull/2186
* @AnandVishesh1301 made their first contribution in https://github.com/oumi-ai/oumi/pull/2225

**Full Changelog**: https://github.com/oumi-ai/oumi/compare/v0.7...v0.8