v1.0.0

huggingface/speech-to-speechv1.0.0Sep 6, 2026by andimarafioti

AI Summary

The first major release introduces a unified Realtime engine, explicit commands for server, client, and local execution, expanded speech backends including Qwen3 and OpenAI-compatible endpoints, and significant improvements to conversation reliability and lifecycle handling.

Key Highlights

  • Three explicit commands: `serve` (server), `talk` (client), and `local` (both in one terminal).
  • Expanded speech backends: Support for Qwen3-ASR, vLLM Realtime transcription, OmniVoice, Supertonic, and OpenAI-compatible endpoints.
  • Improved conversation reliability: Better handling of ordered text/audio/tool-call output, interruption support, and session cleanup.
  • Updated Apple Silicon setup: Refreshed MLX dependencies with default 4-bit Qwen3 models.
  • Enhanced setup and diagnostics: Local and hosted configurations in README, opt-in transcript logging via `--log_transcripts`.

Breaking Changes

  • Command invocation changed: `speech-to-speech` now requires `speech-to-speech serve`; `--mode realtime` and `--mode local` are deprecated.
  • Removed transport modes: `socket` and `raw-websocket` modes are no longer supported.
  • Removed installation extras: `facebook-mms`, `language-detection`, and `websocket` extras were removed.
  • Flag behavior change: `--mac-optimal-settings` no longer selects a command; use `local` or `serve` explicitly.
  • Default model changed: Hosted LLM default is now `gpt-5.6-terra`.

New Features

  • New command-line interface: `serve`, `talk`, and `local` commands.
  • New speech recognition backend: Qwen3-ASR.
  • Experimental vLLM Realtime transcription support.
  • Optional TTS backends: OmniVoice and Supertonic.
  • Stateful OpenAI Realtime transcription integration.
  • Opt-in transcript logging with `--log_transcripts` flag.
  • Microphone blocking during playback: `--local_audio_block_mic_during_playback` flag.

Full Release Notes

# speech-to-speech v1.0.0

The first major release brings a single Realtime engine, explicit server and
microphone-client commands, more speech backends, and improved conversation
lifecycle handling. These notes cover changes since v0.2.12.

## Highlights

- **Three commands: `serve`, `talk`, and `local`.** Run the Realtime server,
  connect the packaged microphone/speaker client, or start both in one terminal.
  The client also supports local tool execution with `--tool-module`.
- **More speech backends.** Add Qwen3-ASR speech recognition, OpenAI-compatible
  STT and TTS endpoints, stateful OpenAI Realtime transcription, experimental
  vLLM Realtime transcription, and optional OmniVoice and Supertonic TTS.
- **More reliable realtime conversations.** Improve ordered text, audio, and
  tool-call output; transcript and content-part events; interrupted responses;
  remote-request cancellation; and session cleanup. The browser demo uses the
  OpenAI Agents SDK, with integration tests for its WebSocket and WebRTC
  transports against the implemented core Realtime event set.
- **Updated Apple Silicon setup.** Refresh the MLX dependencies and default to
  the 4-bit `mlx-community/Qwen3-4B-Instruct-2507-4bit` LLM with 6-bit Qwen3-TTS.
  The Mac preset supplies defaults while respecting explicit overrides.
- **Clearer setup and diagnostics.** The README provides local Mac, local
  NVIDIA, and hosted-LLM starting configurations, a shared llama.cpp recipe,
  offline setup guidance, and a speaker-feedback workaround. Full transcript
  logging is now opt-in with `--log_transcripts`; operational logs omit
  conversation text by default.

## Upgrading from v0.2.12

In your Python environment:

```bash
pip install --upgrade "speech-to-speech==1.0.0"
```

Update scripts and service definitions to use the explicit commands:

| Previous invocation | v1.0.0 invocation |
|---|---|
| `speech-to-speech` | `speech-to-speech serve` |
| `speech-to-speech --mode realtime` | `speech-to-speech serve` |
| `speech-to-speech --mode local` | `speech-to-speech local` |
| `speech-to-speech --local_mac_optimal_settings` | `speech-to-speech local --mac-optimal-settings` |
| `python scripts/listen_and_play_realtime.py --host 127.0.0.1 --port 8765` | `speech-to-speech talk --url ws://127.0.0.1:8765/v1/realtime` |

The `--mode realtime` and `--mode local` forms still work temporarily with a
deprecation warning. Other mode values, including `socket` and `raw-websocket`,
have been removed. Migrate raw PCM clients to the Realtime WebSocket or WebRTC
API; the old raw transport flags are no longer accepted.

The `--mac-optimal-settings` preset no longer selects a command: use `local`
for microphone/speaker interaction or `serve` for a server. `serve` binds to
`127.0.0.1` by default; set `--host` explicitly for network access. `local`
uses loopback only.

The default hosted LLM is now `gpt-5.6-terra`, with reasoning effort `none`.
Pin `--model_name` if your deployment depends on a particular model. Local
Mac users can likewise override the preset's model with `--model_name`.

The redundant `facebook-mms`, `language-detection`, and `websocket` installation
extras were removed; their dependencies are included in the standard package.
Install `speech-to-speech[webrtc]` for WebRTC, `speech-to-speech[omnivoice]` for
OmniVoice, or `speech-to-speech[supertonic]` for Supertonic. Known options for
inactive backends are accepted but ignored with a warning.

## Compatibility and setup

- Python 3.10+ is supported; the installation smoke tests use Python 3.11 on
  Linux and Apple Silicon. Check the README's CUDA wheel guidance for Linux.
- Realtime compatibility covers the documented core event set, not every
  OpenAI Realtime API feature. See the
  [protocol reference](https://github.com/huggingface/speech-to-speech/blob/v1.0.0/src/speech_to_speech/api/openai_realtime/README.md).
- `--local_audio_block_mic_during_playback` pauses microphone capture during
  assistant playback to reduce speaker feedback. It disables spoken
  interruptions during playback; it does not perform acoustic echo cancellation.

See the [README](https://github.com/huggingface/speech-to-speech/blob/v1.0.0/README.md) for starting configurations and the
[full changelog](https://github.com/huggingface/speech-to-speech/compare/v0.2.12...v1.0.0)
for all changes and contributors.