v0.0.74

pipecat-ai/pipecatv0.0.74Jul 3, 2025by aconchillo

AI Summary

This release adds Speechmatics STT support, Direct Function calling, and the new `SOXRStreamAudioResampler`. It also adds Gemini Multimodal Live File API support.

Key Highlights

  • Added `SpeechmaticsSTTService` for real-time transcription
  • Added Direct Function calling support (metadata extraction from docstrings)
  • Added `SOXRStreamAudioResampler` for better audio processing
  • Added `run_llm` field to message frames

Breaking Changes

  • `FishAudioTTSService` deprecated `model` in favor of `reference_id`

New Features

  • `SpeechmaticsSTTService`
  • Direct Function calling
  • `SOXRStreamAudioResampler`
  • `run_llm` in `LLMMessagesAppendFrame`/`LLMMessagesUpdateFrame`
  • Gemini Multimodal Live File API
  • `user_id` in `TranscriptionFrame`
  • `watchdog_coroutine()` helper
  • `session_token` parameter for `AWSNovaSonicLLMService`

Full Release Notes

**This release has been yanked due to resampling issues affecting audio output
quality and critical bugs impacting `ParallelPipelines` functionality.**

**Please upgrade to version 0.0.76 or later.**

### Added

- Added a new STT service, `SpeechmaticsSTTService`. This service provides real-time speech-to-text transcription using the Speechmatics API. It supports partial and final transcriptions, multiple languages, various audio formats, and speaker diarization.

- Added `normalize` and `model_id` to `FishAudioTTSService`.

- Added `http_options` argument to `GoogleLLMService`.

- Added `run_llm` field to `LLMMessagesAppendFrame` and `LLMMessagesUpdateFrame` frames. If true, a context frame will be pushed triggering the LLM to respond.

- Added a new `SOXRStreamAudioResampler` for processing audio in chunks or streams. If you write your own processor and need to use an audio resampler, use the new `create_stream_resampler()`.

- Added new `DailyParams.audio_in_user_tracks` to allow receiving one track per user (default) or a single track from the room (all participants mixed).

- Added support for providing "direct" functions, which don't need an accompanying `FunctionSchema` or function definition dict. Instead, metadata (i.e. `name`, `description`, `properties`, and `required`) are automatically extracted from a combination of the function signature and docstring.

  Usage:

  ```python
  # "Direct" function
  # `params` must be the first parameter
  async def do_something(params: FunctionCallParams, foo: int, bar: str = ""):
    """
    Do something interesting.

    Args:
      foo (int): The foo to do something interesting with.
      bar (string): The bar to do something interesting with.
    """

    result = await process(foo, bar)
    await params.result_callback({"result": result})

  # ...

  llm.register_direct_function(do_something)

  # ...

  tools = ToolsSchema(standard_tools=[do_something])
  ```

- `user_id` is now populated in the `TranscriptionFrame` and `InterimTranscriptionFrame` when using a transport that provides a `user_id`, like `DailyTransport` or `LiveKitTransport`.

- Added `watchdog_coroutine()`. This is a watchdog helper for couroutines. So, if you have a coroutine that is waiting for a result and that takes a long time, you will need to wrap it with `watchdog_coroutine()` so the watchdog timers are reset regularly.

- Added `session_token` parameter to `AWSNovaSonicLLMService`.

- Added Gemini Multimodal Live File API for uploading, fetching, listing, and deleting files. See `26f-gemini-multimodal-live-files-api.py` for example usage.

### Changed

- Updated all the services to use the new `SOXRStreamAudioResampler`, ensuring smooth transitions and eliminating clicks.

- Upgraded `daily-python` to 0.19.4.

- Updated `google` optional dependency to use `google-genai` version `1.24.0`.

### Fixed

- Fixed an issue where audio would get stuck in the queue when an interrupt occurs during Azure TTS synthesis.

- Fixed a race condition that occurs in Python 3.10+ where the task could miss the `CancelledError` and continue running indefinitely, freezing the pipeline.

- Fixed a `AWSNovaSonicLLMService` issue introduced in 0.0.72.

### Deprecated

- In `FishAudioTTSService`, deprecated `model` and replaced with `reference_id`. This change is to better align with Fish Audio's variable naming and to reduce confusion about what functionality the variable controls.