v0.0.40
pipecat-ai/pipecatv0.0.40Aug 20, 2024by aconchillo
AI Summary
Added RTVI 0.1 protocol support, FatalErrorFrame, and GStreamer pipeline support. Enabled usage metrics and added control frames for bot speaking states.
Key Highlights
- Added support for RTVI message protocol 0.1 (webhooks, actions, config).
- Added `FatalErrorFrame` and `GStreamerPipelineSource`.
- Added `BotStartedSpeakingFrame` and `BotStoppedSpeakingFrame` control frames.
- Enabled LLM and TTS usage metrics.
Breaking Changes
- Changed `StartFrame` from a system frame to a control frame.
- Changed `BotSpeakingFrame` to a control frame.
- Changed `SileroVAD` dependency import to `silero-vad` package.
New Features
- RTVI 0.1 protocol support
- FatalErrorFrame
- GStreamerPipelineSource
- Usage metrics
- BotStartedSpeakingFrame and BotStoppedSpeakingFrame
Full Release Notes
### Added - VAD parameters can now be dynamicallt updated using the `VADParamsUpdateFrame`. - `ErrorFrame` has now a `fatal` field to indicate the bot should exit if a fatal error is pushed upstream (false by default). A new `FatalErrorFrame` that sets this flag to true has been added. - `AnthropicLLMService` now supports function calling and initial support for prompt caching. (see https://www.anthropic.com/news/prompt-caching) - `ElevenLabsTTSService` can now specify ElevenLabs input parameters such as `output_format`. - `TwilioFrameSerializer` can now specify Twilio's and Pipecat's desired sample rates to use. - Added new `on_participant_updated` event to `DailyTransport`. - Added `DailyRESTHelper.delete_room_by_name()` and `DailyRESTHelper.delete_room_by_url()`. - Added LLM and TTS usage metrics. Those are enabled when `PipelineParams.enable_usage_metrics` is True. - `AudioRawFrame`s are now pushed downstream from the base output transport. This allows capturing the exact words the bot says by adding an STT service at the end of the pipeline. - Added new `GStreamerPipelineSource`. This processor can generate image or audio frames from a GStreamer pipeline (e.g. reading an MP4 file, and RTP stream or anything supported by GStreamer). - Added `TransportParams.audio_out_is_live`. This flag is False by default and it is useful to indicate we should not synchronize audio with sporadic images. - Added new `BotStartedSpeakingFrame` and `BotStoppedSpeakingFrame` control frames. These frames are pushed upstream and they should wrap `BotSpeakingFrame`. - Transports now allow you to register event handlers without decorators. ### Changed - Support RTVI message protocol 0.1. This includes new messages, support for messages responses, support for actions, configuration, webhooks and a bunch of new cool stuff. (see https://docs.rtvi.ai/) - `SileroVAD` dependency is now imported via pip's `silero-vad` package. - `ElevenLabsTTSService` now uses `eleven_turbo_v2_5` model by default. - `BotSpeakingFrame` is now a control frame. - `StartFrame` is now a control frame similar to `EndFrame`. - `DeepgramTTSService` now is more customizable. You can adjust the encoding and sample rate. ### Fixed - `TTSStartFrame` and `TTSStopFrame` are now sent when TTS really starts and stops. This allows for knowing when the bot starts and stops speaking even with asynchronous services (like Cartesia). - Fixed `AzureSTTService` transcription frame timestamps. - Fixed an issue with `DailyRESTHelper.create_room()` expirations which would cause this function to stop working after the initial expiration elapsed. - Improved `EndFrame` and `CancelFrame` handling. `EndFrame` should end things gracefully while a `CancelFrame` should cancel all running tasks as soon as possible. - Fixed an issue in `AIService` that would cause a yielded `None` value to be processed. - RTVI's `bot-ready` message is now sent when the RTVI pipeline is ready and a first participant joins. - Fixed a `BaseInputTransport` issue that was causing incoming system frames to be queued instead of being pushed immediately. - Fixed a `BaseInputTransport` issue that was causing start/stop interruptions incoming frames to not cancel tasks and be processed properly. ### Other - Added `studypal` example (from to the Cartesia folks!). - Most examples now use Cartesia. - Added examples `foundational/19a-tools-anthropic.py`, `foundational/19b-tools-video-anthropic.py` and `foundational/19a-tools-togetherai.py`. - Added examples `foundational/18-gstreamer-filesrc.py` and `foundational/18a-gstreamer-videotestsrc.py` that show how to use `GStreamerPipelineSource`. - Remove `requests` library usage. - Cleanup examples and use `DailyRESTHelper`.