v0.0.63
pipecat-ai/pipecatv0.0.63Apr 12, 2025by aconchillo
AI Summary
This release focuses on enhancing the Gemini integration with better media control and Voice Activity Detection (VAD) settings, while improving the SmallWebRTCTransport for video and audio handling. It also introduces new pipeline processors for frame routing and updates dependencies for better stability.
Key Highlights
- Added media resolution control and Voice Activity Detection (VAD) configuration to Gemini services.
- Introduced ProducerProcessor and ConsumerProcessor for frame routing between consumers.
- Improved SmallWebRTCTransport with remote track mute detection and video capture support.
Breaking Changes
- Removed transcribe_model_audio parameter in favor of native output transcription.
- Updated default model to gemini-2.0-flash-live-001 and base_url to v1beta.
New Features
- GeminiMediaResolution enum
- GeminiVADParams configuration
- ProducerProcessor
- ConsumerProcessor
- SmallWebRTCTransport video capture
Full Release Notes
### Added - Added media resolution control to `GeminiMultimodalLiveLLMService` with `GeminiMediaResolution` enum, allowing configuration of token usage for image processing (LOW: 64 tokens, MEDIUM: 256 tokens, HIGH: zoomed reframing with 256 tokens). - Added Gemini's Voice Activity Detection (VAD) configuration to `GeminiMultimodalLiveLLMService` with `GeminiVADParams`, allowing fine control over speech detection sensitivity and timing, including: - Start sensitivity (how quickly speech is detected) - End sensitivity (how quickly turns end after pauses) - Prefix padding (milliseconds of audio to keep before speech is detected) - Silence duration (milliseconds of silence required to end a turn) - Added comprehensive language support to `GeminiMultimodalLiveLLMService`, supporting over 30 languages via the `language` parameter, with proper mapping between Pipecat's `Language` enum and Gemini's language codes. - Added support in `SmallWebRTCTransport` to detect when remote tracks are muted. - Added support for image capture from a video stream to the `SmallWebRTCTransport`. - Added a new iOS client option to the `SmallWebRTCTransport` **video-transform** example. - Added new processors `ProducerProcessor` and `ConsumerProcessor`. The producer processor processes frames from the pipeline and decides whether the consumers should consume it or not. If so, the same frame that is received by the producer is sent to the consumer. There can be multiple consumers per producer. These processors can be useful to push frames from one part of a pipeline to a different one (e.g. when using `ParallelPipeline`). - Improvements for the `SmallWebRTCTransport`: - Wait until the pipeline is ready before triggering the `connected` event. - Queue messages if the data channel is not ready. - Update the aiortc dependency to fix an issue where the 'video/rtx' MIME type was incorrectly handled as a codec retransmission. - Avoid initial video delays. ### Changed - In `GeminiMultimodalLiveLLMService`, removed the `transcribe_model_audio` parameter in favor of Gemini Live's native output transcription support. Now text transcriptions are produced directly by the model. No configuration is required. - Updated `GeminiMultimodalLiveLLMService`’s default `model` to `models/gemini-2.0-flash-live-001` and `base_url` to the `v1beta` websocket URL. ### Fixed - Updated `daily-python` to 0.17.0 to fix an issue that was preventing to run on older platforms. - Fixed an issue where `CartesiaTTSService`'s spell feature would result in the spelled word in the context appearing as "F,O,O,B,A,R" instead of "FOOBAR". - Fixed an issue in the Azure TTS services where the language was being set incorrectly. - Fixed `SmallWebRTCTransport` to support dynamic values for `TransportParams.audio_out_10ms_chunks`. Previously, it only worked with 20ms chunks. - Fixed an issue with `GeminiMultimodalLiveLLMService` where the assistant context messages had no space between words. - Fixed an issue where `LLMAssistantContextAggregator` would prevent a `BotStoppedSpeakingFrame` from moving through the pipeline.