v0.0.44

pipecat-ai/pipecatv0.0.44Oct 16, 2024by aconchillo

AI Summary

This release adds OpenAI Realtime API support, RTVI bot transcription processor, AudioBufferProcessor, and CanonicalMetricsService.

Key Highlights

  • Added OpenAI Realtime API support with OpenAILLMServiceRealtimeBeta
  • Added RTVIBotTranscriptionProcessor for TTS text aggregation
  • Added AudioBufferProcessor for buffering mixed user and bot audio
  • Added CanonicalMetricsService
  • Added on_first_participant_joined event to LiveKitTransport

New Features

  • Updated MarkdownTextFilter with filter_code and filter_tables options
  • LLM text responses now logged properly as unicode characters
  • Updated frame types to be based from SystemFrame

Full Release Notes

### Added

- Added support for OpenAI Realtime API with the new `OpenAILLMServiceRealtimeBeta` processor. (see https://platform.openai.com/docs/guides/realtime/overview)

- Added `RTVIBotTranscriptionProcessor` which will send the RTVI `bot-transcription` protocol message. These are TTS text aggregated (into sentences) messages.

- Added new input params to the `MarkdownTextFilter` utility. You can set `filter_code` to filter code from text and `filter_tables` to filter tables from text.

- Added `CanonicalMetricsService`. This processor uses the new `AudioBufferProcessor` to capture conversation audio and later send it to Canonical AI. (see https://canonical.chat/)

- Added `AudioBufferProcessor`. This processor can be used to buffer mixed user and bot audio. This can later be saved into an audio file or processed by some audio analyzer.

- Added `on_first_participant_joined` event to `LiveKitTransport`.

### Changed

- LLM text responses are now logged properly as unicode characters.

- `UserStartedSpeakingFrame`, `UserStoppedSpeakingFrame`, `BotStartedSpeakingFrame`, `BotStoppedSpeakingFrame`, `BotSpeakingFrame` and `UserImageRequestFrame` are now based from `SystemFrame`

### Fixed

- Merge `RTVIBotLLMProcessor`/`RTVIBotLLMTextProcessor` and `RTVIBotTTSProcessor`/`RTVIBotTTSTextProcessor` to avoid out of order issues.

- Fixed an issue in RTVI protocol that could cause a `bot-llm-stopped` or `bot-tts-stopped` message to be sent before a `bot-llm-text` or `bot-tts-text` message.

- Fixed `DeepgramSTTService` constructor settings not being merged with default ones.

- Fixed an issue in Daily transport that would cause tasks to be hanging if urgent transport messages were being sent from a transport event handler.

- Fixed an issue in `BaseOutputTransport` that would cause `EndFrame` to be pushed downed too early and call `FrameProcessor.cleanup()` before letting the transport stop properly.