Starstruk Logo MarkStarstruk

v0.0.36

pipecat-ai/pipecatv0.0.36Jul 2, 2024by aconchillo

AI Summary

Added Gladia and Coqui X (XTTS) STT services. Introduced idle processors and AsyncFrameProcessor for handling asynchronous services like Deepgram.

Key Highlights

  • Added `GladiaSTTService` and `XTTSService`.
  • Added `UserIdleProcessor` and `IdleFrameProcessor`.
  • Added `AsyncFrameProcessor` for async services.
  • Added `BotSpeakingFrame` pushed upstream while bot is talking.

New Features

  • Gladia STT service
  • Coqui X (XTTS) TTS service
  • UserIdleProcessor
  • IdleFrameProcessor
  • AsyncFrameProcessor
  • BotSpeakingFrame (upstream)

Full Release Notes

### Added

- Added `GladiaSTTService`. https://docs.gladia.io/chapters/speech-to-text-api/pages/live-speech-recognition

- Added `XTTSService`. This is a local Text-To-Speech service. https://github.com/coqui-ai/TTS

- Added `UserIdleProcessor`. This processor can be used to wait for any interaction with the user. If the user doesn't say anything within a given timeout a provided callback is called.

- Added `IdleFrameProcessor`. This processor can be used to wait for frames within a given timeout. If no frame is received within the timeout a provided callback is called.

- Added new frame `BotSpeakingFrame`. This frame will be continuously pushed upstream while the bot is talking.

- It is now possible to specify a Silero VAD version when using `SileroVADAnalyzer` or `SileroVAD`.

- Added `AysncFrameProcessor` and `AsyncAIService`.  Some services like `DeepgramSTTService` need to process things asynchronously. For example, audio is sent to Deepgram but transcriptions are not returned immediately. In these cases we still require all frames (except system frames) to be pushed downstream from a single task. That's what `AsyncFrameProcessor` is for. It creates a task and all frames should be pushed from that task. So, whenever a new Deepgram transcription is ready that transcription will also be pushed from this internal task.

- The `MetricsFrame` now includes processing metrics if metrics are enabled. The processing metrics indicate the time a processor needs to generate all its output. Note that not all processors generate these kind of metrics.

### Changed

- `WhisperSTTService` model can now also be a string.

- Added missing * keyword separators in services.

### Fixed

- `WebsocketServerTransport` doesn't try to send frames anymore if serializers returns `None`.

- Fixed an issue where exceptions that occurred inside frame processors were being swallowed and not displayed.

- Fixed an issue in `FastAPIWebsocketTransport` where it would still try to send data to the websocket after being closed.

### Other

- Added Fly.io deployment example in `examples/deployment/flyio-example`.

- Added new `17-detect-user-idle.py` example that shows how to use the new `UserIdleProcessor`.