diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml
index b806efad4..bd6961dd5 100644
--- a/.github/workflows/tests.yaml
+++ b/.github/workflows/tests.yaml
@@ -1,4 +1,4 @@
-name: test
+name: tests
on:
workflow_dispatch:
@@ -49,4 +49,4 @@ jobs:
- name: Test with pytest
run: |
source .venv/bin/activate
- pytest --ignore-glob="*to_be_updated*" --ignore-glob=*pipeline_source* src tests
+ pytest
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
new file mode 100644
index 000000000..6906b571b
--- /dev/null
+++ b/.pre-commit-config.yaml
@@ -0,0 +1,7 @@
+repos:
+ - repo: local
+ hooks:
+ - id: ruff-format-hook
+ name: Check ruff formatting
+ entry: sh scripts/pre-commit.sh
+ language: system
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 80c7c0c7d..6811f08e9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,9 +9,134 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- Added a `metadata` field to `Frame` which makes it possible to pass custom
+ data to all frames.
+
+- Added `test/utils.py` inside of pipecat package.
+
+### Fixed
+
+- Fixed an issue where `ElevenLabsTTSService` messages would return a 1009
+ websocket error by increasing the max message size limit to 16MB.
+
+- Fixed a `DailyTransport` issue that would cause events to be triggered before
+ join finished.
+
+- Fixed a `PipelineTask` issue that was preventing processors to be cleaned up
+ after cancelling the task.
+
+- Fixed an issue where queuing a `CancelFrame` to a pipeline task would not
+ cause the task to finish. However, using `PipelineTask.cancel()` is still the
+ recommended way to cancel a task.
+
+### Other
+
+- Updated all examples to use `task.cancel()` instead of pushing an `EndFrame`
+ when a participant leaves/disconnects. If you push an `EndFrame` this will
+ cause the bot to run through everything that is internally queued (which could
+ take seconds). Instead, if a participant disconnects there is nothing else to
+ be sent and therefore we should stop immediately.
+
+## [0.0.54] - 2025-01-27
+
+### Added
+
+- In order to create tasks in Pipecat frame processors it is now recommended to
+ use `FrameProcessor.create_task()` (which uses the new
+ `utils.asyncio.create_task()`). It takes care of uncaught exceptions, task
+ cancellation handling and task management. To cancel or wait for a task there
+ is `FrameProcessor.cancel_task()` and `FrameProcessor.wait_for_task()`. All of
+ Pipecat processors have been updated accordingly. Also, when a pipeline runner
+ finishes, a warning about dangling tasks might appear, which indicates if any
+ of the created tasks was never cancelled or awaited for (using these new
+ functions).
+
+- It is now possible to specify the period of the `PipelineTask` heartbeat
+ frames with `heartbeats_period_secs`.
+
+- Added `DailyMeetingTokenProperties` and `DailyMeetingTokenParams` Pydantic models
+ for meeting token creation in `get_token` method of `DailyRESTHelper`.
+
+- Added `enable_recording` and `geo` parameters to `DailyRoomProperties`.
+
+- Added `RecordingsBucketConfig` to `DailyRoomProperties` to upload recordings to a custom AWS bucket.
+
+### Changed
+
+- Enhanced `UserIdleProcessor` with retry functionality and control over idle
+ monitoring via new callback signature `(processor, retry_count) -> bool`.
+ Updated the `17-detect-user-idle.py` to show how to use the `retry_count`.
+
+- Add defensive error handling for `OpenAIRealtimeBetaLLMService`'s audio
+ truncation. Audio truncation errors during interruptions now log a warning
+ and allow the session to continue instead of throwing an exception.
+
+- Modified `TranscriptProcessor` to use TTS text frames for more accurate assistant
+ transcripts. Assistant messages are now aggregated based on bot speaking boundaries
+ rather than LLM context, providing better handling of interruptions and partial
+ utterances.
+
+- Updated foundational examples `28a-transcription-processor-openai.py`,
+ `28b-transcript-processor-anthropic.py`, and
+ `28c-transcription-processor-gemini.py` to use the updated
+ `TranscriptProcessor`.
+
+### Fixed
+
+- Fixed an `GeminiMultimodalLiveLLMService` issue that was preventing the user
+ to push initial LLM assistant messages (using `LLMMessagesAppendFrame`).
+
+- Added missing `FrameProcessor.cleanup()` calls to `Pipeline`,
+ `ParallelPipeline` and `UserIdleProcessor`.
+
+- Fixed a type error when using `voice_settings` in `ElevenLabsHttpTTSService`.
+
+- Fixed an issue where `OpenAIRealtimeBetaLLMService` function calling resulted
+ in an error.
+
+- Fixed an issue in `AudioBufferProcessor` where the last audio buffer was not
+ being processed, in cases where the `_user_audio_buffer` was smaller than the
+ buffer size.
+
+### Performance
+
+- Replaced audio resampling library `resampy` with `soxr`. Resampling a 2:21s
+ audio file from 24KHz to 16KHz took 1.41s with `resampy` and 0.031s with
+ `soxr` with similar audio quality.
+
+### Other
+
+- Added initial unit test infrastructure.
+
+## [0.0.53] - 2025-01-18
+
+### Added
+
+- Added `ElevenLabsHttpTTSService` which uses EleveLabs' HTTP API instead of the
+ websocket one.
+
+- Introduced pipeline frame observers. Observers can view all the frames that go
+ through the pipeline without the need to inject processors in the
+ pipeline. This can be useful, for example, to implement frame loggers or
+ debuggers among other things. The example
+ `examples/foundational/30-observer.py` shows how to add an observer to a
+ pipeline for debugging.
+
+- Introduced heartbeat frames. The pipeline task can now push periodic
+ heartbeats down the pipeline when `enable_heartbeats=True`. Heartbeats are
+ system frames that are supposed to make it all the way to the end of the
+ pipeline. When a heartbeat frame is received the traversing time (i.e. the
+ time it took to go through the whole pipeline) will be displayed (with TRACE
+ logging) otherwise a warning will be shown. The example
+ `examples/foundational/31-heartbeats.py` shows how to enable heartbeats and
+ forces warnings to be displayed.
+
+- Added `LLMTextFrame` and `TTSTextFrame` which should be pushed by LLM and TTS
+ services respectively instead of `TextFrame`s.
+
- Added `OpenRouter` for OpenRouter integration with an OpenAI-compatible
interface. Added foundational example `14m-function-calling-openrouter.py`.
-
+
- Added a new `WebsocketService` based class for TTS services, containing
base functions and retry logic.
@@ -48,6 +173,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
+- Modified `UserIdleProcessor` to start monitoring only after first
+ conversation activity (`UserStartedSpeakingFrame` or
+ `BotStartedSpeakingFrame`) instead of immediately.
+
- Modified `OpenAIAssistantContextAggregator` to support controlled completions
and to emit context update callbacks via `FunctionCallResultProperties`.
@@ -71,6 +200,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- Fixed an issue where `DeepgramSTTService` was not generating metrics using
+ pipeline's VAD.
+
+- Fixed `UserIdleProcessor` not properly propagating `EndFrame`s through the
+ pipeline.
+
- Fixed an issue where websocket based TTS services could incorrectly terminate
their connection due to a retry counter not resetting.
@@ -87,6 +222,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Fixed an issue where setting the voice and model for `RimeHttpTTSService`
wasn't working.
+- Fixed an issue where `IdleFrameProcessor` and `UserIdleProcessor` were getting
+ initialized before the start of the pipeline.
+
## [0.0.52] - 2024-12-24
### Added
diff --git a/README.md b/README.md
index ae7a91538..5a915041d 100644
--- a/README.md
+++ b/README.md
@@ -2,7 +2,7 @@
-
-
AI Chatbot
+
+
+
+
+
+
+ Status: Disconnected
+
+
+ Connect
+ Disconnect
+
+
+
+
+
+
+
+
+
= ({ handleLeave }) => {
setTimeout(() => daily.setLocalAudio(true), 500);
setStoryState("user");
} else {
- daily.setLocalAudio(false);
+ // Uncomment the next line to mute the mic while the
+ // assistant it talking. Leave it commented to allow for interruptions
+ // daily.setLocalAudio(false);
setStoryState("assistant");
}
},
@@ -58,7 +60,7 @@ const Story: React.FC = ({ handleLeave }) => {
{participantIds.length >= 1 ? (
@@ -71,7 +73,7 @@ const Story: React.FC = ({ handleLeave }) => {
)}
([]);
+ const timeoutRef = useRef();
+
+ const resetTimeout = () => {
+ if (timeoutRef.current) {
+ clearTimeout(timeoutRef.current);
+ }
+ timeoutRef.current = setTimeout(() => {
+ setTranscription([]);
+ }, 5000);
+ };
+
+ useEffect(() => {
+ return () => {
+ if (timeoutRef.current) {
+ clearTimeout(timeoutRef.current);
+ }
+ };
+ }, []);
useAppMessage({
onAppMessage: (e: DailyEventObjectAppMessage) => {
if (e.fromId && e.fromId === "transcription") {
if (e.data.user_id === "" && e.data.is_final) {
setTranscription((t) => [...t, ...e.data.text.split(" ")]);
+ resetTimeout();
}
}
},
diff --git a/examples/storytelling-chatbot/requirements.txt b/examples/storytelling-chatbot/requirements.txt
index 0cebe6edb..970d3898c 100644
--- a/examples/storytelling-chatbot/requirements.txt
+++ b/examples/storytelling-chatbot/requirements.txt
@@ -2,4 +2,4 @@ async_timeout
fastapi
uvicorn
python-dotenv
-pipecat-ai[daily,elevenlabs,openai,fal]
+pipecat-ai[daily,openai,fal,google,cartesia]
diff --git a/examples/storytelling-chatbot/src/bot.py b/examples/storytelling-chatbot/src/bot.py
index bc4eb62cc..8c896dc86 100644
--- a/examples/storytelling-chatbot/src/bot.py
+++ b/examples/storytelling-chatbot/src/bot.py
@@ -13,17 +13,18 @@ import aiohttp
from dotenv import load_dotenv
from loguru import logger
from processors import StoryImageProcessor, StoryProcessor
-from prompts import CUE_USER_TURN, LLM_BASE_PROMPT, LLM_INTRO_PROMPT
+from prompts import CUE_USER_TURN, LLM_BASE_PROMPT
from utils.helpers import load_images, load_sounds
-from pipecat.frames.frames import EndFrame, LLMMessagesFrame, StopTaskFrame
+from pipecat.audio.vad.silero import SileroVADAnalyzer
+from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
-from pipecat.pipeline.task import PipelineTask
+from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.services.elevenlabs import ElevenLabsTTSService
from pipecat.services.fal import FalImageGenService
-from pipecat.services.openai import OpenAILLMService
+from pipecat.services.google import GoogleLLMService
from pipecat.transports.services.daily import (
DailyParams,
DailyTransport,
@@ -53,6 +54,7 @@ async def main(room_url, token=None):
camera_out_width=768,
camera_out_height=768,
transcription_enabled=True,
+ vad_analyzer=SileroVADAnalyzer(),
vad_enabled=True,
),
)
@@ -61,11 +63,10 @@ async def main(room_url, token=None):
# -------------- Services --------------- #
- llm_service = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
+ llm_service = GoogleLLMService(api_key=os.getenv("GOOGLE_API_KEY"))
tts_service = ElevenLabsTTSService(
- api_key=os.getenv("ELEVENLABS_API_KEY"),
- voice_id=os.getenv("ELEVENLABS_VOICE_ID"),
+ api_key=os.getenv("ELEVENLABS_API_KEY"), voice_id=os.getenv("ELEVENLABS_VOICE_ID")
)
fal_service_params = FalImageGenService.InputParams(
@@ -74,7 +75,7 @@ async def main(room_url, token=None):
fal_service = FalImageGenService(
aiohttp_session=session,
- model="fal-ai/fast-lightning-sdxl",
+ model="fal-ai/stable-diffusion-v35-medium",
params=fal_service_params,
key=os.getenv("FAL_KEY"),
)
@@ -97,35 +98,8 @@ async def main(room_url, token=None):
runner = PipelineRunner()
- # The intro pipeline is used to start
- # the story (as per LLM_INTRO_PROMPT)
- intro_pipeline = Pipeline([llm_service, tts_service, transport.output()])
-
- intro_task = PipelineTask(intro_pipeline)
-
logger.debug("Waiting for participant...")
- @transport.event_handler("on_first_participant_joined")
- async def on_first_participant_joined(transport, participant):
- logger.debug("Participant joined, storytime commence!")
- await transport.capture_participant_transcription(participant["id"])
- await intro_task.queue_frames(
- [
- images["book1"],
- LLMMessagesFrame([LLM_INTRO_PROMPT]),
- DailyTransportMessageFrame(CUE_USER_TURN),
- sounds["listening"],
- images["book2"],
- StopTaskFrame(),
- ]
- )
-
- # We run the intro pipeline. This will start the transport. The intro
- # task will exit after StopTaskFrame is processed.
- await runner.run(intro_task)
-
- # The main story pipeline is used to continue the story based on user
- # input.
main_pipeline = Pipeline(
[
transport.input(),
@@ -139,16 +113,39 @@ async def main(room_url, token=None):
]
)
- main_task = PipelineTask(main_pipeline)
+ main_task = PipelineTask(
+ main_pipeline,
+ PipelineParams(
+ allow_interruptions=True,
+ enable_metrics=True,
+ enable_usage_metrics=True,
+ report_only_initial_ttfb=True,
+ ),
+ )
+
+ @transport.event_handler("on_first_participant_joined")
+ async def on_first_participant_joined(transport, participant):
+ logger.debug("Participant joined, storytime commence!")
+ await transport.capture_participant_transcription(participant["id"])
+ await main_task.queue_frames(
+ [
+ images["book1"],
+ context_aggregator.user().get_context_frame(),
+ DailyTransportMessageFrame(CUE_USER_TURN),
+ # sounds["listening"],
+ images["book2"],
+ ]
+ )
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
- await intro_task.queue_frame(EndFrame())
- await main_task.queue_frame(EndFrame())
+ await main_task.cancel()
@transport.event_handler("on_call_state_updated")
async def on_call_state_updated(transport, state):
if state == "left":
+ # Here we don't want to cancel, we just want to finish sending
+ # whatever is queued, so we use an EndFrame().
await main_task.queue_frame(EndFrame())
await runner.run(main_task)
diff --git a/examples/storytelling-chatbot/src/bot_runner.py b/examples/storytelling-chatbot/src/bot_runner.py
index 97a60b72e..2d949c768 100644
--- a/examples/storytelling-chatbot/src/bot_runner.py
+++ b/examples/storytelling-chatbot/src/bot_runner.py
@@ -114,7 +114,7 @@ async def start_bot(request: Request) -> JSONResponse:
else:
try:
subprocess.Popen(
- [f"python3 -m bot -u {room.url} -t {token}"],
+ [f"python -m bot -u {room.url} -t {token}"],
shell=True,
bufsize=1,
cwd=os.path.dirname(os.path.abspath(__file__)),
@@ -175,7 +175,7 @@ async def virtualize_bot(room_url: str, token: str):
image = data[0]["config"]["image"]
# Machine configuration
- cmd = f"python3 src/bot.py -u {room_url} -t {token}"
+ cmd = f"python src/bot.py -u {room_url} -t {token}"
cmd = cmd.split()
worker_props = {
"config": {
@@ -215,7 +215,7 @@ async def virtualize_bot(room_url: str, token: str):
if __name__ == "__main__":
# Check environment variables
required_env_vars = [
- "OPENAI_API_KEY",
+ "GOOGLE_API_KEY",
"DAILY_API_KEY",
"FAL_KEY",
"ELEVENLABS_VOICE_ID",
diff --git a/examples/storytelling-chatbot/src/processors.py b/examples/storytelling-chatbot/src/processors.py
index 096efd577..6d30b543c 100644
--- a/examples/storytelling-chatbot/src/processors.py
+++ b/examples/storytelling-chatbot/src/processors.py
@@ -37,8 +37,7 @@ class StoryPromptFrame(TextFrame):
class StoryImageProcessor(FrameProcessor):
- """
- Processor for image prompt frames that will be sent to the FAL service.
+ """Processor for image prompt frames that will be sent to the FAL service.
This processor is responsible for consuming frames of type `StoryImageFrame`.
It processes them by passing it to the FAL service.
@@ -68,8 +67,7 @@ class StoryImageProcessor(FrameProcessor):
class StoryProcessor(FrameProcessor):
- """
- Primary frame processor. It takes the frames generated by the LLM
+ """Primary frame processor. It takes the frames generated by the LLM
and processes them into image prompts and story pages (sentences).
For a clearer picture of how this works, reference prompts.py
@@ -97,44 +95,10 @@ class StoryProcessor(FrameProcessor):
await self.push_frame(sounds["talking"])
elif isinstance(frame, TextFrame):
- # We want to look for sentence breaks in the text
- # but since TextFrames are streamed from the LLM
- # we need to keep a buffer of the text we've seen so far
+ # Add new text to the buffer
self._text += frame.text
-
- # IMAGE PROMPT
- # Looking for: < [image prompt] > in the LLM response
- # We prompted our LLM to add an image prompt in the response
- # so we use regex matching to find it and yield a StoryImageFrame
- if re.search(r"<.*?>", self._text):
- if not re.search(r"<.*?>.*?>", self._text):
- # Pass any frames until we have a closing bracket
- # otherwise the image prompt will be passed to TTS
- pass
- # Extract the image prompt from the text using regex
- image_prompt = re.search(r"<(.*?)>", self._text).group(1)
- # Remove the image prompt from the text
- self._text = re.sub(r"<.*?>", "", self._text, count=1)
- # Process the image prompt frame
- await self.push_frame(StoryImageFrame(image_prompt))
-
- # STORY PAGE
- # Looking for: [break] in the LLM response
- # We prompted our LLM to add a [break] after each sentence
- # so we use regex matching to find it in the LLM response
- if re.search(r".*\[[bB]reak\].*", self._text):
- # Remove the [break] token from the text
- # so it isn't spoken out loud by the TTS
- self._text = re.sub(r"\[[bB]reak\]", "", self._text, flags=re.IGNORECASE)
- self._text = self._text.replace("\n", " ")
- if len(self._text) > 2:
- # Append the sentence to the story
- self._story.append(self._text)
- await self.push_frame(StoryPageFrame(self._text))
- # Assert that it's the LLMs turn, until we're finished
- await self.push_frame(DailyTransportMessageFrame(CUE_ASSISTANT_TURN))
- # Clear the buffer
- self._text = ""
+ # Process any complete patterns in the order they appear
+ await self.process_text_content()
# End of a full LLM response
# Driven by the prompt, the LLM should have asked the user for input
@@ -150,3 +114,38 @@ class StoryProcessor(FrameProcessor):
# Anything that is not a TextFrame pass through
else:
await self.push_frame(frame)
+
+ async def process_text_content(self):
+ """Process text content in order of appearance, handling both image prompts and story breaks."""
+ while True:
+ # Find the first occurrence of each pattern
+ image_match = re.search(r"<(.*?)>", self._text)
+ break_match = re.search(r"\[[bB]reak\]", self._text)
+
+ # If neither pattern is found, we're done processing
+ if not image_match and not break_match:
+ break
+
+ # Find which pattern comes first in the text
+ image_pos = image_match.start() if image_match else float("inf")
+ break_pos = break_match.start() if break_match else float("inf")
+
+ if image_pos < break_pos:
+ # Process image prompt first
+ image_prompt = image_match.group(1)
+ # Remove the image prompt from the text
+ self._text = self._text[: image_match.start()] + self._text[image_match.end() :]
+ await self.push_frame(StoryImageFrame(image_prompt))
+ else:
+ # Process story break first
+ parts = re.split(r"\[[bB]reak\]", self._text, flags=re.IGNORECASE, maxsplit=1)
+ before_break = parts[0].replace("\n", " ").strip()
+
+ if len(before_break) > 2:
+ self._story.append(before_break)
+ await self.push_frame(StoryPageFrame(before_break))
+ # await self.push_frame(sounds["ding"])
+ await self.push_frame(DailyTransportMessageFrame(CUE_ASSISTANT_TURN))
+
+ # Keep the remainder (if any) in the buffer
+ self._text = parts[1].strip() if len(parts) > 1 else ""
diff --git a/examples/storytelling-chatbot/src/prompts.py b/examples/storytelling-chatbot/src/prompts.py
index 08abbc93c..186828932 100644
--- a/examples/storytelling-chatbot/src/prompts.py
+++ b/examples/storytelling-chatbot/src/prompts.py
@@ -1,31 +1,34 @@
-LLM_INTRO_PROMPT = {
- "role": "system",
- "content": "You are a creative storyteller who loves to tell whimsical, fantastical stories. \
- Your goal is to craft an engaging and fun story. \
- Start by asking the user what kind of story they'd like to hear. Don't provide any examples. \
- Keep your response to only a few sentences.",
-}
-
-
LLM_BASE_PROMPT = {
"role": "system",
- "content": "You are a creative storyteller who loves tell whimsical, fantastical stories. \
- Your goal is to craft an engaging and fun story. \
- Keep all responses short and no more than a few sentences. Include [break] after each sentence of the story. \
- \
- Start each sentence with an image prompt, wrapped in triangle braces, that I can use to generate an illustration representing the upcoming scene. \
- Image prompts should always be wrapped in triangle braces, like this: . \
- You should provide as much descriptive detail in your image prompt as you can to help recreate the current scene depicted by the sentence. \
- For any recurring characters, you should provide a description of them in the image prompt each time, for example: . \
- Please do not include any character names in the image prompts, just their descriptions. \
- Image prompts should focus on key visual attributes of all characters each time, for example . \
- Please use the following structure for your image prompts: characters, setting, action, and mood. \
- Image prompts should be less than 150-200 characters and start in lowercase. \
- \
- Responses should use the format: <...> story sentence [break] <...> story sentence [break] ... \
- After each response, ask me how I'd like the story to continue and wait for my input. \
- Please ensure your responses are less than 3-4 sentences long. \
- Please refrain from using any explicit language or content. Do not tell scary stories.",
+ "content": """You are a creative storyteller who loves tell whimsical, fantastical stories.
+ Your goal is to craft an engaging and fun story.
+ Keep all responses short and no more than a few sentences.
+ Start by asking the user what kind of story they'd like to hear. Don't provide any examples.
+ After they've answered the question, start telling the story. Include [break] after each sentence of the story.
+
+ Start each sentence with an image prompt, wrapped in triangle braces, that I can use to generate an illustration representing the upcoming scene.
+ Image prompts should always be wrapped in triangle braces, like this: .
+ You should provide as much descriptive detail in your image prompt as you can to help recreate the current scene depicted by the sentence.
+ For any recurring characters, you should provide a description of them in the image prompt each time, for example: .
+ Please do not include any character names in the image prompts, just their descriptions.
+ Image prompts should focus on key visual attributes of all characters each time, for example .
+ Please use the following structure for your image prompts: characters, setting, action, and mood.
+ Image prompts should be less than 150-200 characters and start in lowercase.
+
+ STORY SENTENCE OUTPUT FORMAT:
+
+ story sentence 1 [break]
+
+ story sentence 2 [break]
+
+ story sentence 3 [break]
+ How would you like the story to continue?
+ END OF EXAMPLE OUTPUT
+
+ Generate three story sentences, then ask what should happen next and wait for my input. You can propose an idea for how the story should proceed, but make sure to tell me I can suggest whatever I want. \
+ Please ensure your responses are less than 5 sentences long. \
+ Please refrain from using any explicit language or content. Do not tell scary stories.
+ Once you've started telling the story, EVERY RESPONSE should follow the story sentence output format. It is VERY IMPORTANT that you continue to include and [break] between story sentences. DO NOT RESPOND without image descriptions and break tags.""",
}
diff --git a/examples/studypal/studypal.py b/examples/studypal/studypal.py
index 8e45b609f..b4691fc5f 100644
--- a/examples/studypal/studypal.py
+++ b/examples/studypal/studypal.py
@@ -12,7 +12,6 @@ from pypdf import PdfReader
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
-from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -171,7 +170,7 @@ Your task is to help the user understand and learn from this article in 2 senten
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
- await task.queue_frame(EndFrame())
+ await task.cancel()
runner = PipelineRunner()
diff --git a/examples/translation-chatbot/bot.py b/examples/translation-chatbot/bot.py
index f9fd2377a..d35cf7af2 100644
--- a/examples/translation-chatbot/bot.py
+++ b/examples/translation-chatbot/bot.py
@@ -7,26 +7,33 @@
import asyncio
import os
import sys
+from typing import List
import aiohttp
from dotenv import load_dotenv
from loguru import logger
from runner import configure
-from pipecat.frames.frames import Frame, LLMMessagesFrame, TextFrame
+from pipecat.audio.vad.silero import SileroVADAnalyzer
+from pipecat.frames.frames import (
+ Frame,
+ LLMMessagesFrame,
+ TranscriptionFrame,
+ TranscriptionMessage,
+ TranscriptionUpdateFrame,
+)
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask
-from pipecat.processors.aggregators.llm_response import LLMFullResponseAggregator
-from pipecat.processors.aggregators.sentence import SentenceAggregator
+from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
-from pipecat.services.azure import AzureTTSService
+from pipecat.processors.transcript_processor import TranscriptProcessor
+from pipecat.services.cartesia import CartesiaTTSService
+from pipecat.services.deepgram import DeepgramSTTService
from pipecat.services.openai import OpenAILLMService
from pipecat.transports.services.daily import (
DailyParams,
- DailyTranscriptionSettings,
DailyTransport,
- DailyTransportMessageFrame,
)
load_dotenv(override=True)
@@ -44,18 +51,34 @@ It also isn't saving what the user or bot says into the context object for use i
# We need to use a custom service here to yield LLM frames without saving
# any context
class TranslationProcessor(FrameProcessor):
- def __init__(self, language):
+ """A processor that translates text frames from a source language to a target language."""
+
+ def __init__(self, in_language, out_language):
+ """Initialize the TranslationProcessor with source and target languages.
+
+ Args:
+ in_language (str): The language of the input text.
+ out_language (str): The language to translate the text into.
+ """
super().__init__()
- self._language = language
+ self._out_language = out_language
+ self._in_language = in_language
async def process_frame(self, frame: Frame, direction: FrameDirection):
+ """Process a frame and translate text frames.
+
+ Args:
+ frame (Frame): The frame to process.
+ direction (FrameDirection): The direction of the frame.
+ """
await super().process_frame(frame, direction)
- if isinstance(frame, TextFrame):
+ if isinstance(frame, TranscriptionFrame):
+ logger.debug(f"Translating {self._in_language}: {frame.text} to {self._out_language}")
context = [
{
"role": "system",
- "content": f"You will be provided with a sentence in English, and your task is to translate it into {self._language}.",
+ "content": f"You will be provided with a sentence in {self._in_language}, and your task is to only translate it into {self._out_language}.",
},
{"role": "user", "content": frame.text},
]
@@ -64,28 +87,45 @@ class TranslationProcessor(FrameProcessor):
await self.push_frame(frame)
-class TranslationSubtitles(FrameProcessor):
- def __init__(self, language):
- super().__init__()
- self._language = language
+class TranscriptHandler:
+ """Simple handler to demonstrate transcript processing.
- #
- # This doesn't do anything unless the receiver recognizes the message being
- # sent. For example, in this case, we are sending a message to the transport
- # so an application running at the other end of the transport could display
- # subtitles.
- #
- async def process_frame(self, frame: Frame, direction: FrameDirection):
- await super().process_frame(frame, direction)
+ Maintains a list of conversation messages and logs them with timestamps.
+ """
- if isinstance(frame, TextFrame):
- message = {"language": self._language, "text": frame.text}
- await self.push_frame(DailyTransportMessageFrame(message))
+ def __init__(self, in_language="English", out_language="Spanish"):
+ """Initialize the TranscriptHandler with an empty list of messages."""
+ self.messages: List[TranscriptionMessage] = []
+ self.in_language = in_language
+ self.out_language = out_language
- await self.push_frame(frame)
+ async def on_transcript_update(
+ self, processor: TranscriptProcessor, frame: TranscriptionUpdateFrame
+ ):
+ """Handle new transcript messages.
+
+ Args:
+ processor: The TranscriptProcessor that emitted the update
+ frame: TranscriptionUpdateFrame containing new messages
+ """
+ self.messages.extend(frame.messages)
+
+ # Log the new messages
+ logger.info("New transcript messages:")
+ for msg in frame.messages:
+ timestamp = f"[{msg.timestamp}] " if msg.timestamp else ""
+ message = {
+ "event": "translation",
+ "timestamp": msg.timestamp,
+ "role": msg.role,
+ "language": self.out_language if msg.role == "assistant" else self.in_language,
+ "text": msg.content,
+ }
+ logger.info(f"{timestamp}{msg.role}: {msg.content}")
async def main():
+ """Main function to set up and run the translation chatbot pipeline."""
async with aiohttp.ClientSession() as session:
(room_url, token) = await configure(session)
@@ -95,31 +135,60 @@ async def main():
"Translator",
DailyParams(
audio_out_enabled=True,
- transcription_enabled=True,
- transcription_settings=DailyTranscriptionSettings(extra={"interim_results": False}),
+ vad_enabled=True,
+ vad_analyzer=SileroVADAnalyzer(),
+ vad_audio_passthrough=True,
),
)
- tts = AzureTTSService(
- api_key=os.getenv("AZURE_SPEECH_API_KEY"),
- region=os.getenv("AZURE_SPEECH_REGION"),
- voice="es-ES-AlvaroNeural",
+ stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
+
+ tts = CartesiaTTSService(
+ api_key=os.getenv("CARTESIA_API_KEY"),
+ voice_id="34dbb662-8e98-413c-a1ef-1a3407675fe7", # Spanish Narrator Man
+ model="sonic-multilingual",
)
+ in_language = "English"
+ out_language = "Spanish"
+
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
+ context = OpenAILLMContext()
+ context_aggregator = llm.create_context_aggregator(context)
- sa = SentenceAggregator()
- tp = TranslationProcessor("Spanish")
- lfra = LLMFullResponseAggregator()
- ts = TranslationSubtitles("spanish")
+ tp = TranslationProcessor(in_language=in_language, out_language=out_language)
- pipeline = Pipeline([transport.input(), sa, tp, llm, lfra, ts, tts, transport.output()])
+ transcript = TranscriptProcessor()
+ transcript_handler = TranscriptHandler(in_language=in_language, out_language=out_language)
+
+ # Register event handler for transcript updates
+ @transcript.event_handler("on_transcript_update")
+ async def on_transcript_update(processor, frame):
+ await transcript_handler.on_transcript_update(processor, frame)
+
+ pipeline = Pipeline(
+ [
+ transport.input(),
+ stt,
+ transcript.user(), # User transcripts
+ tp,
+ llm,
+ tts,
+ transport.output(),
+ context_aggregator.assistant(),
+ transcript.assistant(), # Assistant transcripts
+ ]
+ )
task = PipelineTask(pipeline)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
- await transport.capture_participant_transcription(participant["id"])
+ logger.info("First participant joined")
+
+ @transport.event_handler("on_participant_left")
+ async def on_participant_left(transport, participant, reason):
+ await task.cancel()
runner = PipelineRunner()
diff --git a/examples/translation-chatbot/server.py b/examples/translation-chatbot/server.py
index b19242ae2..1dd2ac966 100644
--- a/examples/translation-chatbot/server.py
+++ b/examples/translation-chatbot/server.py
@@ -121,7 +121,7 @@ if __name__ == "__main__":
default_host = os.getenv("HOST", "0.0.0.0")
default_port = int(os.getenv("FAST_API_PORT", "7860"))
- parser = argparse.ArgumentParser(description="Daily Storyteller FastAPI server")
+ parser = argparse.ArgumentParser(description="Daily Translator FastAPI server")
parser.add_argument("--host", type=str, default=default_host, help="Host address")
parser.add_argument("--port", type=int, default=default_port, help="Port number")
parser.add_argument("--reload", action="store_true", help="Reload code on change")
diff --git a/examples/twilio-chatbot/bot.py b/examples/twilio-chatbot/bot.py
index aed7041f8..e715c02e1 100644
--- a/examples/twilio-chatbot/bot.py
+++ b/examples/twilio-chatbot/bot.py
@@ -11,7 +11,6 @@ from dotenv import load_dotenv
from loguru import logger
from pipecat.audio.vad.silero import SileroVADAnalyzer
-from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -85,7 +84,7 @@ async def run_bot(websocket_client, stream_sid):
@transport.event_handler("on_client_disconnected")
async def on_client_disconnected(transport, client):
- await task.queue_frames([EndFrame()])
+ await task.cancel()
runner = PipelineRunner(handle_sigint=False)
diff --git a/pyproject.toml b/pyproject.toml
index d0e7cba79..0d5e605d4 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -20,18 +20,19 @@ classifiers = [
"Topic :: Scientific/Engineering :: Artificial Intelligence"
]
dependencies = [
- "aiohttp~=3.11.10",
+ "aiohttp~=3.11.11",
"audioop-lts~=0.2.1; python_version>='3.13'",
+ # We need an older version of `httpx` that doesn't remove the deprecated
+ # `proxies` argument. This is necessary for Azure and Anthropic clients.
+ "httpx~=0.27.2",
"loguru~=0.7.3",
"Markdown~=3.7",
- "numpy~=2.1.3",
- "numba~=0.61.0rc1",
- "Pillow~=11.0.0",
- "protobuf~=5.29.1",
- "pydantic~=2.10.3",
+ "numpy~=1.26.4",
+ "Pillow~=11.1.0",
+ "protobuf~=5.29.3",
+ "pydantic~=2.10.5",
"pyloudnorm~=0.1.1",
- "resampy~=0.4.3",
- "tenacity~=9.0.0"
+ "soxr~=0.5.0"
]
[project.urls]
@@ -39,59 +40,65 @@ Source = "https://github.com/pipecat-ai/pipecat"
Website = "https://pipecat.ai"
[project.optional-dependencies]
-anthropic = [ "anthropic~=0.40.0" ]
-assemblyai = [ "assemblyai~=0.34.0" ]
-aws = [ "boto3~=1.35.27" ]
-azure = [ "azure-cognitiveservices-speech~=1.41.1", "openai~=1.59.0" ]
+anthropic = [ "anthropic~=0.39.0" ]
+assemblyai = [ "assemblyai~=0.36.0" ]
+aws = [ "boto3~=1.35.99" ]
+azure = [ "azure-cognitiveservices-speech~=1.42.0", "openai~=1.59.6" ]
canonical = [ "aiofiles~=24.1.0" ]
-cartesia = [ "cartesia~=1.0.13", "websockets~=13.1" ]
-cerebras = [ "openai~=1.59.0" ]
-deepseek = [ "openai~=1.59.0" ]
+cartesia = [ "cartesia~=1.3.1", "websockets~=13.1" ]
+cerebras = [ "openai~=1.59.6" ]
+deepseek = [ "openai~=1.59.6" ]
daily = [ "daily-python~=0.14.2" ]
-deepgram = [ "deepgram-sdk~=3.7.7" ]
+deepgram = [ "deepgram-sdk~=3.8.0" ]
elevenlabs = [ "websockets~=13.1" ]
-fal = [ "fal-client~=0.4.1" ]
+fal = [ "fal-client~=0.5.6" ]
fish = [ "ormsgpack~=1.7.0", "websockets~=13.1" ]
gladia = [ "websockets~=13.1" ]
-google = [ "google-generativeai~=0.8.3", "google-cloud-texttospeech~=2.21.1" ]
-grok = [ "openai~=1.59.0" ]
-groq = [ "openai~=1.59.0" ]
-gstreamer = [ "pygobject~=3.48.2" ]
-fireworks = [ "openai~=1.59.0" ]
+google = [ "google-generativeai~=0.8.3", "google-cloud-texttospeech~=2.24.0" ]
+grok = [ "openai~=1.59.6" ]
+groq = [ "openai~=1.59.6" ]
+gstreamer = [ "pygobject~=3.50.0" ]
+fireworks = [ "openai~=1.59.6" ]
krisp = [ "pipecat-ai-krisp~=0.3.0" ]
-koala = [ "pvkoala~=2.0.2" ]
-langchain = [ "langchain~=0.3.12", "langchain-community~=0.3.12", "langchain-openai~=0.2.12" ]
-livekit = [ "livekit~=0.17.5", "livekit-api~=0.7.1" ]
-lmnt = [ "lmnt~=1.1.4" ]
+koala = [ "pvkoala~=2.0.3" ]
+langchain = [ "langchain~=0.3.14", "langchain-community~=0.3.14", "langchain-openai~=0.3.0" ]
+livekit = [ "livekit~=0.19.1", "livekit-api~=0.8.1", "tenacity~=9.0.0" ]
+lmnt = [ "websockets~=13.1" ]
local = [ "pyaudio~=0.2.14" ]
-moondream = [ "einops~=0.8.0", "timm~=1.0.8", "transformers~=4.44.0" ]
-nim = [ "openai~=1.59.0" ]
+moondream = [ "einops~=0.8.0", "timm~=1.0.13", "transformers~=4.48.0" ]
+nim = [ "openai~=1.59.6" ]
noisereduce = [ "noisereduce~=3.0.3" ]
-openai = [ "openai~=1.59.0", "websockets~=13.1", "python-deepcompare~=1.0.1" ]
-openpipe = [ "openpipe~=4.40.0" ]
-playht = [ "pyht~=0.1.9", "websockets~=13.1" ]
+openai = [ "openai~=1.59.6", "websockets~=13.1", "python-deepcompare~=2.1.0" ]
+openpipe = [ "openpipe~=4.43.0" ]
+playht = [ "pyht~=0.1.6", "websockets~=13.1" ]
riva = [ "nvidia-riva-client~=2.18.0" ]
silero = [ "onnxruntime~=1.20.1" ]
-simli = [ "simli-ai~=0.1.7"]
-soundfile = [ "soundfile~=0.12.1" ]
-together = [ "openai~=1.59.0" ]
-websocket = [ "websockets~=13.1", "fastapi~=0.115.0" ]
-whisper = [ "faster-whisper~=1.1.0" ]
-openrouter = [ "openai~=1.59.0" ]
+simli = [ "simli-ai~=0.1.10"]
+soundfile = [ "soundfile~=0.13.0" ]
+together = [ "openai~=1.59.6" ]
+websocket = [ "websockets~=13.1", "fastapi~=0.115.6" ]
+whisper = [ "faster-whisper~=1.1.1" ]
+openrouter = [ "openai~=1.59.6" ]
[tool.setuptools.packages.find]
# All the following settings are optional:
where = ["src"]
[tool.pytest.ini_options]
+addopts = "--verbose"
+testpaths = ["tests"]
pythonpath = ["src"]
+asyncio_default_fixture_loop_scope = "function"
+filterwarnings = [
+ "ignore:'audioop' is deprecated:DeprecationWarning",
+]
[tool.setuptools_scm]
local_scheme = "no-local-version"
fallback_version = "0.0.0-dev"
[tool.ruff]
-exclude = ["*_pb2.py"]
+exclude = [".git", "*_pb2.py"]
line-length = 100
[tool.ruff.lint]
diff --git a/scripts/pre-commit.sh b/scripts/pre-commit.sh
new file mode 100755
index 000000000..22f00313d
--- /dev/null
+++ b/scripts/pre-commit.sh
@@ -0,0 +1,3 @@
+#!/bin/sh
+
+NO_COLOR=1 ruff format --diff
diff --git a/src/pipecat/audio/utils.py b/src/pipecat/audio/utils.py
index 5706c2c5f..7ed44befd 100644
--- a/src/pipecat/audio/utils.py
+++ b/src/pipecat/audio/utils.py
@@ -8,14 +8,14 @@ import audioop
import numpy as np
import pyloudnorm as pyln
-import resampy
+import soxr
def resample_audio(audio: bytes, original_rate: int, target_rate: int) -> bytes:
if original_rate == target_rate:
return audio
audio_data = np.frombuffer(audio, dtype=np.int16)
- resampled_audio = resampy.resample(audio_data, original_rate, target_rate)
+ resampled_audio = soxr.resample(audio_data, original_rate, target_rate)
return resampled_audio.astype(np.int16).tobytes()
@@ -80,14 +80,14 @@ def ulaw_to_pcm(ulaw_bytes: bytes, in_sample_rate: int, out_sample_rate: int):
in_pcm_bytes = audioop.ulaw2lin(ulaw_bytes, 2)
# Resample
- out_pcm_bytes = audioop.ratecv(in_pcm_bytes, 2, 1, in_sample_rate, out_sample_rate, None)[0]
+ out_pcm_bytes = resample_audio(in_pcm_bytes, in_sample_rate, out_sample_rate)
return out_pcm_bytes
def pcm_to_ulaw(pcm_bytes: bytes, in_sample_rate: int, out_sample_rate: int):
# Resample
- in_pcm_bytes = audioop.ratecv(pcm_bytes, 2, 1, in_sample_rate, out_sample_rate, None)[0]
+ in_pcm_bytes = resample_audio(pcm_bytes, in_sample_rate, out_sample_rate)
# Convert PCM to μ-law
ulaw_bytes = audioop.lin2ulaw(in_pcm_bytes, 2)
diff --git a/src/pipecat/audio/vad/silero.py b/src/pipecat/audio/vad/silero.py
index fab01ae70..00a1358a5 100644
--- a/src/pipecat/audio/vad/silero.py
+++ b/src/pipecat/audio/vad/silero.py
@@ -159,5 +159,5 @@ class SileroVADAnalyzer(VADAnalyzer):
return new_confidence
except Exception as e:
# This comes from an empty audio array
- logger.exception(f"Error analyzing audio with Silero VAD: {e}")
+ logger.error(f"Error analyzing audio with Silero VAD: {e}")
return 0
diff --git a/src/pipecat/frames/frames.py b/src/pipecat/frames/frames.py
index 321f8514b..51d278a93 100644
--- a/src/pipecat/frames/frames.py
+++ b/src/pipecat/frames/frames.py
@@ -5,7 +5,8 @@
#
from dataclasses import dataclass, field
-from typing import Any, Awaitable, Callable, List, Literal, Mapping, Optional, Tuple
+from enum import Enum
+from typing import TYPE_CHECKING, Any, Awaitable, Callable, List, Literal, Mapping, Optional, Tuple
from pipecat.audio.vad.vad_analyzer import VADParams
from pipecat.clocks.base_clock import BaseClock
@@ -14,6 +15,26 @@ from pipecat.transcriptions.language import Language
from pipecat.utils.time import nanoseconds_to_str
from pipecat.utils.utils import obj_count, obj_id
+if TYPE_CHECKING:
+ from pipecat.observers.base_observer import BaseObserver
+
+
+class KeypadEntry(str, Enum):
+ """DTMF entries."""
+
+ ONE = "1"
+ TWO = "2"
+ THREE = "3"
+ FOUR = "4"
+ FIVE = "5"
+ SIX = "6"
+ SEVEN = "7"
+ EIGHT = "8"
+ NINE = "9"
+ ZERO = "0"
+ POUND = "#"
+ STAR = "*"
+
def format_pts(pts: int | None):
return nanoseconds_to_str(pts) if pts else None
@@ -26,11 +47,13 @@ class Frame:
id: int = field(init=False)
name: str = field(init=False)
pts: Optional[int] = field(init=False)
+ metadata: dict = field(init=False)
def __post_init__(self):
self.id: int = obj_id()
self.name: str = f"{self.__class__.__name__}#{obj_count(self)}"
self.pts: Optional[int] = None
+ self.metadata: dict = {}
def __str__(self):
return self.name
@@ -177,6 +200,20 @@ class TextFrame(DataFrame):
return f"{self.name}(pts: {pts}, text: [{self.text}])"
+@dataclass
+class LLMTextFrame(TextFrame):
+ """A text frame generated by LLM services."""
+
+ pass
+
+
+@dataclass
+class TTSTextFrame(TextFrame):
+ """A text frame generated by TTS services."""
+
+ pass
+
+
@dataclass
class TranscriptionFrame(TextFrame):
"""A text frame with transcription-specific data. Will be placed in the
@@ -358,6 +395,13 @@ class TransportMessageFrame(DataFrame):
return f"{self.name}(message: {self.message})"
+@dataclass
+class InputDTMFFrame(DataFrame):
+ """A DTMF button input"""
+
+ button: KeypadEntry
+
+
#
# System frames
#
@@ -372,6 +416,7 @@ class StartFrame(SystemFrame):
enable_metrics: bool = False
enable_usage_metrics: bool = False
report_only_initial_ttfb: bool = False
+ observer: Optional["BaseObserver"] = None
@dataclass
@@ -406,6 +451,16 @@ class FatalErrorFrame(ErrorFrame):
fatal: bool = field(default=True, init=False)
+@dataclass
+class HeartbeatFrame(SystemFrame):
+ """This frame is used by the pipeline task as a mechanism to know if the
+ pipeline is running properly.
+
+ """
+
+ timestamp: int
+
+
@dataclass
class EndTaskFrame(SystemFrame):
"""This is used to notify the pipeline task that the pipeline should be
diff --git a/src/pipecat/observers/__init__.py b/src/pipecat/observers/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/src/pipecat/observers/base_observer.py b/src/pipecat/observers/base_observer.py
new file mode 100644
index 000000000..46f746946
--- /dev/null
+++ b/src/pipecat/observers/base_observer.py
@@ -0,0 +1,44 @@
+#
+# Copyright (c) 2024–2025, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+from abc import ABC, abstractmethod
+
+from pipecat.frames.frames import Frame
+from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
+
+
+class BaseObserver(ABC):
+ """This is the base class for pipeline frame observers. Observers can view
+ all the frames that go through the pipeline without the need to inject
+ processors in the pipeline. This can be useful, for example, to implement
+ frame loggers or debuggers among other things.
+
+ """
+
+ @abstractmethod
+ async def on_push_frame(
+ self,
+ src: FrameProcessor,
+ dst: FrameProcessor,
+ frame: Frame,
+ direction: FrameDirection,
+ timestamp: int,
+ ):
+ """Abstract method to handle the event when a frame is pushed from one
+ processor to another.
+
+ Args:
+ src (FrameProcessor): The source frame processor that is sending the frame.
+ dst (FrameProcessor): The destination frame processor that will receive the frame.
+ frame (Frame): The frame being transferred between processors.
+ direction (FrameDirection): The direction of the frame transfer.
+ timestamp (int): The timestamp when the frame was pushed (based on the pipeline clock).
+
+ This method should be implemented by subclasses to define specific behavior
+ when a frame is pushed.
+
+ """
+ pass
diff --git a/src/pipecat/pipeline/base_task.py b/src/pipecat/pipeline/base_task.py
new file mode 100644
index 000000000..714c721e3
--- /dev/null
+++ b/src/pipecat/pipeline/base_task.py
@@ -0,0 +1,68 @@
+#
+# Copyright (c) 2024–2025, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+from abc import ABC, abstractmethod
+from typing import AsyncIterable, Iterable
+
+from pipecat.frames.frames import Frame
+
+
+class BaseTask(ABC):
+ @property
+ @abstractmethod
+ def id(self) -> int:
+ """Returns the unique indetifier for this task."""
+ pass
+
+ @property
+ @abstractmethod
+ def name(self) -> str:
+ """Returns the name of this task."""
+ pass
+
+ @abstractmethod
+ def has_finished(self) -> bool:
+ """Indicates whether the tasks has finished. That is, all processors
+ have stopped.
+
+ """
+ pass
+
+ @abstractmethod
+ async def stop_when_done(self):
+ """This is a helper function that sends an EndFrame to the pipeline in
+ order to stop the task after everything in it has been processed.
+
+ """
+ pass
+
+ @abstractmethod
+ async def cancel(self):
+ """
+ Stops the running pipeline immediately.
+ """
+ pass
+
+ @abstractmethod
+ async def run(self):
+ """
+ Starts running the given pipeline.
+ """
+ pass
+
+ @abstractmethod
+ async def queue_frame(self, frame: Frame):
+ """
+ Queue a frame to be pushed down the pipeline.
+ """
+ pass
+
+ @abstractmethod
+ async def queue_frames(self, frames: Iterable[Frame] | AsyncIterable[Frame]):
+ """
+ Queues multiple frames to be pushed down the pipeline.
+ """
+ pass
diff --git a/src/pipecat/pipeline/parallel_pipeline.py b/src/pipecat/pipeline/parallel_pipeline.py
index 874a8c3a5..f77d09cc6 100644
--- a/src/pipecat/pipeline/parallel_pipeline.py
+++ b/src/pipecat/pipeline/parallel_pipeline.py
@@ -23,7 +23,7 @@ from pipecat.pipeline.pipeline import Pipeline
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
-class Source(FrameProcessor):
+class ParallelPipelineSource(FrameProcessor):
def __init__(
self,
upstream_queue: asyncio.Queue,
@@ -46,7 +46,7 @@ class Source(FrameProcessor):
await self.push_frame(frame, direction)
-class Sink(FrameProcessor):
+class ParallelPipelineSink(FrameProcessor):
def __init__(
self,
downstream_queue: asyncio.Queue,
@@ -92,8 +92,8 @@ class ParallelPipeline(BasePipeline):
raise TypeError(f"ParallelPipeline argument {processors} is not a list")
# We will add a source before the pipeline and a sink after.
- source = Source(self._up_queue, self._parallel_push_frame)
- sink = Sink(self._down_queue, self._parallel_push_frame)
+ source = ParallelPipelineSource(self._up_queue, self._parallel_push_frame)
+ sink = ParallelPipelineSink(self._down_queue, self._parallel_push_frame)
self._sources.append(source)
self._sinks.append(sink)
@@ -117,6 +117,7 @@ class ParallelPipeline(BasePipeline):
#
async def cleanup(self):
+ await super().cleanup()
await asyncio.gather(*[s.cleanup() for s in self._sources])
await asyncio.gather(*[p.cleanup() for p in self._pipelines])
await asyncio.gather(*[s.cleanup() for s in self._sinks])
@@ -150,22 +151,18 @@ class ParallelPipeline(BasePipeline):
async def _stop(self):
# The up task doesn't receive an EndFrame, so we just cancel it.
- self._up_task.cancel()
- await self._up_task
- # The down tasks waits for the last EndFrame send by the internal
+ await self.cancel_task(self._up_task)
+ # The down tasks waits for the last EndFrame sent by the internal
# pipelines.
await self._down_task
async def _cancel(self):
- self._up_task.cancel()
- await self._up_task
- self._down_task.cancel()
- await self._down_task
+ await self.cancel_task(self._up_task)
+ await self.cancel_task(self._down_task)
async def _create_tasks(self):
- loop = self.get_event_loop()
- self._up_task = loop.create_task(self._process_up_queue())
- self._down_task = loop.create_task(self._process_down_queue())
+ self._up_task = self.create_task(self._process_up_queue())
+ self._down_task = self.create_task(self._process_down_queue())
async def _drain_queues(self):
while not self._up_queue.empty:
@@ -185,32 +182,26 @@ class ParallelPipeline(BasePipeline):
async def _process_up_queue(self):
while True:
- try:
- frame = await self._up_queue.get()
- await self._parallel_push_frame(frame, FrameDirection.UPSTREAM)
- self._up_queue.task_done()
- except asyncio.CancelledError:
- break
+ frame = await self._up_queue.get()
+ await self._parallel_push_frame(frame, FrameDirection.UPSTREAM)
+ self._up_queue.task_done()
async def _process_down_queue(self):
running = True
while running:
- try:
- frame = await self._down_queue.get()
+ frame = await self._down_queue.get()
- endframe_counter = self._endframe_counter.get(frame.id, 0)
+ endframe_counter = self._endframe_counter.get(frame.id, 0)
- # If we have a counter, decrement it.
- if endframe_counter > 0:
- self._endframe_counter[frame.id] -= 1
- endframe_counter = self._endframe_counter[frame.id]
+ # If we have a counter, decrement it.
+ if endframe_counter > 0:
+ self._endframe_counter[frame.id] -= 1
+ endframe_counter = self._endframe_counter[frame.id]
- # If we don't have a counter or we reached 0, push the frame.
- if endframe_counter == 0:
- await self._parallel_push_frame(frame, FrameDirection.DOWNSTREAM)
+ # If we don't have a counter or we reached 0, push the frame.
+ if endframe_counter == 0:
+ await self._parallel_push_frame(frame, FrameDirection.DOWNSTREAM)
- running = not (endframe_counter == 0 and isinstance(frame, EndFrame))
+ running = not (endframe_counter == 0 and isinstance(frame, EndFrame))
- self._down_queue.task_done()
- except asyncio.CancelledError:
- break
+ self._down_queue.task_done()
diff --git a/src/pipecat/pipeline/pipeline.py b/src/pipecat/pipeline/pipeline.py
index e6b874ce9..270e100bd 100644
--- a/src/pipecat/pipeline/pipeline.py
+++ b/src/pipecat/pipeline/pipeline.py
@@ -71,6 +71,7 @@ class Pipeline(BasePipeline):
#
async def cleanup(self):
+ await super().cleanup()
await self._cleanup_processors()
async def process_frame(self, frame: Frame, direction: FrameDirection):
diff --git a/src/pipecat/pipeline/runner.py b/src/pipecat/pipeline/runner.py
index 34f7a0a8c..9963454b6 100644
--- a/src/pipecat/pipeline/runner.py
+++ b/src/pipecat/pipeline/runner.py
@@ -10,6 +10,7 @@ import signal
from loguru import logger
from pipecat.pipeline.task import PipelineTask
+from pipecat.utils.asyncio import current_tasks
from pipecat.utils.utils import obj_count, obj_id
@@ -19,6 +20,7 @@ class PipelineRunner:
self.name: str = name or f"{self.__class__.__name__}#{obj_count(self)}"
self._tasks = {}
+ self._sig_task = None
if handle_sigint:
self._setup_sigint()
@@ -28,6 +30,11 @@ class PipelineRunner:
self._tasks[task.name] = task
await task.run()
del self._tasks[task.name]
+ # If we are cancelling through a signal, make sure we wait for it so
+ # everything gets cleaned up nicely.
+ if self._sig_task:
+ await self._sig_task
+ self._print_dangling_tasks()
logger.debug(f"Runner {self} finished running {task}")
async def stop_when_done(self):
@@ -40,16 +47,21 @@ class PipelineRunner:
def _setup_sigint(self):
loop = asyncio.get_running_loop()
- loop.add_signal_handler(
- signal.SIGINT, lambda *args: asyncio.create_task(self._sig_handler())
- )
- loop.add_signal_handler(
- signal.SIGTERM, lambda *args: asyncio.create_task(self._sig_handler())
- )
+ loop.add_signal_handler(signal.SIGINT, lambda *args: self._sig_handler())
+ loop.add_signal_handler(signal.SIGTERM, lambda *args: self._sig_handler())
- async def _sig_handler(self):
+ def _sig_handler(self):
+ if not self._sig_task:
+ self._sig_task = asyncio.create_task(self._sig_cancel())
+
+ async def _sig_cancel(self):
logger.warning(f"Interruption detected. Canceling runner {self}")
await self.cancel()
+ def _print_dangling_tasks(self):
+ tasks = [t.get_name() for t in current_tasks()]
+ if tasks:
+ logger.warning(f"Dangling tasks detected: {tasks}")
+
def __str__(self):
return self.name
diff --git a/src/pipecat/pipeline/sync_parallel_pipeline.py b/src/pipecat/pipeline/sync_parallel_pipeline.py
index 78ddff074..7fa3577a5 100644
--- a/src/pipecat/pipeline/sync_parallel_pipeline.py
+++ b/src/pipecat/pipeline/sync_parallel_pipeline.py
@@ -24,7 +24,7 @@ class SyncFrame(ControlFrame):
pass
-class Source(FrameProcessor):
+class SyncParallelPipelineSource(FrameProcessor):
def __init__(self, upstream_queue: asyncio.Queue):
super().__init__()
self._up_queue = upstream_queue
@@ -39,7 +39,7 @@ class Source(FrameProcessor):
await self.push_frame(frame, direction)
-class Sink(FrameProcessor):
+class SyncParallelPipelineSink(FrameProcessor):
def __init__(self, downstream_queue: asyncio.Queue):
super().__init__()
self._down_queue = downstream_queue
@@ -76,8 +76,8 @@ class SyncParallelPipeline(BasePipeline):
# We add a source at the beginning of the pipeline and a sink at the end.
up_queue = asyncio.Queue()
down_queue = asyncio.Queue()
- source = Source(up_queue)
- sink = Sink(down_queue)
+ source = SyncParallelPipelineSource(up_queue)
+ sink = SyncParallelPipelineSink(down_queue)
processors: List[FrameProcessor] = [source] + processors + [sink]
# Keep track of sources and sinks. We also keep the output queue of
@@ -101,6 +101,10 @@ class SyncParallelPipeline(BasePipeline):
# Frame processor
#
+ async def cleanup(self):
+ await super().cleanup()
+ await asyncio.gather(*[p.cleanup() for p in self._pipelines])
+
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
diff --git a/src/pipecat/pipeline/task.py b/src/pipecat/pipeline/task.py
index 79a708c74..b862583ef 100644
--- a/src/pipecat/pipeline/task.py
+++ b/src/pipecat/pipeline/task.py
@@ -5,10 +5,10 @@
#
import asyncio
-from typing import AsyncIterable, Iterable
+from typing import AsyncIterable, Iterable, List
from loguru import logger
-from pydantic import BaseModel
+from pydantic import BaseModel, ConfigDict
from pipecat.clocks.base_clock import BaseClock
from pipecat.clocks.system_clock import SystemClock
@@ -19,27 +19,47 @@ from pipecat.frames.frames import (
EndTaskFrame,
ErrorFrame,
Frame,
+ HeartbeatFrame,
MetricsFrame,
StartFrame,
StopTaskFrame,
)
from pipecat.metrics.metrics import ProcessingMetricsData, TTFBMetricsData
+from pipecat.observers.base_observer import BaseObserver
from pipecat.pipeline.base_pipeline import BasePipeline
+from pipecat.pipeline.base_task import BaseTask
+from pipecat.pipeline.task_observer import TaskObserver
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
+from pipecat.utils.asyncio import cancel_task, create_task, wait_for_task
from pipecat.utils.utils import obj_count, obj_id
+HEARTBEAT_SECONDS = 1.0
+HEARTBEAT_MONITOR_SECONDS = HEARTBEAT_SECONDS * 5
+
class PipelineParams(BaseModel):
+ model_config = ConfigDict(arbitrary_types_allowed=True)
+
allow_interruptions: bool = False
+ enable_heartbeats: bool = False
enable_metrics: bool = False
enable_usage_metrics: bool = False
send_initial_empty_metrics: bool = True
report_only_initial_ttfb: bool = False
+ observers: List[BaseObserver] = []
+ heartbeats_period_secs: float = HEARTBEAT_SECONDS
-class Source(FrameProcessor):
- def __init__(self, up_queue: asyncio.Queue):
- super().__init__()
+class PipelineTaskSource(FrameProcessor):
+ """This is the source processor that is linked at the beginning of the
+ pipeline given to the pipeline task. It allows us to easily push frames
+ downstream to the pipeline and also receive upstream frames coming from the
+ pipeline.
+
+ """
+
+ def __init__(self, up_queue: asyncio.Queue, **kwargs):
+ super().__init__(**kwargs)
self._up_queue = up_queue
async def process_frame(self, frame: Frame, direction: FrameDirection):
@@ -47,92 +67,127 @@ class Source(FrameProcessor):
match direction:
case FrameDirection.UPSTREAM:
- await self._handle_upstream_frame(frame)
+ await self._up_queue.put(frame)
case FrameDirection.DOWNSTREAM:
await self.push_frame(frame, direction)
- async def _handle_upstream_frame(self, frame: Frame):
- if isinstance(frame, EndTaskFrame):
- # Tell the task we should end nicely.
- await self._up_queue.put(EndTaskFrame())
- elif isinstance(frame, CancelTaskFrame):
- # Tell the task we should end right away.
- await self._up_queue.put(CancelTaskFrame())
- elif isinstance(frame, ErrorFrame):
- logger.error(f"Error running app: {frame}")
- if frame.fatal:
- # Cancel all tasks downstream.
- await self.push_frame(CancelFrame())
- # Tell the task we should stop.
- await self._up_queue.put(StopTaskFrame())
+class PipelineTaskSink(FrameProcessor):
+ """This is the sink processor that is linked at the end of the pipeline
+ given to the pipeline task. It allows us to receive downstream frames and
+ act on them, for example, waiting to receive an EndFrame.
-class Sink(FrameProcessor):
- def __init__(self, down_queue: asyncio.Queue):
- super().__init__()
+ """
+
+ def __init__(self, down_queue: asyncio.Queue, **kwargs):
+ super().__init__(**kwargs)
self._down_queue = down_queue
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
-
- # We really just want to know when the EndFrame reached the sink.
- if isinstance(frame, EndFrame):
- await self._down_queue.put(frame)
+ await self._down_queue.put(frame)
-class PipelineTask:
+class PipelineTask(BaseTask):
def __init__(
self,
pipeline: BasePipeline,
params: PipelineParams = PipelineParams(),
clock: BaseClock = SystemClock(),
):
- self.id: int = obj_id()
- self.name: str = f"{self.__class__.__name__}#{obj_count(self)}"
+ self._id: int = obj_id()
+ self._name: str = f"{self.__class__.__name__}#{obj_count(self)}"
self._pipeline = pipeline
self._clock = clock
self._params = params
self._finished = False
+ # This queue receives frames coming from the pipeline upstream.
self._up_queue = asyncio.Queue()
+ # This queue receives frames coming from the pipeline downstream.
self._down_queue = asyncio.Queue()
+ # This queue is the queue used to push frames to the pipeline.
self._push_queue = asyncio.Queue()
+ # This is the heartbeat queue. When a heartbeat frame is received in the
+ # down queue we add it to the heartbeat queue for processing.
+ self._heartbeat_queue = asyncio.Queue()
+ # This event is used to indicate an EndFrame has been received in the
+ # down queue.
+ self._endframe_event = asyncio.Event()
- self._source = Source(self._up_queue)
+ self._source = PipelineTaskSource(self._up_queue)
self._source.link(pipeline)
- self._sink = Sink(self._down_queue)
+ self._sink = PipelineTaskSink(self._down_queue)
pipeline.link(self._sink)
- def has_finished(self):
+ self._observer = TaskObserver(params.observers)
+
+ @property
+ def id(self) -> int:
+ """Returns the unique indetifier for this task."""
+ return self._id
+
+ @property
+ def name(self) -> str:
+ """Returns the name of this task."""
+ return self._name
+
+ def has_finished(self) -> bool:
+ """Indicates whether the tasks has finished. That is, all processors
+ have stopped.
+
+ """
return self._finished
async def stop_when_done(self):
+ """This is a helper function that sends an EndFrame to the pipeline in
+ order to stop the task after everything in it has been processed.
+
+ """
logger.debug(f"Task {self} scheduled to stop when done")
await self.queue_frame(EndFrame())
async def cancel(self):
+ """
+ Stops the running pipeline immediately.
+ """
logger.debug(f"Canceling pipeline task {self}")
# Make sure everything is cleaned up downstream. This is sent
# out-of-band from the main streaming task which is what we want since
# we want to cancel right away.
await self._source.push_frame(CancelFrame())
- self._process_push_task.cancel()
- self._process_up_task.cancel()
- await self._process_push_task
- await self._process_up_task
+ # Only cancel the push task. Everything else will be cancelled in run().
+ await cancel_task(self._process_push_task)
async def run(self):
- self._process_up_task = asyncio.create_task(self._process_up_queue())
- self._process_push_task = asyncio.create_task(self._process_push_queue())
- await asyncio.gather(self._process_up_task, self._process_push_task)
+ """
+ Starts running the given pipeline.
+ """
+ try:
+ push_task = self._create_tasks()
+ await wait_for_task(push_task)
+ except asyncio.CancelledError:
+ # We are awaiting on the push task and it might be cancelled
+ # (e.g. Ctrl-C). This means we will get a CancelledError here as
+ # well, because you get a CancelledError in every place you are
+ # awaiting a task.
+ pass
+ await self._cancel_tasks()
+ await self._cleanup()
self._finished = True
async def queue_frame(self, frame: Frame):
+ """
+ Queue a frame to be pushed down the pipeline.
+ """
await self._push_queue.put(frame)
async def queue_frames(self, frames: Iterable[Frame] | AsyncIterable[Frame]):
+ """
+ Queues multiple frames to be pushed down the pipeline.
+ """
if isinstance(frames, AsyncIterable):
async for frame in frames:
await self.queue_frame(frame)
@@ -140,6 +195,43 @@ class PipelineTask:
for frame in frames:
await self.queue_frame(frame)
+ def _create_tasks(self):
+ loop = asyncio.get_running_loop()
+ self._process_up_task = create_task(
+ loop, self._process_up_queue(), f"{self}::_process_up_queue"
+ )
+ self._process_down_task = create_task(
+ loop, self._process_down_queue(), f"{self}::_process_down_queue"
+ )
+ self._process_push_task = create_task(
+ loop, self._process_push_queue(), f"{self}::_process_push_queue"
+ )
+
+ return self._process_push_task
+
+ def _maybe_start_heartbeat_tasks(self):
+ if self._params.enable_heartbeats:
+ loop = asyncio.get_running_loop()
+ self._heartbeat_push_task = create_task(
+ loop, self._heartbeat_push_handler(), f"{self}::_heartbeat_push_handler"
+ )
+ self._heartbeat_monitor_task = create_task(
+ loop, self._heartbeat_monitor_handler(), f"{self}::_heartbeat_monitor_handler"
+ )
+
+ async def _cancel_tasks(self):
+ await self._maybe_cancel_heartbeat_tasks()
+
+ await cancel_task(self._process_up_task)
+ await cancel_task(self._process_down_task)
+
+ await self._observer.stop()
+
+ async def _maybe_cancel_heartbeat_tasks(self):
+ if self._params.enable_heartbeats:
+ await cancel_task(self._heartbeat_push_task)
+ await cancel_task(self._heartbeat_monitor_task)
+
def _initial_metrics_frame(self) -> MetricsFrame:
processors = self._pipeline.processors_with_metrics()
data = []
@@ -148,14 +240,31 @@ class PipelineTask:
data.append(ProcessingMetricsData(processor=p.name, value=0.0))
return MetricsFrame(data=data)
+ async def _wait_for_endframe(self):
+ await self._endframe_event.wait()
+ self._endframe_event.clear()
+
+ async def _cleanup(self):
+ await self._source.cleanup()
+ await self._pipeline.cleanup()
+ await self._sink.cleanup()
+
async def _process_push_queue(self):
+ """This is the task that runs the pipeline for the first time by sending
+ a StartFrame and by pushing any other frames queued by the user. It runs
+ until the tasks is canceled or stopped (e.g. with an EndFrame).
+
+ """
self._clock.start()
+ self._maybe_start_heartbeat_tasks()
+
start_frame = StartFrame(
allow_interruptions=self._params.allow_interruptions,
enable_metrics=self._params.enable_metrics,
enable_usage_metrics=self._params.enable_usage_metrics,
report_only_initial_ttfb=self._params.report_only_initial_ttfb,
+ observer=self._observer,
clock=self._clock,
)
await self._source.queue_frame(start_frame, FrameDirection.DOWNSTREAM)
@@ -166,44 +275,87 @@ class PipelineTask:
running = True
should_cleanup = True
while running:
- try:
- frame = await self._push_queue.get()
- await self._source.queue_frame(frame, FrameDirection.DOWNSTREAM)
- if isinstance(frame, EndFrame):
- await self._wait_for_endframe()
- running = not isinstance(frame, (StopTaskFrame, EndFrame))
- should_cleanup = not isinstance(frame, StopTaskFrame)
- self._push_queue.task_done()
- except asyncio.CancelledError:
- break
+ frame = await self._push_queue.get()
+ await self._source.queue_frame(frame, FrameDirection.DOWNSTREAM)
+ if isinstance(frame, EndFrame):
+ await self._wait_for_endframe()
+ running = not isinstance(frame, (CancelFrame, EndFrame, StopTaskFrame))
+ should_cleanup = not isinstance(frame, StopTaskFrame)
+ self._push_queue.task_done()
# Cleanup only if we need to.
if should_cleanup:
- await self._source.cleanup()
- await self._pipeline.cleanup()
- await self._sink.cleanup()
- # We just enqueue None to terminate the task gracefully.
- self._process_up_task.cancel()
- await self._process_up_task
-
- async def _wait_for_endframe(self):
- # NOTE(aleix): the Sink element just pushes EndFrames to the down queue,
- # so just wait for it. In the future we might do something else here,
- # but for now this is fine.
- await self._down_queue.get()
+ await self._cleanup()
async def _process_up_queue(self):
+ """This is the task that processes frames coming upstream from the
+ pipeline. These frames might indicate, for example, that we want the
+ pipeline to be stopped (e.g. EndTaskFrame) in which case we would send
+ an EndFrame down the pipeline.
+
+ """
+ while True:
+ frame = await self._up_queue.get()
+ if isinstance(frame, EndTaskFrame):
+ # Tell the task we should end nicely.
+ await self.queue_frame(EndFrame())
+ elif isinstance(frame, CancelTaskFrame):
+ # Tell the task we should end right away.
+ await self.queue_frame(CancelFrame())
+ elif isinstance(frame, StopTaskFrame):
+ await self.queue_frame(StopTaskFrame())
+ elif isinstance(frame, ErrorFrame):
+ logger.error(f"Error running app: {frame}")
+ if frame.fatal:
+ # Cancel all tasks downstream.
+ await self.queue_frame(CancelFrame())
+ # Tell the task we should stop.
+ await self.queue_frame(StopTaskFrame())
+ self._up_queue.task_done()
+
+ async def _process_down_queue(self):
+ """This tasks process frames coming downstream from the pipeline. For
+ example, heartbeat frames or an EndFrame which would indicate all
+ processors have handled the EndFrame and therefore we can exit the task
+ cleanly.
+
+ """
+ while True:
+ frame = await self._down_queue.get()
+ if isinstance(frame, EndFrame):
+ self._endframe_event.set()
+ elif isinstance(frame, HeartbeatFrame):
+ await self._heartbeat_queue.put(frame)
+ self._down_queue.task_done()
+
+ async def _heartbeat_push_handler(self):
+ """
+ This tasks pushes a heartbeat frame every heartbeat period.
+ """
+ while True:
+ # Don't use `queue_frame()` because if an EndFrame is queued the
+ # task will just stop waiting for the pipeline to finish not
+ # allowing more frames to be pushed.
+ await self._source.queue_frame(HeartbeatFrame(timestamp=self._clock.get_time()))
+ await asyncio.sleep(self._params.heartbeats_period_secs)
+
+ async def _heartbeat_monitor_handler(self):
+ """This tasks monitors heartbeat frames. If a heartbeat frame has not
+ been received for a long period a warning will be logged. It also logs
+ the time that a heartbeat frame takes to processes, that is how long it
+ takes for the heartbeat frame to traverse all the pipeline.
+
+ """
+ wait_time = HEARTBEAT_MONITOR_SECONDS
while True:
try:
- frame = await self._up_queue.get()
- if isinstance(frame, EndTaskFrame):
- await self.queue_frame(EndFrame())
- elif isinstance(frame, CancelTaskFrame):
- await self.queue_frame(CancelFrame())
- elif isinstance(frame, StopTaskFrame):
- await self.queue_frame(StopTaskFrame())
- self._up_queue.task_done()
- except asyncio.CancelledError:
- break
+ frame = await asyncio.wait_for(self._heartbeat_queue.get(), timeout=wait_time)
+ process_time = (self._clock.get_time() - frame.timestamp) / 1_000_000_000
+ logger.trace(f"{self}: heartbeat frame processed in {process_time} seconds")
+ self._heartbeat_queue.task_done()
+ except asyncio.TimeoutError:
+ logger.warning(
+ f"{self}: heartbeat frame not received for more than {wait_time} seconds"
+ )
def __str__(self):
return self.name
diff --git a/src/pipecat/pipeline/task_observer.py b/src/pipecat/pipeline/task_observer.py
new file mode 100644
index 000000000..c9816f7d3
--- /dev/null
+++ b/src/pipecat/pipeline/task_observer.py
@@ -0,0 +1,113 @@
+#
+# Copyright (c) 2024–2025, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import asyncio
+from typing import List
+
+from attr import dataclass
+
+from pipecat.frames.frames import Frame
+from pipecat.observers.base_observer import BaseObserver
+from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
+from pipecat.utils.asyncio import cancel_task, create_task
+from pipecat.utils.utils import obj_count, obj_id
+
+
+@dataclass
+class Proxy:
+ """This is the data we receive from the main observer and that we put into
+ a queue for later processing.
+
+ """
+
+ queue: asyncio.Queue
+ task: asyncio.Task
+ observer: BaseObserver
+
+
+@dataclass
+class ObserverData:
+ """This is the data we receive from the main observer and that we put into a
+ proxy queue for later processing.
+
+ """
+
+ src: FrameProcessor
+ dst: FrameProcessor
+ frame: Frame
+ direction: FrameDirection
+ timestamp: int
+
+
+class TaskObserver(BaseObserver):
+ """This is a pipeline frame observer that is meant to be used as a proxy to
+ the user provided observers. That is, this is the observer that should be
+ passed to the frame processors. Then, every time a frame is pushed this
+ observer will call all the observers registered to the pipeline task.
+
+ This observer makes sure that passing frames to observers doesn't block the
+ pipeline by creating a queue and a task for each user observer. When a frame
+ is received, it will be put in a queue for efficiency and later processed by
+ each task.
+
+ """
+
+ def __init__(self, observers: List[BaseObserver] = []):
+ self._id: int = obj_id()
+ self._name: str = f"{self.__class__.__name__}#{obj_count(self)}"
+ self._proxies: List[Proxy] = self._create_proxies(observers)
+
+ @property
+ def id(self) -> int:
+ return self._id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ async def stop(self):
+ """Stops all proxy observer tasks."""
+ for proxy in self._proxies:
+ await cancel_task(proxy.task)
+
+ async def on_push_frame(
+ self,
+ src: FrameProcessor,
+ dst: FrameProcessor,
+ frame: Frame,
+ direction: FrameDirection,
+ timestamp: int,
+ ):
+ for proxy in self._proxies:
+ await proxy.queue.put(
+ ObserverData(
+ src=src, dst=dst, frame=frame, direction=direction, timestamp=timestamp
+ )
+ )
+
+ def _create_proxies(self, observers) -> List[Proxy]:
+ proxies = []
+ loop = asyncio.get_running_loop()
+ for observer in observers:
+ queue = asyncio.Queue()
+ task = create_task(
+ loop,
+ self._proxy_task_handler(queue, observer),
+ f"{self}::{observer.__class__.__name__}",
+ )
+ proxy = Proxy(queue=queue, task=task, observer=observer)
+ proxies.append(proxy)
+ return proxies
+
+ async def _proxy_task_handler(self, queue: asyncio.Queue, observer: BaseObserver):
+ while True:
+ data = await queue.get()
+ await observer.on_push_frame(
+ data.src, data.dst, data.frame, data.direction, data.timestamp
+ )
+
+ def __str__(self):
+ return self.name
diff --git a/src/pipecat/processors/aggregators/gated_openai_llm_context.py b/src/pipecat/processors/aggregators/gated_openai_llm_context.py
index a185528f2..8c3f496eb 100644
--- a/src/pipecat/processors/aggregators/gated_openai_llm_context.py
+++ b/src/pipecat/processors/aggregators/gated_openai_llm_context.py
@@ -4,8 +4,6 @@
# SPDX-License-Identifier: BSD 2-Clause License
#
-import asyncio
-
from pipecat.frames.frames import CancelFrame, EndFrame, Frame, StartFrame
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContextFrame
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
@@ -38,18 +36,14 @@ class GatedOpenAILLMContextAggregator(FrameProcessor):
await self.push_frame(frame, direction)
async def _start(self):
- self._gate_task = self.get_event_loop().create_task(self._gate_task_handler())
+ self._gate_task = self.create_task(self._gate_task_handler())
async def _stop(self):
- self._gate_task.cancel()
- await self._gate_task
+ await self.cancel_task(self._gate_task)
async def _gate_task_handler(self):
while True:
- try:
- await self._notifier.wait()
- if self._last_context_frame:
- await self.push_frame(self._last_context_frame)
- self._last_context_frame = None
- except asyncio.CancelledError:
- break
+ await self._notifier.wait()
+ if self._last_context_frame:
+ await self.push_frame(self._last_context_frame)
+ self._last_context_frame = None
diff --git a/src/pipecat/processors/audio/audio_buffer_processor.py b/src/pipecat/processors/audio/audio_buffer_processor.py
index 34825ab20..fed4db3c8 100644
--- a/src/pipecat/processors/audio/audio_buffer_processor.py
+++ b/src/pipecat/processors/audio/audio_buffer_processor.py
@@ -6,6 +6,7 @@
from pipecat.audio.utils import interleave_stereo_audio, mix_audio, resample_audio
from pipecat.frames.frames import (
+ EndFrame,
Frame,
InputAudioRawFrame,
OutputAudioRawFrame,
@@ -86,6 +87,9 @@ class AudioBufferProcessor(FrameProcessor):
if self._buffer_size > 0 and len(self._user_audio_buffer) > self._buffer_size:
await self._call_on_audio_data_handler()
+ if isinstance(frame, EndFrame):
+ await self._call_on_audio_data_handler()
+
await self.push_frame(frame, direction)
async def _call_on_audio_data_handler(self):
diff --git a/src/pipecat/processors/filters/frame_filter.py b/src/pipecat/processors/filters/frame_filter.py
index a2fb90505..43e2dab95 100644
--- a/src/pipecat/processors/filters/frame_filter.py
+++ b/src/pipecat/processors/filters/frame_filter.py
@@ -11,7 +11,7 @@ from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
class FrameFilter(FrameProcessor):
- def __init__(self, types: Tuple[Type[Frame]]):
+ def __init__(self, types: Tuple[Type[Frame], ...]):
super().__init__()
self._types = types
diff --git a/src/pipecat/processors/frame_processor.py b/src/pipecat/processors/frame_processor.py
index 653b313b8..3b228e639 100644
--- a/src/pipecat/processors/frame_processor.py
+++ b/src/pipecat/processors/frame_processor.py
@@ -6,15 +6,15 @@
import asyncio
import inspect
+import sys
from enum import Enum
-from typing import Awaitable, Callable, Optional
+from typing import Awaitable, Callable, Coroutine, Optional
from loguru import logger
from pipecat.clocks.base_clock import BaseClock
from pipecat.frames.frames import (
CancelFrame,
- EndFrame,
ErrorFrame,
Frame,
StartFrame,
@@ -24,6 +24,7 @@ from pipecat.frames.frames import (
)
from pipecat.metrics.metrics import LLMTokenUsage, MetricsData
from pipecat.processors.metrics.frame_processor_metrics import FrameProcessorMetrics
+from pipecat.utils.asyncio import cancel_task, create_task, wait_for_task
from pipecat.utils.utils import obj_count, obj_id
@@ -41,8 +42,8 @@ class FrameProcessor:
loop: asyncio.AbstractEventLoop | None = None,
**kwargs,
):
- self.id: int = obj_id()
- self.name = name or f"{self.__class__.__name__}#{obj_count(self)}"
+ self._id: int = obj_id()
+ self._name = name or f"{self.__class__.__name__}#{obj_count(self)}"
self._parent: "FrameProcessor" | None = None
self._prev: "FrameProcessor" | None = None
self._next: "FrameProcessor" | None = None
@@ -58,6 +59,7 @@ class FrameProcessor:
self._enable_metrics = False
self._enable_usage_metrics = False
self._report_only_initial_ttfb = False
+ self._observer = None
# Cancellation is done through CancelFrame (a system frame). This could
# cause other events being triggered (e.g. closing a transport) which
@@ -82,6 +84,14 @@ class FrameProcessor:
# the exception to this rule. This create this task.
self.__create_push_task()
+ @property
+ def id(self) -> int:
+ return self._id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
@property
def interruptions_allowed(self):
return self._allow_interruptions
@@ -140,6 +150,16 @@ class FrameProcessor:
await self.stop_ttfb_metrics()
await self.stop_processing_metrics()
+ def create_task(self, coroutine: Coroutine) -> asyncio.Task:
+ name = f"{self}::{coroutine.cr_code.co_name}"
+ return create_task(self.get_event_loop(), coroutine, name)
+
+ async def cancel_task(self, task: asyncio.Task, timeout: Optional[float] = None):
+ await cancel_task(task, timeout)
+
+ async def wait_for_task(self, task: asyncio.Task, timeout: Optional[float] = None):
+ await wait_for_task(task, timeout)
+
async def cleanup(self):
await self.__cancel_input_task()
await self.__cancel_push_task()
@@ -181,11 +201,12 @@ class FrameProcessor:
await self.__input_queue.put((frame, direction, callback))
async def pause_processing_frames(self):
+ logger.trace(f"{self}: pausing frame processing")
self.__should_block_frames = True
async def resume_processing_frames(self):
+ logger.trace(f"{self}: resuming frame processing")
self.__input_event.set()
- self.__should_block_frames = False
async def process_frame(self, frame: Frame, direction: FrameDirection):
if isinstance(frame, StartFrame):
@@ -194,6 +215,7 @@ class FrameProcessor:
self._enable_metrics = frame.enable_metrics
self._enable_usage_metrics = frame.enable_usage_metrics
self._report_only_initial_ttfb = frame.report_only_initial_ttfb
+ self._observer = frame.observer
elif isinstance(frame, StartInterruptionFrame):
await self._start_interruption()
await self.stop_all_metrics()
@@ -256,11 +278,20 @@ class FrameProcessor:
async def __internal_push_frame(self, frame: Frame, direction: FrameDirection):
try:
+ timestamp = self._clock.get_time() if self._clock else 0
if direction == FrameDirection.DOWNSTREAM and self._next:
logger.trace(f"Pushing {frame} from {self} to {self._next}")
+ if self._observer:
+ await self._observer.on_push_frame(
+ self, self._next, frame, direction, timestamp
+ )
await self._next.queue_frame(frame, direction)
elif direction == FrameDirection.UPSTREAM and self._prev:
logger.trace(f"Pushing {frame} upstream from {self} to {self._prev}")
+ if self._observer:
+ await self._observer.on_push_frame(
+ self, self._prev, frame, direction, timestamp
+ )
await self._prev.queue_frame(frame, direction)
except Exception as e:
logger.exception(f"Uncaught exception in {self}: {e}")
@@ -270,64 +301,44 @@ class FrameProcessor:
def __create_input_task(self):
self.__should_block_frames = False
self.__input_queue = asyncio.Queue()
- self.__input_frame_task = self.get_event_loop().create_task(
- self.__input_frame_task_handler()
- )
self.__input_event = asyncio.Event()
+ self.__input_frame_task = self.create_task(self.__input_frame_task_handler())
async def __cancel_input_task(self):
- self.__input_frame_task.cancel()
- await self.__input_frame_task
+ await self.cancel_task(self.__input_frame_task)
async def __input_frame_task_handler(self):
- running = True
- while running:
- try:
- if self.__should_block_frames:
- await self.__input_event.wait()
- self.__input_event.clear()
+ while True:
+ if self.__should_block_frames:
+ logger.trace(f"{self}: frame processing paused")
+ await self.__input_event.wait()
+ self.__input_event.clear()
+ self.__should_block_frames = False
+ logger.trace(f"{self}: frame processing resumed")
- (frame, direction, callback) = await self.__input_queue.get()
+ (frame, direction, callback) = await self.__input_queue.get()
- # Process the frame.
- await self.process_frame(frame, direction)
+ # Process the frame.
+ await self.process_frame(frame, direction)
- # If this frame has an associated callback, call it now.
- if callback:
- await callback(self, frame, direction)
+ # If this frame has an associated callback, call it now.
+ if callback:
+ await callback(self, frame, direction)
- running = not isinstance(frame, EndFrame)
-
- self.__input_queue.task_done()
- except asyncio.CancelledError:
- logger.trace(f"Cancelled input task in {self}")
- break
- except Exception as e:
- logger.exception(f"Uncaught exception in {self}: {e}")
- await self.push_error(ErrorFrame(str(e)))
+ self.__input_queue.task_done()
def __create_push_task(self):
self.__push_queue = asyncio.Queue()
- self.__push_frame_task = self.get_event_loop().create_task(self.__push_frame_task_handler())
+ self.__push_frame_task = self.create_task(self.__push_frame_task_handler())
async def __cancel_push_task(self):
- self.__push_frame_task.cancel()
- await self.__push_frame_task
+ await self.cancel_task(self.__push_frame_task)
async def __push_frame_task_handler(self):
- running = True
- while running:
- try:
- (frame, direction) = await self.__push_queue.get()
- await self.__internal_push_frame(frame, direction)
- running = not isinstance(frame, EndFrame)
- self.__push_queue.task_done()
- except asyncio.CancelledError:
- logger.trace(f"Cancelled push task in {self}")
- break
- except Exception as e:
- logger.exception(f"Uncaught exception in {self}: {e}")
- await self.push_error(ErrorFrame(str(e)))
+ while True:
+ (frame, direction) = await self.__push_queue.get()
+ await self.__internal_push_frame(frame, direction)
+ self.__push_queue.task_done()
async def _call_event_handler(self, event_name: str, *args, **kwargs):
try:
diff --git a/src/pipecat/processors/frameworks/rtvi.py b/src/pipecat/processors/frameworks/rtvi.py
index 96d51cb5d..f59b6d48f 100644
--- a/src/pipecat/processors/frameworks/rtvi.py
+++ b/src/pipecat/processors/frameworks/rtvi.py
@@ -34,14 +34,15 @@ from pipecat.frames.frames import (
InterimTranscriptionFrame,
LLMFullResponseEndFrame,
LLMFullResponseStartFrame,
+ LLMTextFrame,
MetricsFrame,
StartFrame,
SystemFrame,
- TextFrame,
TranscriptionFrame,
TransportMessageUrgentFrame,
TTSStartedFrame,
TTSStoppedFrame,
+ TTSTextFrame,
UserStartedSpeakingFrame,
UserStoppedSpeakingFrame,
)
@@ -51,15 +52,20 @@ from pipecat.metrics.metrics import (
TTFBMetricsData,
TTSUsageMetricsData,
)
+from pipecat.observers.base_observer import BaseObserver
from pipecat.processors.aggregators.openai_llm_context import (
OpenAILLMContext,
OpenAILLMContextFrame,
)
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
+from pipecat.services.google.frames import LLMSearchOrigin, LLMSearchResponseFrame
from pipecat.utils.string import match_endofsentence
RTVI_PROTOCOL_VERSION = "0.3.0"
+RTVI_MESSAGE_LABEL = "rtvi-ai"
+RTVIMessageLiteral = Literal["rtvi-ai"]
+
ActionResult = Union[bool, int, float, str, list, dict]
@@ -152,7 +158,7 @@ class RTVIActionFrame(DataFrame):
class RTVIMessage(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: str
id: str
data: Optional[Dict[str, Any]] = None
@@ -168,7 +174,7 @@ class RTVIErrorResponseData(BaseModel):
class RTVIErrorResponse(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["error-response"] = "error-response"
id: str
data: RTVIErrorResponseData
@@ -180,7 +186,7 @@ class RTVIErrorData(BaseModel):
class RTVIError(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["error"] = "error"
data: RTVIErrorData
@@ -190,7 +196,7 @@ class RTVIDescribeConfigData(BaseModel):
class RTVIDescribeConfig(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["config-available"] = "config-available"
id: str
data: RTVIDescribeConfigData
@@ -201,14 +207,14 @@ class RTVIDescribeActionsData(BaseModel):
class RTVIDescribeActions(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["actions-available"] = "actions-available"
id: str
data: RTVIDescribeActionsData
class RTVIConfigResponse(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["config"] = "config"
id: str
data: RTVIConfig
@@ -219,7 +225,7 @@ class RTVIActionResponseData(BaseModel):
class RTVIActionResponse(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["action-response"] = "action-response"
id: str
data: RTVIActionResponseData
@@ -231,7 +237,7 @@ class RTVIBotReadyData(BaseModel):
class RTVIBotReady(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["bot-ready"] = "bot-ready"
id: str
data: RTVIBotReadyData
@@ -244,7 +250,7 @@ class RTVILLMFunctionCallMessageData(BaseModel):
class RTVILLMFunctionCallMessage(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["llm-function-call"] = "llm-function-call"
data: RTVILLMFunctionCallMessageData
@@ -254,7 +260,7 @@ class RTVILLMFunctionCallStartMessageData(BaseModel):
class RTVILLMFunctionCallStartMessage(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["llm-function-call-start"] = "llm-function-call-start"
data: RTVILLMFunctionCallStartMessageData
@@ -267,22 +273,22 @@ class RTVILLMFunctionCallResultData(BaseModel):
class RTVIBotLLMStartedMessage(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["bot-llm-started"] = "bot-llm-started"
class RTVIBotLLMStoppedMessage(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["bot-llm-stopped"] = "bot-llm-stopped"
class RTVIBotTTSStartedMessage(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["bot-tts-started"] = "bot-tts-started"
class RTVIBotTTSStoppedMessage(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["bot-tts-stopped"] = "bot-tts-stopped"
@@ -290,20 +296,32 @@ class RTVITextMessageData(BaseModel):
text: str
+class RTVISearchResponseMessageData(BaseModel):
+ search_result: Optional[str]
+ rendered_content: Optional[str]
+ origins: List[LLMSearchOrigin]
+
+
class RTVIBotTranscriptionMessage(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["bot-transcription"] = "bot-transcription"
data: RTVITextMessageData
class RTVIBotLLMTextMessage(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["bot-llm-text"] = "bot-llm-text"
data: RTVITextMessageData
-class RTVIBotTTSTextMessage(BaseModel):
+class RTVIBotLLMSearchResponseMessage(BaseModel):
label: Literal["rtvi-ai"] = "rtvi-ai"
+ type: Literal["bot-llm-search-response"] = "bot-llm-search-response"
+ data: RTVISearchResponseMessageData
+
+
+class RTVIBotTTSTextMessage(BaseModel):
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["bot-tts-text"] = "bot-tts-text"
data: RTVITextMessageData
@@ -315,7 +333,7 @@ class RTVIAudioMessageData(BaseModel):
class RTVIBotTTSAudioMessage(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["bot-tts-audio"] = "bot-tts-audio"
data: RTVIAudioMessageData
@@ -328,39 +346,39 @@ class RTVIUserTranscriptionMessageData(BaseModel):
class RTVIUserTranscriptionMessage(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["user-transcription"] = "user-transcription"
data: RTVIUserTranscriptionMessageData
class RTVIUserLLMTextMessage(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["user-llm-text"] = "user-llm-text"
data: RTVITextMessageData
class RTVIUserStartedSpeakingMessage(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["user-started-speaking"] = "user-started-speaking"
class RTVIUserStoppedSpeakingMessage(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["user-stopped-speaking"] = "user-stopped-speaking"
class RTVIBotStartedSpeakingMessage(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["bot-started-speaking"] = "bot-started-speaking"
class RTVIBotStoppedSpeakingMessage(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["bot-stopped-speaking"] = "bot-stopped-speaking"
class RTVIMetricsMessage(BaseModel):
- label: Literal["rtvi-ai"] = "rtvi-ai"
+ label: RTVIMessageLiteral = RTVI_MESSAGE_LABEL
type: Literal["metrics"] = "metrics"
data: Mapping[str, Any]
@@ -479,7 +497,7 @@ class RTVIBotTranscriptionProcessor(RTVIFrameProcessor):
if isinstance(frame, UserStartedSpeakingFrame):
await self._push_aggregation()
- elif isinstance(frame, TextFrame):
+ elif isinstance(frame, LLMTextFrame):
self._aggregation += frame.text
if match_endofsentence(self._aggregation):
await self._push_aggregation()
@@ -504,7 +522,7 @@ class RTVIBotLLMProcessor(RTVIFrameProcessor):
await self._push_transport_message_urgent(RTVIBotLLMStartedMessage())
elif isinstance(frame, LLMFullResponseEndFrame):
await self._push_transport_message_urgent(RTVIBotLLMStoppedMessage())
- elif type(frame) is TextFrame:
+ elif isinstance(frame, LLMTextFrame):
message = RTVIBotLLMTextMessage(data=RTVITextMessageData(text=frame.text))
await self._push_transport_message_urgent(message)
@@ -522,7 +540,7 @@ class RTVIBotTTSProcessor(RTVIFrameProcessor):
await self._push_transport_message_urgent(RTVIBotTTSStartedMessage())
elif isinstance(frame, TTSStoppedFrame):
await self._push_transport_message_urgent(RTVIBotTTSStoppedMessage())
- elif type(frame) is TextFrame:
+ elif isinstance(frame, TTSTextFrame):
message = RTVIBotTTSTextMessage(data=RTVITextMessageData(text=frame.text))
await self._push_transport_message_urgent(message)
@@ -563,6 +581,168 @@ class RTVIMetricsProcessor(RTVIFrameProcessor):
await self._push_transport_message_urgent(message)
+class RTVIObserver(BaseObserver):
+ """This is a pipeline frame observer that is used to send RTVI server
+ messages to clients. The observer does not handle incoming RTVI client
+ messages, which is done by the RTVIProcessor.
+
+ """
+
+ def __init__(self, rtvi: FrameProcessor):
+ super().__init__()
+ self._rtvi = rtvi
+ self._bot_transcription = ""
+ self._frames_seen = set()
+
+ async def on_push_frame(
+ self,
+ src: FrameProcessor,
+ dst: FrameProcessor,
+ frame: Frame,
+ direction: FrameDirection,
+ timestamp: int,
+ ):
+ # If we have already seen this frame, let's skip it.
+ if frame.id in self._frames_seen:
+ return
+ self._frames_seen.add(frame.id)
+
+ if isinstance(frame, (UserStartedSpeakingFrame, UserStoppedSpeakingFrame)):
+ await self._handle_interruptions(frame)
+ elif isinstance(frame, (BotStartedSpeakingFrame, BotStoppedSpeakingFrame)):
+ await self._handle_bot_speaking(frame)
+ elif isinstance(frame, (TranscriptionFrame, InterimTranscriptionFrame)):
+ await self._handle_user_transcriptions(frame)
+ elif isinstance(frame, OpenAILLMContextFrame):
+ await self._handle_context(frame)
+ elif isinstance(frame, UserStartedSpeakingFrame):
+ await self._push_bot_transcription()
+ elif isinstance(frame, LLMFullResponseStartFrame):
+ await self._push_transport_message_urgent(RTVIBotLLMStartedMessage())
+ elif isinstance(frame, LLMFullResponseEndFrame):
+ await self._push_transport_message_urgent(RTVIBotLLMStoppedMessage())
+ elif isinstance(frame, LLMTextFrame):
+ await self._handle_llm_text_frame(frame)
+ elif isinstance(frame, LLMSearchResponseFrame):
+ await self._handle_llm_search_response_frame(frame)
+ elif isinstance(frame, TTSStartedFrame):
+ await self._push_transport_message_urgent(RTVIBotTTSStartedMessage())
+ elif isinstance(frame, TTSStoppedFrame):
+ await self._push_transport_message_urgent(RTVIBotTTSStoppedMessage())
+ elif isinstance(frame, TTSTextFrame):
+ message = RTVIBotTTSTextMessage(data=RTVITextMessageData(text=frame.text))
+ await self._push_transport_message_urgent(message)
+ elif isinstance(frame, MetricsFrame):
+ await self._handle_metrics(frame)
+
+ async def _push_transport_message_urgent(self, model: BaseModel, exclude_none: bool = True):
+ frame = TransportMessageUrgentFrame(message=model.model_dump(exclude_none=exclude_none))
+ await self._rtvi.push_frame(frame)
+
+ async def _push_bot_transcription(self):
+ if len(self._bot_transcription) > 0:
+ message = RTVIBotTranscriptionMessage(
+ data=RTVITextMessageData(text=self._bot_transcription)
+ )
+ await self._push_transport_message_urgent(message)
+ self._bot_transcription = ""
+
+ async def _handle_interruptions(self, frame: Frame):
+ message = None
+ if isinstance(frame, UserStartedSpeakingFrame):
+ message = RTVIUserStartedSpeakingMessage()
+ elif isinstance(frame, UserStoppedSpeakingFrame):
+ message = RTVIUserStoppedSpeakingMessage()
+
+ if message:
+ await self._push_transport_message_urgent(message)
+
+ async def _handle_bot_speaking(self, frame: Frame):
+ message = None
+ if isinstance(frame, BotStartedSpeakingFrame):
+ message = RTVIBotStartedSpeakingMessage()
+ elif isinstance(frame, BotStoppedSpeakingFrame):
+ message = RTVIBotStoppedSpeakingMessage()
+
+ if message:
+ await self._push_transport_message_urgent(message)
+
+ async def _handle_llm_text_frame(self, frame: LLMTextFrame):
+ message = RTVIBotLLMTextMessage(data=RTVITextMessageData(text=frame.text))
+ await self._push_transport_message_urgent(message)
+
+ self._bot_transcription += frame.text
+ if match_endofsentence(self._bot_transcription):
+ await self._push_bot_transcription()
+
+ async def _handle_llm_search_response_frame(self, frame: LLMSearchResponseFrame):
+ message = RTVIBotLLMSearchResponseMessage(
+ data=RTVISearchResponseMessageData(
+ search_result=frame.search_result,
+ origins=frame.origins,
+ rendered_content=frame.rendered_content,
+ )
+ )
+ await self._push_transport_message_urgent(message)
+
+ async def _handle_user_transcriptions(self, frame: Frame):
+ message = None
+ if isinstance(frame, TranscriptionFrame):
+ message = RTVIUserTranscriptionMessage(
+ data=RTVIUserTranscriptionMessageData(
+ text=frame.text, user_id=frame.user_id, timestamp=frame.timestamp, final=True
+ )
+ )
+ elif isinstance(frame, InterimTranscriptionFrame):
+ message = RTVIUserTranscriptionMessage(
+ data=RTVIUserTranscriptionMessageData(
+ text=frame.text, user_id=frame.user_id, timestamp=frame.timestamp, final=False
+ )
+ )
+
+ if message:
+ await self._push_transport_message_urgent(message)
+
+ async def _handle_context(self, frame: OpenAILLMContextFrame):
+ try:
+ messages = frame.context.messages
+ if len(messages) > 0:
+ message = messages[-1]
+ if message["role"] == "user":
+ content = message["content"]
+ if isinstance(content, list):
+ text = " ".join(item["text"] for item in content if "text" in item)
+ else:
+ text = content
+ rtvi_message = RTVIUserLLMTextMessage(data=RTVITextMessageData(text=text))
+ await self._push_transport_message_urgent(rtvi_message)
+ except TypeError as e:
+ logger.warning(f"Caught an error while trying to handle context: {e}")
+
+ async def _handle_metrics(self, frame: MetricsFrame):
+ metrics = {}
+ for d in frame.data:
+ if isinstance(d, TTFBMetricsData):
+ if "ttfb" not in metrics:
+ metrics["ttfb"] = []
+ metrics["ttfb"].append(d.model_dump(exclude_none=True))
+ elif isinstance(d, ProcessingMetricsData):
+ if "processing" not in metrics:
+ metrics["processing"] = []
+ metrics["processing"].append(d.model_dump(exclude_none=True))
+ elif isinstance(d, LLMUsageMetricsData):
+ if "tokens" not in metrics:
+ metrics["tokens"] = []
+ metrics["tokens"].append(d.value.model_dump(exclude_none=True))
+ elif isinstance(d, TTSUsageMetricsData):
+ if "characters" not in metrics:
+ metrics["characters"] = []
+ metrics["characters"].append(d.model_dump(exclude_none=True))
+
+ message = RTVIMetricsMessage(data=metrics)
+ await self._push_transport_message_urgent(message)
+
+
class RTVIProcessor(FrameProcessor):
def __init__(
self,
@@ -584,15 +764,18 @@ class RTVIProcessor(FrameProcessor):
# A task to process incoming action frames.
self._action_queue = asyncio.Queue()
- self._action_task = self.get_event_loop().create_task(self._action_task_handler())
+ self._action_task = self.create_task(self._action_task_handler())
# A task to process incoming transport messages.
self._message_queue = asyncio.Queue()
- self._message_task = self.get_event_loop().create_task(self._message_task_handler())
+ self._message_task = self.create_task(self._message_task_handler())
self._register_event_handler("on_bot_started")
self._register_event_handler("on_client_ready")
+ def observer(self) -> RTVIObserver:
+ return RTVIObserver(self)
+
def register_action(self, action: RTVIAction):
id = self._action_id(action.service, action.action)
self._registered_actions[id] = action
@@ -690,13 +873,11 @@ class RTVIProcessor(FrameProcessor):
async def _cancel_tasks(self):
if self._action_task:
- self._action_task.cancel()
- await self._action_task
+ await self.cancel_task(self._action_task)
self._action_task = None
if self._message_task:
- self._message_task.cancel()
- await self._message_task
+ await self.cancel_task(self._message_task)
self._message_task = None
async def _push_transport_message(self, model: BaseModel, exclude_none: bool = True):
@@ -705,25 +886,23 @@ class RTVIProcessor(FrameProcessor):
async def _action_task_handler(self):
while True:
- try:
- frame = await self._action_queue.get()
- await self._handle_action(frame.message_id, frame.rtvi_action_run)
- self._action_queue.task_done()
- except asyncio.CancelledError:
- break
+ frame = await self._action_queue.get()
+ await self._handle_action(frame.message_id, frame.rtvi_action_run)
+ self._action_queue.task_done()
async def _message_task_handler(self):
while True:
- try:
- message = await self._message_queue.get()
- await self._handle_message(message)
- self._message_queue.task_done()
- except asyncio.CancelledError:
- break
+ message = await self._message_queue.get()
+ await self._handle_message(message)
+ self._message_queue.task_done()
async def _handle_transport_message(self, frame: TransportMessageUrgentFrame):
try:
- message = RTVIMessage.model_validate(frame.message)
+ transport_message = frame.message
+ if transport_message.get("label") != RTVI_MESSAGE_LABEL:
+ logger.warning(f"Ignoring not RTVI message: {transport_message}")
+ return
+ message = RTVIMessage.model_validate(transport_message)
await self._message_queue.put(message)
except ValidationError as e:
await self.send_error(f"Invalid RTVI transport message: {e}")
diff --git a/src/pipecat/processors/idle_frame_processor.py b/src/pipecat/processors/idle_frame_processor.py
index 3ed52c354..85ea215cf 100644
--- a/src/pipecat/processors/idle_frame_processor.py
+++ b/src/pipecat/processors/idle_frame_processor.py
@@ -7,7 +7,7 @@
import asyncio
from typing import Awaitable, Callable, List
-from pipecat.frames.frames import Frame
+from pipecat.frames.frames import Frame, StartFrame
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
@@ -31,11 +31,12 @@ class IdleFrameProcessor(FrameProcessor):
self._timeout = timeout
self._types = types
- self._create_idle_task()
-
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
+ if isinstance(frame, StartFrame):
+ self._create_idle_task()
+
await self.push_frame(frame, direction)
# If we are not waiting for any specific frame set the event, otherwise
@@ -48,12 +49,11 @@ class IdleFrameProcessor(FrameProcessor):
self._idle_event.set()
async def cleanup(self):
- self._idle_task.cancel()
- await self._idle_task
+ await self.cancel_task(self._idle_task)
def _create_idle_task(self):
self._idle_event = asyncio.Event()
- self._idle_task = self.get_event_loop().create_task(self._idle_task_handler())
+ self._idle_task = self.create_task(self._idle_task_handler())
async def _idle_task_handler(self):
while True:
@@ -61,7 +61,5 @@ class IdleFrameProcessor(FrameProcessor):
await asyncio.wait_for(self._idle_event.wait(), timeout=self._timeout)
except asyncio.TimeoutError:
await self._callback(self)
- except asyncio.CancelledError:
- break
finally:
self._idle_event.clear()
diff --git a/src/pipecat/processors/transcript_processor.py b/src/pipecat/processors/transcript_processor.py
index 5379e22e6..d31310f84 100644
--- a/src/pipecat/processors/transcript_processor.py
+++ b/src/pipecat/processors/transcript_processor.py
@@ -4,17 +4,23 @@
# SPDX-License-Identifier: BSD 2-Clause License
#
-from typing import List
+from typing import List, Optional
+
+from loguru import logger
from pipecat.frames.frames import (
+ BotStoppedSpeakingFrame,
+ CancelFrame,
+ EndFrame,
Frame,
- OpenAILLMContextAssistantTimestampFrame,
+ StartInterruptionFrame,
TranscriptionFrame,
TranscriptionMessage,
TranscriptionUpdateFrame,
+ TTSTextFrame,
)
-from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContextFrame
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
+from pipecat.utils.time import time_now_iso8601
class BaseTranscriptProcessor(FrameProcessor):
@@ -64,89 +70,74 @@ class UserTranscriptProcessor(BaseTranscriptProcessor):
class AssistantTranscriptProcessor(BaseTranscriptProcessor):
- """Processes assistant LLM context frames into timestamped conversation messages."""
+ """Processes assistant TTS text frames into timestamped conversation messages.
+
+ This processor aggregates TTS text frames into complete utterances and emits them as
+ transcript messages. Utterances are completed when:
+ - The bot stops speaking (BotStoppedSpeakingFrame)
+ - The bot is interrupted (StartInterruptionFrame)
+ - The pipeline ends (EndFrame)
+
+ Attributes:
+ _current_text_parts: List of text fragments being aggregated for current utterance
+ _aggregation_start_time: Timestamp when the current utterance began
+ """
def __init__(self, **kwargs):
- """Initialize processor with empty message stores."""
+ """Initialize processor with aggregation state."""
super().__init__(**kwargs)
- self._pending_assistant_messages: List[TranscriptionMessage] = []
+ self._current_text_parts: List[str] = []
+ self._aggregation_start_time: Optional[str] | None = None
- def _extract_messages(self, messages: List[dict]) -> List[TranscriptionMessage]:
- """Extract assistant messages from the OpenAI standard message format.
+ async def _emit_aggregated_text(self):
+ """Emit aggregated text as a transcript message."""
+ if self._current_text_parts and self._aggregation_start_time:
+ content = " ".join(self._current_text_parts).strip()
+ if content:
+ logger.debug(f"Emitting aggregated assistant message: {content}")
+ message = TranscriptionMessage(
+ role="assistant",
+ content=content,
+ timestamp=self._aggregation_start_time,
+ )
+ await self._emit_update([message])
+ else:
+ logger.debug("No content to emit after stripping whitespace")
- Args:
- messages: List of messages in OpenAI format, which can be either:
- - Simple format: {"role": "user", "content": "Hello"}
- - Content list: {"role": "user", "content": [{"type": "text", "text": "Hello"}]}
-
- Returns:
- List[TranscriptionMessage]: Normalized conversation messages
- """
- result = []
- for msg in messages:
- if msg["role"] != "assistant":
- continue
-
- content = msg.get("content")
- if isinstance(content, str):
- if content:
- result.append(TranscriptionMessage(role="assistant", content=content))
- elif isinstance(content, list):
- text_parts = []
- for part in content:
- if isinstance(part, dict) and part.get("type") == "text":
- text_parts.append(part["text"])
-
- if text_parts:
- result.append(
- TranscriptionMessage(role="assistant", content=" ".join(text_parts))
- )
-
- return result
-
- def _find_new_messages(self, current: List[TranscriptionMessage]) -> List[TranscriptionMessage]:
- """Find unprocessed messages from current list.
-
- Args:
- current: List of current messages
-
- Returns:
- List[TranscriptionMessage]: New messages not yet processed
- """
- if not self._processed_messages:
- return current
-
- processed_len = len(self._processed_messages)
- if len(current) <= processed_len:
- return []
-
- return current[processed_len:]
+ # Reset aggregation state
+ self._current_text_parts = []
+ self._aggregation_start_time = None
async def process_frame(self, frame: Frame, direction: FrameDirection):
"""Process frames into assistant conversation messages.
+ Handles different frame types:
+ - TTSTextFrame: Aggregates text for current utterance
+ - BotStoppedSpeakingFrame: Completes current utterance
+ - StartInterruptionFrame: Completes current utterance due to interruption
+ - EndFrame: Completes current utterance at pipeline end
+ - CancelFrame: Completes current utterance due to cancellation
+
Args:
frame: Input frame to process
direction: Frame processing direction
"""
await super().process_frame(frame, direction)
- if isinstance(frame, OpenAILLMContextFrame):
- standard_messages = []
- for msg in frame.context.messages:
- converted = frame.context.to_standard_messages(msg)
- standard_messages.extend(converted)
+ if isinstance(frame, TTSTextFrame):
+ # Start timestamp on first text part
+ if not self._aggregation_start_time:
+ self._aggregation_start_time = time_now_iso8601()
- current_messages = self._extract_messages(standard_messages)
- new_messages = self._find_new_messages(current_messages)
- self._pending_assistant_messages.extend(new_messages)
+ self._current_text_parts.append(frame.text)
- elif isinstance(frame, OpenAILLMContextAssistantTimestampFrame):
- if self._pending_assistant_messages:
- for msg in self._pending_assistant_messages:
- msg.timestamp = frame.timestamp
- await self._emit_update(self._pending_assistant_messages)
- self._pending_assistant_messages = []
+ elif isinstance(frame, (BotStoppedSpeakingFrame, StartInterruptionFrame, CancelFrame)):
+ # Emit accumulated text when bot finishes speaking or is interrupted
+ await self._emit_aggregated_text()
+
+ elif isinstance(frame, EndFrame):
+ # Emit any remaining text when pipeline ends
+ await self._emit_aggregated_text()
await self.push_frame(frame, direction)
@@ -170,8 +161,8 @@ class TranscriptProcessor:
llm,
tts,
transport.output(),
+ transcript.assistant_tts(), # Assistant transcripts
context_aggregator.assistant(),
- transcript.assistant(), # Assistant transcripts
]
)
diff --git a/src/pipecat/processors/user_idle_processor.py b/src/pipecat/processors/user_idle_processor.py
index 44027787a..e65bd437f 100644
--- a/src/pipecat/processors/user_idle_processor.py
+++ b/src/pipecat/processors/user_idle_processor.py
@@ -5,7 +5,8 @@
#
import asyncio
-from typing import Awaitable, Callable
+import inspect
+from typing import Awaitable, Callable, Union
from pipecat.frames.frames import (
BotSpeakingFrame,
@@ -19,63 +20,160 @@ from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
class UserIdleProcessor(FrameProcessor):
- """This class is useful to check if the user is interacting with the bot
- within a given timeout. If the timeout is reached before any interaction
- occurred the provided callback will be called.
+ """Monitors user inactivity and triggers callbacks after timeout periods.
+ Starts monitoring only after the first conversation activity (UserStartedSpeaking
+ or BotSpeaking).
+
+ Args:
+ callback: Function to call when user is idle. Can be either:
+ - Basic callback(processor) -> None
+ - Retry callback(processor, retry_count) -> bool
+ Return True to continue monitoring for idle events,
+ Return False to stop the idle monitoring task
+ timeout: Seconds to wait before considering user idle
+ **kwargs: Additional arguments passed to FrameProcessor
+
+ Example:
+ # Retry callback:
+ async def handle_idle(processor: "UserIdleProcessor", retry_count: int) -> bool:
+ if retry_count < 3:
+ await send_reminder("Are you still there?")
+ return True
+ return False
+
+ # Basic callback:
+ async def handle_idle(processor: "UserIdleProcessor") -> None:
+ await send_reminder("Are you still there?")
+
+ processor = UserIdleProcessor(
+ callback=handle_idle,
+ timeout=5.0
+ )
"""
def __init__(
self,
*,
- callback: Callable[["UserIdleProcessor"], Awaitable[None]],
+ callback: Union[
+ Callable[["UserIdleProcessor"], Awaitable[None]], # Basic
+ Callable[["UserIdleProcessor", int], Awaitable[bool]], # Retry
+ ],
timeout: float,
**kwargs,
):
super().__init__(**kwargs)
- self._callback = callback
+ self._callback = self._wrap_callback(callback)
self._timeout = timeout
+ self._retry_count = 0
self._interrupted = False
- self._create_idle_task()
+ self._conversation_started = False
+ self._idle_task = None
+ self._idle_event = asyncio.Event()
- async def _stop(self):
- self._idle_task.cancel()
- await self._idle_task
+ def _wrap_callback(
+ self,
+ callback: Union[
+ Callable[["UserIdleProcessor"], Awaitable[None]],
+ Callable[["UserIdleProcessor", int], Awaitable[bool]],
+ ],
+ ) -> Callable[["UserIdleProcessor", int], Awaitable[bool]]:
+ """Wraps callback to support both basic and retry signatures.
- async def process_frame(self, frame: Frame, direction: FrameDirection):
+ Args:
+ callback: The callback function to wrap.
+
+ Returns:
+ A wrapped callback that returns bool to indicate whether to continue monitoring.
+ """
+ sig = inspect.signature(callback)
+ param_count = len(sig.parameters)
+
+ async def wrapper(processor: "UserIdleProcessor", retry_count: int) -> bool:
+ if param_count == 1:
+ # Basic callback
+ await callback(processor) # type: ignore
+ return True
+ else:
+ # Retry callback
+ return await callback(processor, retry_count) # type: ignore
+
+ return wrapper
+
+ def _create_idle_task(self) -> None:
+ """Creates the idle task if it hasn't been created yet."""
+ if self._idle_task is None:
+ self._idle_task = self.create_task(self._idle_task_handler())
+
+ @property
+ def retry_count(self) -> int:
+ """Returns the current retry count."""
+ return self._retry_count
+
+ async def _stop(self) -> None:
+ """Stops and cleans up the idle monitoring task."""
+ if self._idle_task is not None:
+ await self.cancel_task(self._idle_task)
+ self._idle_task = None
+
+ async def process_frame(self, frame: Frame, direction: FrameDirection) -> None:
+ """Processes incoming frames and manages idle monitoring state.
+
+ Args:
+ frame: The frame to process
+ direction: Direction of the frame flow
+ """
await super().process_frame(frame, direction)
# Check for end frames before processing
if isinstance(frame, (EndFrame, CancelFrame)):
- await self._stop()
+ await self.push_frame(frame, direction) # Push the frame down the pipeline
+ if self._idle_task:
+ await self._stop() # Stop the idle task, if it exists
+ return
await self.push_frame(frame, direction)
- # We shouldn't call the idle callback if the user or the bot are speaking
- if isinstance(frame, UserStartedSpeakingFrame):
- self._interrupted = True
- self._idle_event.set()
- elif isinstance(frame, UserStoppedSpeakingFrame):
- self._interrupted = False
- self._idle_event.set()
- elif isinstance(frame, BotSpeakingFrame):
- self._idle_event.set()
+ # Start monitoring on first conversation activity
+ if not self._conversation_started and isinstance(
+ frame, (UserStartedSpeakingFrame, BotSpeakingFrame)
+ ):
+ self._conversation_started = True
+ self._create_idle_task()
- async def cleanup(self):
- await self._stop()
+ # Only process these events if conversation has started
+ if self._conversation_started:
+ # We shouldn't call the idle callback if the user or the bot are speaking
+ if isinstance(frame, UserStartedSpeakingFrame):
+ self._retry_count = 0 # Reset retry count when user speaks
+ self._interrupted = True
+ self._idle_event.set()
+ elif isinstance(frame, UserStoppedSpeakingFrame):
+ self._interrupted = False
+ self._idle_event.set()
+ elif isinstance(frame, BotSpeakingFrame):
+ self._idle_event.set()
- def _create_idle_task(self):
- self._idle_event = asyncio.Event()
- self._idle_task = self.get_event_loop().create_task(self._idle_task_handler())
+ async def cleanup(self) -> None:
+ """Cleans up resources when processor is shutting down."""
+ await super().cleanup()
+ if self._idle_task: # Only stop if task exists
+ await self._stop()
- async def _idle_task_handler(self):
+ async def _idle_task_handler(self) -> None:
+ """Monitors for idle timeout and triggers callbacks.
+
+ Runs in a loop until cancelled or callback indicates completion.
+ """
while True:
try:
await asyncio.wait_for(self._idle_event.wait(), timeout=self._timeout)
except asyncio.TimeoutError:
if not self._interrupted:
- await self._callback(self)
- except asyncio.CancelledError:
- break
+ self._retry_count += 1
+ should_continue = await self._callback(self, self._retry_count)
+ if not should_continue:
+ await self._stop()
+ break
finally:
self._idle_event.clear()
diff --git a/src/pipecat/serializers/protobuf.py b/src/pipecat/serializers/protobuf.py
index f35a1e17a..13d7ded34 100644
--- a/src/pipecat/serializers/protobuf.py
+++ b/src/pipecat/serializers/protobuf.py
@@ -93,11 +93,11 @@ class ProtobufFrameSerializer(FrameSerializer):
id = getattr(args, "id", None)
name = getattr(args, "name", None)
pts = getattr(args, "pts", None)
- if not id and "id" in args_dict:
+ if "id" in args_dict:
del args_dict["id"]
- if not name and "name" in args_dict:
+ if "name" in args_dict:
del args_dict["name"]
- if not pts and "pts" in args_dict:
+ if "pts" in args_dict:
del args_dict["pts"]
# Create the instance
@@ -105,10 +105,10 @@ class ProtobufFrameSerializer(FrameSerializer):
# Set special fields
if id:
- setattr(instance, "id", getattr(args, "id", None))
+ setattr(instance, "id", id)
if name:
- setattr(instance, "name", getattr(args, "name", None))
+ setattr(instance, "name", name)
if pts:
- setattr(instance, "pts", getattr(args, "pts", None))
+ setattr(instance, "pts", pts)
return instance
diff --git a/src/pipecat/serializers/twilio.py b/src/pipecat/serializers/twilio.py
index 558a68046..298e8a6cb 100644
--- a/src/pipecat/serializers/twilio.py
+++ b/src/pipecat/serializers/twilio.py
@@ -10,7 +10,14 @@ import json
from pydantic import BaseModel
from pipecat.audio.utils import pcm_to_ulaw, ulaw_to_pcm
-from pipecat.frames.frames import AudioRawFrame, Frame, InputAudioRawFrame, StartInterruptionFrame
+from pipecat.frames.frames import (
+ AudioRawFrame,
+ Frame,
+ InputAudioRawFrame,
+ InputDTMFFrame,
+ KeypadEntry,
+ StartInterruptionFrame,
+)
from pipecat.serializers.base_serializer import FrameSerializer, FrameSerializerType
@@ -48,9 +55,7 @@ class TwilioFrameSerializer(FrameSerializer):
def deserialize(self, data: str | bytes) -> Frame | None:
message = json.loads(data)
- if message["event"] != "media":
- return None
- else:
+ if message["event"] == "media":
payload_base64 = message["media"]["payload"]
payload = base64.b64decode(payload_base64)
@@ -61,3 +66,13 @@ class TwilioFrameSerializer(FrameSerializer):
audio=deserialized_data, num_channels=1, sample_rate=self._params.sample_rate
)
return audio_frame
+ elif message["event"] == "dtmf":
+ digit = message.get("dtmf", {}).get("digit")
+
+ try:
+ return InputDTMFFrame(KeypadEntry(digit))
+ except ValueError as e:
+ # Handle case where string doesn't match any enum value
+ return None
+ else:
+ return None
diff --git a/src/pipecat/services/ai_services.py b/src/pipecat/services/ai_services.py
index 973332623..bee15ceab 100644
--- a/src/pipecat/services/ai_services.py
+++ b/src/pipecat/services/ai_services.py
@@ -8,7 +8,7 @@ import asyncio
import io
import wave
from abc import abstractmethod
-from typing import Any, AsyncGenerator, Dict, List, Optional, Tuple
+from typing import Any, AsyncGenerator, Dict, List, Mapping, Optional, Tuple
from loguru import logger
@@ -19,6 +19,7 @@ from pipecat.frames.frames import (
EndFrame,
ErrorFrame,
Frame,
+ InterimTranscriptionFrame,
LLMFullResponseEndFrame,
StartFrame,
StartInterruptionFrame,
@@ -30,6 +31,7 @@ from pipecat.frames.frames import (
TTSSpeakFrame,
TTSStartedFrame,
TTSStoppedFrame,
+ TTSTextFrame,
TTSUpdateSettingsFrame,
UserImageRequestFrame,
VisionImageRawFrame,
@@ -67,7 +69,7 @@ class AIService(FrameProcessor):
async def cancel(self, frame: CancelFrame):
pass
- async def _update_settings(self, settings: Dict[str, Any]):
+ async def _update_settings(self, settings: Mapping[str, Any]):
from pipecat.services.openai_realtime_beta.events import (
SessionProperties,
)
@@ -230,11 +232,9 @@ class TTSService(AIService):
def sample_rate(self) -> int:
return self._sample_rate
- @abstractmethod
async def set_model(self, model: str):
self.set_model_name(model)
- @abstractmethod
def set_voice(self, voice: str):
self._voice_id = voice
@@ -253,23 +253,21 @@ class TTSService(AIService):
async def start(self, frame: StartFrame):
await super().start(frame)
if self._push_stop_frames:
- self._stop_frame_task = self.get_event_loop().create_task(self._stop_frame_handler())
+ self._stop_frame_task = self.create_task(self._stop_frame_handler())
async def stop(self, frame: EndFrame):
await super().stop(frame)
if self._stop_frame_task:
- self._stop_frame_task.cancel()
- await self._stop_frame_task
+ await self.cancel_task(self._stop_frame_task)
self._stop_frame_task = None
async def cancel(self, frame: CancelFrame):
await super().cancel(frame)
if self._stop_frame_task:
- self._stop_frame_task.cancel()
- await self._stop_frame_task
+ await self.cancel_task(self._stop_frame_task)
self._stop_frame_task = None
- async def _update_settings(self, settings: Dict[str, Any]):
+ async def _update_settings(self, settings: Mapping[str, Any]):
for key, value in settings.items():
if key in self._settings:
logger.info(f"Updating TTS setting {key} to: [{value}]")
@@ -290,8 +288,11 @@ class TTSService(AIService):
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
-
- if isinstance(frame, TextFrame) and not isinstance(frame, TranscriptionFrame):
+ if (
+ isinstance(frame, TextFrame)
+ and not isinstance(frame, InterimTranscriptionFrame)
+ and not isinstance(frame, TranscriptionFrame)
+ ):
await self._process_text_frame(frame)
elif isinstance(frame, StartInterruptionFrame):
await self._handle_interruption(frame, direction)
@@ -358,26 +359,23 @@ class TTSService(AIService):
if self._push_text_frames:
# We send the original text after the audio. This way, if we are
# interrupted, the text is not added to the assistant context.
- await self.push_frame(TextFrame(text))
+ await self.push_frame(TTSTextFrame(text))
async def _stop_frame_handler(self):
- try:
- has_started = False
- while True:
- try:
- frame = await asyncio.wait_for(
- self._stop_frame_queue.get(), self._stop_frame_timeout_s
- )
- if isinstance(frame, TTSStartedFrame):
- has_started = True
- elif isinstance(frame, (TTSStoppedFrame, StartInterruptionFrame)):
- has_started = False
- except asyncio.TimeoutError:
- if has_started:
- await self.push_frame(TTSStoppedFrame())
- has_started = False
- except asyncio.CancelledError:
- pass
+ has_started = False
+ while True:
+ try:
+ frame = await asyncio.wait_for(
+ self._stop_frame_queue.get(), self._stop_frame_timeout_s
+ )
+ if isinstance(frame, TTSStartedFrame):
+ has_started = True
+ elif isinstance(frame, (TTSStoppedFrame, StartInterruptionFrame)):
+ has_started = False
+ except asyncio.TimeoutError:
+ if has_started:
+ await self.push_frame(TTSStoppedFrame())
+ has_started = False
class WordTTSService(TTSService):
@@ -385,7 +383,7 @@ class WordTTSService(TTSService):
super().__init__(**kwargs)
self._initial_word_timestamp = -1
self._words_queue = asyncio.Queue()
- self._words_task = self.get_event_loop().create_task(self._words_task_handler())
+ self._words_task = self.create_task(self._words_task_handler())
def start_word_timestamps(self):
if self._initial_word_timestamp == -1:
@@ -418,35 +416,29 @@ class WordTTSService(TTSService):
async def _stop_words_task(self):
if self._words_task:
- self._words_task.cancel()
- await self._words_task
+ await self.cancel_task(self._words_task)
self._words_task = None
async def _words_task_handler(self):
last_pts = 0
while True:
- try:
- (word, timestamp) = await self._words_queue.get()
- if word == "Reset" and timestamp == 0:
- self.reset_word_timestamps()
- frame = None
- elif word == "LLMFullResponseEndFrame" and timestamp == 0:
- frame = LLMFullResponseEndFrame()
- frame.pts = last_pts
- elif word == "TTSStoppedFrame" and timestamp == 0:
- frame = TTSStoppedFrame()
- frame.pts = last_pts
- else:
- frame = TextFrame(word)
- frame.pts = self._initial_word_timestamp + timestamp
- if frame:
- last_pts = frame.pts
- await self.push_frame(frame)
- self._words_queue.task_done()
- except asyncio.CancelledError:
- break
- except Exception as e:
- logger.exception(f"{self} exception: {e}")
+ (word, timestamp) = await self._words_queue.get()
+ if word == "Reset" and timestamp == 0:
+ self.reset_word_timestamps()
+ frame = None
+ elif word == "LLMFullResponseEndFrame" and timestamp == 0:
+ frame = LLMFullResponseEndFrame()
+ frame.pts = last_pts
+ elif word == "TTSStoppedFrame" and timestamp == 0:
+ frame = TTSStoppedFrame()
+ frame.pts = last_pts
+ else:
+ frame = TTSTextFrame(word)
+ frame.pts = self._initial_word_timestamp + timestamp
+ if frame:
+ last_pts = frame.pts
+ await self.push_frame(frame)
+ self._words_queue.task_done()
class STTService(AIService):
@@ -476,7 +468,7 @@ class STTService(AIService):
"""Returns transcript as a string"""
pass
- async def _update_settings(self, settings: Dict[str, Any]):
+ async def _update_settings(self, settings: Mapping[str, Any]):
logger.info(f"Updating STT settings: {self._settings}")
for key, value in settings.items():
if key in self._settings:
diff --git a/src/pipecat/services/anthropic.py b/src/pipecat/services/anthropic.py
index 08bde31b7..1590296af 100644
--- a/src/pipecat/services/anthropic.py
+++ b/src/pipecat/services/anthropic.py
@@ -26,10 +26,10 @@ from pipecat.frames.frames import (
LLMFullResponseEndFrame,
LLMFullResponseStartFrame,
LLMMessagesFrame,
+ LLMTextFrame,
LLMUpdateSettingsFrame,
OpenAILLMContextAssistantTimestampFrame,
StartInterruptionFrame,
- TextFrame,
UserImageRawFrame,
UserImageRequestFrame,
VisionImageRawFrame,
@@ -191,7 +191,7 @@ class AnthropicLLMService(LLMService):
if event.type == "content_block_delta":
if hasattr(event.delta, "text"):
- await self.push_frame(TextFrame(event.delta.text))
+ await self.push_frame(LLMTextFrame(event.delta.text))
completion_tokens_estimate += self._estimate_tokens(event.delta.text)
elif hasattr(event.delta, "partial_json") and tool_use_block:
json_accumulator += event.delta.partial_json
diff --git a/src/pipecat/services/canonical.py b/src/pipecat/services/canonical.py
index dca951f8c..884171b10 100644
--- a/src/pipecat/services/canonical.py
+++ b/src/pipecat/services/canonical.py
@@ -15,7 +15,7 @@ import aiohttp
from loguru import logger
from pipecat.frames.frames import CancelFrame, EndFrame, Frame
-from pipecat.processors.audio import audio_buffer_processor
+from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.processors.audio.audio_buffer_processor import AudioBufferProcessor
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_services import AIService
@@ -69,6 +69,7 @@ class CanonicalMetricsService(AIService):
api_url: str = "https://voiceapp.canonical.chat/api/v1",
assistant_speaks_first: bool = True,
output_dir: str = "recordings",
+ context: OpenAILLMContext | None = None,
**kwargs,
):
super().__init__(**kwargs)
@@ -80,6 +81,7 @@ class CanonicalMetricsService(AIService):
self._assistant = assistant
self._assistant_speaks_first = assistant_speaks_first
self._output_dir = output_dir
+ self._context = context
async def stop(self, frame: EndFrame):
await super().stop(frame)
@@ -190,6 +192,9 @@ class CanonicalMetricsService(AIService):
"callId": self._call_id,
"assistant": {"id": self._assistant, "speaksFirst": self._assistant_speaks_first},
}
+ if self._context is not None:
+ params["transcript"] = self._context.messages
+
logger.debug(f"Completing upload for {params['filename']}")
logger.debug(f"Slug: {params['slug']}")
response = await self._aiohttp_session.post(
diff --git a/src/pipecat/services/cartesia.py b/src/pipecat/services/cartesia.py
index 9712ce600..75da116d5 100644
--- a/src/pipecat/services/cartesia.py
+++ b/src/pipecat/services/cartesia.py
@@ -88,7 +88,7 @@ class CartesiaTTSService(WordTTSService, WebsocketService):
voice_id: str,
cartesia_version: str = "2024-06-10",
url: str = "wss://api.cartesia.ai/tts/websocket",
- model: str = "sonic-english",
+ model: str = "sonic",
sample_rate: int = 24000,
encoding: str = "pcm_s16le",
container: str = "raw",
@@ -187,16 +187,13 @@ class CartesiaTTSService(WordTTSService, WebsocketService):
async def _connect(self):
await self._connect_websocket()
- self._receive_task = self.get_event_loop().create_task(
- self._receive_task_handler(self.push_error)
- )
+ self._receive_task = self.create_task(self._receive_task_handler(self.push_error))
async def _disconnect(self):
await self._disconnect_websocket()
if self._receive_task:
- self._receive_task.cancel()
- await self._receive_task
+ await self.cancel_task(self._receive_task)
self._receive_task = None
async def _connect_websocket(self):
@@ -238,7 +235,7 @@ class CartesiaTTSService(WordTTSService, WebsocketService):
async def flush_audio(self):
if not self._context_id or not self._websocket:
return
- logger.trace("Flushing audio")
+ logger.trace(f"{self}: flushing audio")
msg = self._build_msg(text="", continue_transcript=False)
await self._websocket.send(msg)
@@ -273,7 +270,7 @@ class CartesiaTTSService(WordTTSService, WebsocketService):
logger.error(f"{self} error: {msg}")
await self.push_frame(TTSStoppedFrame())
await self.stop_all_metrics()
- await self.push_error(ErrorFrame(f'{self} error: {msg["error"]}'))
+ await self.push_error(ErrorFrame(f"{self} error: {msg['error']}"))
else:
logger.error(f"{self} error, unknown message type: {msg}")
@@ -329,7 +326,7 @@ class CartesiaHttpTTSService(TTSService):
*,
api_key: str,
voice_id: str,
- model: str = "sonic-english",
+ model: str = "sonic",
base_url: str = "https://api.cartesia.ai",
sample_rate: int = 24000,
encoding: str = "pcm_s16le",
diff --git a/src/pipecat/services/deepgram.py b/src/pipecat/services/deepgram.py
index bba2e72ae..a5d36370f 100644
--- a/src/pipecat/services/deepgram.py
+++ b/src/pipecat/services/deepgram.py
@@ -20,6 +20,7 @@ from pipecat.frames.frames import (
TTSAudioRawFrame,
TTSStartedFrame,
TTSStoppedFrame,
+ UserStartedSpeakingFrame,
UserStoppedSpeakingFrame,
)
from pipecat.processors.frame_processor import FrameDirection
@@ -169,7 +170,7 @@ class DeepgramSTTService(STTService):
return self._settings["vad_events"]
def can_generate_metrics(self) -> bool:
- return self.vad_enabled
+ return True
async def set_model(self, model: str):
await super().set_model(model)
@@ -210,9 +211,12 @@ class DeepgramSTTService(STTService):
logger.debug("Disconnecting from Deepgram")
await self._connection.finish()
- async def _on_speech_started(self, *args, **kwargs):
+ async def start_metrics(self):
await self.start_ttfb_metrics()
await self.start_processing_metrics()
+
+ async def _on_speech_started(self, *args, **kwargs):
+ await self.start_metrics()
await self._call_event_handler("on_speech_started", *args, **kwargs)
async def _on_utterance_end(self, *args, **kwargs):
@@ -243,7 +247,10 @@ class DeepgramSTTService(STTService):
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
- if isinstance(frame, UserStoppedSpeakingFrame):
+ if isinstance(frame, UserStartedSpeakingFrame) and not self.vad_enabled:
+ # Start metrics if Deepgram VAD is disabled & pipeline VAD has detected speech
+ await self.start_metrics()
+ elif isinstance(frame, UserStoppedSpeakingFrame):
# https://developers.deepgram.com/docs/finalize
await self._connection.finalize()
- logger.debug(f"Triggering finalize event on: {frame.name=}, {direction=}")
+ logger.trace(f"Triggered finalize event on: {frame.name=}, {direction=}")
diff --git a/src/pipecat/services/elevenlabs.py b/src/pipecat/services/elevenlabs.py
index c1d326dfc..546ec1edd 100644
--- a/src/pipecat/services/elevenlabs.py
+++ b/src/pipecat/services/elevenlabs.py
@@ -7,8 +7,9 @@
import asyncio
import base64
import json
-from typing import Any, AsyncGenerator, Dict, List, Literal, Mapping, Optional, Tuple
+from typing import Any, AsyncGenerator, Dict, List, Literal, Mapping, Optional, Tuple, Union
+import aiohttp
from loguru import logger
from pydantic import BaseModel, model_validator
@@ -16,6 +17,7 @@ from pipecat.frames.frames import (
BotStoppedSpeakingFrame,
CancelFrame,
EndFrame,
+ ErrorFrame,
Frame,
LLMFullResponseEndFrame,
StartFrame,
@@ -26,7 +28,7 @@ from pipecat.frames.frames import (
TTSStoppedFrame,
)
from pipecat.processors.frame_processor import FrameDirection
-from pipecat.services.ai_services import WordTTSService
+from pipecat.services.ai_services import TTSService, WordTTSService
from pipecat.services.websocket_service import WebsocketService
from pipecat.transcriptions.language import Language
@@ -42,10 +44,11 @@ except ModuleNotFoundError as e:
ElevenLabsOutputFormat = Literal["pcm_16000", "pcm_22050", "pcm_24000", "pcm_44100"]
+# Models that support language codes
+# eleven_multilingual_v2 doesn't support language codes, so it's excluded
ELEVENLABS_MULTILINGUAL_MODELS = {
- "eleven_turbo_v2_5",
- "eleven_multilingual_v2",
"eleven_flash_v2_5",
+ "eleven_turbo_v2_5",
}
@@ -296,20 +299,16 @@ class ElevenLabsTTSService(WordTTSService, WebsocketService):
async def _connect(self):
await self._connect_websocket()
- self._receive_task = self.get_event_loop().create_task(
- self._receive_task_handler(self.push_error)
- )
- self._keepalive_task = self.get_event_loop().create_task(self._keepalive_task_handler())
+ self._receive_task = self.create_task(self._receive_task_handler(self.push_error))
+ self._keepalive_task = self.create_task(self._keepalive_task_handler())
async def _disconnect(self):
if self._receive_task:
- self._receive_task.cancel()
- await self._receive_task
+ await self.cancel_task(self._receive_task)
self._receive_task = None
if self._keepalive_task:
- self._keepalive_task.cancel()
- await self._keepalive_task
+ await self.cancel_task(self._keepalive_task)
self._keepalive_task = None
await self._disconnect_websocket()
@@ -335,7 +334,8 @@ class ElevenLabsTTSService(WordTTSService, WebsocketService):
f"Language code [{language}] not applied. Language codes can only be used with multilingual models: {', '.join(sorted(ELEVENLABS_MULTILINGUAL_MODELS))}"
)
- self._websocket = await websockets.connect(url)
+ # Set max websocket message size to 16MB for large audio responses
+ self._websocket = await websockets.connect(url, max_size=16 * 1024 * 1024)
# According to ElevenLabs, we should always start with a single space.
msg: Dict[str, Any] = {
@@ -380,13 +380,8 @@ class ElevenLabsTTSService(WordTTSService, WebsocketService):
async def _keepalive_task_handler(self):
while True:
- try:
- await asyncio.sleep(10)
- await self._send_text("")
- except asyncio.CancelledError:
- break
- except Exception as e:
- logger.error(f"{self} exception: {e}")
+ await asyncio.sleep(10)
+ await self._send_text("")
async def _send_text(self, text: str):
if self._websocket:
@@ -418,3 +413,160 @@ class ElevenLabsTTSService(WordTTSService, WebsocketService):
yield None
except Exception as e:
logger.error(f"{self} exception: {e}")
+
+
+class ElevenLabsHttpTTSService(TTSService):
+ """ElevenLabs Text-to-Speech service using HTTP streaming.
+
+ Args:
+ api_key: ElevenLabs API key
+ voice_id: ID of the voice to use
+ aiohttp_session: aiohttp ClientSession
+ model: Model ID (default: "eleven_flash_v2_5" for low latency)
+ base_url: API base URL
+ output_format: Audio output format (PCM)
+ params: Additional parameters for voice configuration
+ """
+
+ class InputParams(BaseModel):
+ language: Optional[Language] = Language.EN
+ optimize_streaming_latency: Optional[int] = None
+ stability: Optional[float] = None
+ similarity_boost: Optional[float] = None
+ style: Optional[float] = None
+ use_speaker_boost: Optional[bool] = None
+
+ def __init__(
+ self,
+ *,
+ api_key: str,
+ voice_id: str,
+ aiohttp_session: aiohttp.ClientSession,
+ model: str = "eleven_flash_v2_5",
+ base_url: str = "https://api.elevenlabs.io",
+ output_format: ElevenLabsOutputFormat = "pcm_24000",
+ params: InputParams = InputParams(),
+ **kwargs,
+ ):
+ super().__init__(sample_rate=sample_rate_from_output_format(output_format), **kwargs)
+
+ self._api_key = api_key
+ self._base_url = base_url
+ self._output_format = output_format
+ self._params = params
+ self._session = aiohttp_session
+
+ self._settings = {
+ "sample_rate": sample_rate_from_output_format(output_format),
+ "language": self.language_to_service_language(params.language)
+ if params.language
+ else "en",
+ "output_format": output_format,
+ "optimize_streaming_latency": params.optimize_streaming_latency,
+ "stability": params.stability,
+ "similarity_boost": params.similarity_boost,
+ "style": params.style,
+ "use_speaker_boost": params.use_speaker_boost,
+ }
+ self.set_model_name(model)
+ self.set_voice(voice_id)
+ self._voice_settings = self._set_voice_settings()
+
+ def can_generate_metrics(self) -> bool:
+ return True
+
+ def _set_voice_settings(self) -> Optional[Dict[str, Union[float, bool]]]:
+ """Configure voice settings if stability and similarity_boost are provided.
+
+ Returns:
+ Dictionary of voice settings or None if required parameters are missing.
+ """
+ voice_settings: Dict[str, Union[float, bool]] = {}
+ if (
+ self._settings["stability"] is not None
+ and self._settings["similarity_boost"] is not None
+ ):
+ voice_settings["stability"] = float(self._settings["stability"])
+ voice_settings["similarity_boost"] = float(self._settings["similarity_boost"])
+ if self._settings["style"] is not None:
+ voice_settings["style"] = float(self._settings["style"])
+ if self._settings["use_speaker_boost"] is not None:
+ voice_settings["use_speaker_boost"] = bool(self._settings["use_speaker_boost"])
+ else:
+ if self._settings["style"] is not None:
+ logger.warning(
+ "'style' is set but will not be applied because 'stability' and 'similarity_boost' are not both set."
+ )
+ if self._settings["use_speaker_boost"] is not None:
+ logger.warning(
+ "'use_speaker_boost' is set but will not be applied because 'stability' and 'similarity_boost' are not both set."
+ )
+
+ return voice_settings or None
+
+ async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]:
+ """Generate speech from text using ElevenLabs streaming API.
+
+ Args:
+ text: The text to convert to speech
+
+ Yields:
+ Frames containing audio data and status information
+ """
+ logger.debug(f"Generating TTS: [{text}]")
+
+ url = f"{self._base_url}/v1/text-to-speech/{self._voice_id}/stream"
+
+ payload: Dict[str, Union[str, Dict[str, Union[float, bool]]]] = {
+ "text": text,
+ "model_id": self._model_name,
+ }
+
+ if self._voice_settings:
+ payload["voice_settings"] = self._voice_settings
+
+ if self._settings["language"]:
+ payload["language_code"] = self._settings["language"]
+
+ headers = {
+ "xi-api-key": self._api_key,
+ "Content-Type": "application/json",
+ }
+
+ # Build query parameters
+ params = {
+ "output_format": self._output_format,
+ }
+ if self._settings["optimize_streaming_latency"] is not None:
+ params["optimize_streaming_latency"] = self._settings["optimize_streaming_latency"]
+
+ logger.debug(f"ElevenLabs request - payload: {payload}, params: {params}")
+
+ try:
+ await self.start_ttfb_metrics()
+
+ async with self._session.post(
+ url, json=payload, headers=headers, params=params
+ ) as response:
+ if response.status != 200:
+ error_text = await response.text()
+ logger.error(f"{self} error: {error_text}")
+ yield ErrorFrame(error=f"ElevenLabs API error: {error_text}")
+ return
+
+ await self.start_tts_usage_metrics(text)
+ yield TTSStartedFrame()
+
+ async for chunk in response.content:
+ if chunk:
+ await self.stop_ttfb_metrics()
+ yield TTSAudioRawFrame(chunk, self._settings["sample_rate"], 1)
+
+ yield TTSStoppedFrame()
+
+ except Exception as e:
+ logger.error(f"Error in run_tts: {e}")
+ yield ErrorFrame(error=str(e))
+
+ finally:
+ yield TTSStoppedFrame()
diff --git a/src/pipecat/services/fish.py b/src/pipecat/services/fish.py
index 245497b1d..6fadab683 100644
--- a/src/pipecat/services/fish.py
+++ b/src/pipecat/services/fish.py
@@ -4,13 +4,11 @@
# SPDX-License-Identifier: BSD 2-Clause License
#
-import asyncio
import uuid
from typing import AsyncGenerator, Literal, Optional
from loguru import logger
from pydantic import BaseModel
-from tenacity import AsyncRetrying, RetryCallState, stop_after_attempt, wait_exponential
from pipecat.frames.frames import (
BotStoppedSpeakingFrame,
@@ -28,6 +26,7 @@ from pipecat.frames.frames import (
)
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_services import TTSService
+from pipecat.services.websocket_service import WebsocketService
from pipecat.transcriptions.language import Language
try:
@@ -44,7 +43,7 @@ except ModuleNotFoundError as e:
FishAudioOutputFormat = Literal["opus", "mp3", "pcm", "wav"]
-class FishAudioTTSService(TTSService):
+class FishAudioTTSService(TTSService, WebsocketService):
class InputParams(BaseModel):
language: Optional[Language] = Language.EN
latency: Optional[str] = "normal" # "normal" or "balanced"
@@ -105,13 +104,12 @@ class FishAudioTTSService(TTSService):
async def _connect(self):
await self._connect_websocket()
- self._receive_task = self.get_event_loop().create_task(self._receive_task_handler())
+ self._receive_task = self.create_task(self._receive_task_handler(self.push_error))
async def _disconnect(self):
await self._disconnect_websocket()
if self._receive_task:
- self._receive_task.cancel()
- await self._receive_task
+ await self.cancel_task(self._receive_task)
self._receive_task = None
async def _connect_websocket(self):
@@ -169,30 +167,6 @@ class FishAudioTTSService(TTSService):
except Exception as e:
logger.error(f"Error processing message: {e}")
- async def _reconnect_websocket(self, retry_state: RetryCallState):
- logger.warning(f"Fish Audio reconnecting (attempt: {retry_state.attempt_number})")
- await self._disconnect_websocket()
- await self._connect_websocket()
-
- async def _receive_task_handler(self):
- while True:
- try:
- async for attempt in AsyncRetrying(
- stop=stop_after_attempt(3),
- wait=wait_exponential(multiplier=1, min=4, max=10),
- before_sleep=self._reconnect_websocket,
- reraise=True,
- ):
- with attempt:
- await self._receive_messages()
- except asyncio.CancelledError:
- break
- except Exception as e:
- message = f"Fish Audio error receiving messages: {e}"
- logger.error(message)
- await self.push_error(ErrorFrame(message, fatal=True))
- break
-
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
diff --git a/src/pipecat/services/gemini_multimodal_live/audio_transcriber.py b/src/pipecat/services/gemini_multimodal_live/audio_transcriber.py
index 54eccd56b..3ac60c5a7 100644
--- a/src/pipecat/services/gemini_multimodal_live/audio_transcriber.py
+++ b/src/pipecat/services/gemini_multimodal_live/audio_transcriber.py
@@ -70,7 +70,7 @@ class AudioTranscriber:
history = ""
for msg in previous_messages:
- content = msg.get("content")
+ content = msg.get("content", [])
if isinstance(content, str):
history += f"{msg.get('role')}: {content}\n"
else:
diff --git a/src/pipecat/services/gemini_multimodal_live/gemini.py b/src/pipecat/services/gemini_multimodal_live/gemini.py
index 6ce09e721..9f355d1ba 100644
--- a/src/pipecat/services/gemini_multimodal_live/gemini.py
+++ b/src/pipecat/services/gemini_multimodal_live/gemini.py
@@ -28,10 +28,10 @@ from pipecat.frames.frames import (
LLMFullResponseStartFrame,
LLMMessagesAppendFrame,
LLMSetToolsFrame,
+ LLMTextFrame,
LLMUpdateSettingsFrame,
StartFrame,
StartInterruptionFrame,
- TextFrame,
TranscriptionFrame,
TTSAudioRawFrame,
TTSStartedFrame,
@@ -182,9 +182,13 @@ class GeminiMultimodalLiveLLMService(LLMService):
self._audio_input_paused = start_audio_paused
self._video_input_paused = start_video_paused
+ self._context = None
self._websocket = None
self._receive_task = None
- self._context = None
+ self._transcribe_audio_task = None
+ self._transcribe_model_audio_task = None
+ self._transcribe_audio_queue = asyncio.Queue()
+ self._transcribe_model_audio_queue = asyncio.Queue()
self._disconnecting = False
self._api_session_ready = False
@@ -244,6 +248,7 @@ class GeminiMultimodalLiveLLMService(LLMService):
async def start(self, frame: StartFrame):
await super().start(frame)
+ await self._connect()
async def stop(self, frame: EndFrame):
await super().stop(frame)
@@ -275,7 +280,7 @@ class GeminiMultimodalLiveLLMService(LLMService):
)
await self.send_client_event(evt)
if self._transcribe_user_audio and self._context:
- asyncio.create_task(self._handle_transcribe_user_audio(audio, self._context))
+ await self._transcribe_audio_queue.put(audio)
async def _handle_transcribe_user_audio(self, audio, context):
text = await self._transcribe_audio(audio, context)
@@ -288,6 +293,10 @@ class GeminiMultimodalLiveLLMService(LLMService):
)
async def _handle_transcribe_model_audio(self, audio, context):
+ # Early return if modalities are not set to audio.
+ if self._settings["modalities"] != GeminiMultimodalModalities.AUDIO:
+ return
+
text = await self._transcribe_audio(audio, context)
logger.debug(f"[Transcription:model] {text}")
# We add user messages directly to the context. We don't do that for assistant messages,
@@ -295,7 +304,7 @@ class GeminiMultimodalLiveLLMService(LLMService):
# definitely feels like a hack. Need to revisit when the API evolves.
# context.add_message({"role": "assistant", "content": [{"type": "text", "text": text}]})
await self.push_frame(LLMFullResponseStartFrame())
- await self.push_frame(TextFrame(text=text))
+ await self.push_frame(LLMTextFrame(text=text))
await self.push_frame(LLMFullResponseEndFrame())
async def _transcribe_audio(self, audio, context):
@@ -377,17 +386,21 @@ class GeminiMultimodalLiveLLMService(LLMService):
await self._ws_send(event.model_dump(exclude_none=True))
async def _connect(self):
+ if self._websocket:
+ # Here we assume that if we have a websocket, we are connected. We
+ # handle disconnections in the send/recv code paths.
+ return
+
logger.info("Connecting to Gemini service")
try:
- if self._websocket:
- # Here we assume that if we have a websocket, we are connected. We
- # handle disconnections in the send/recv code paths.
- return
-
uri = f"wss://{self.base_url}/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContent?key={self.api_key}"
logger.info(f"Connecting to {uri}")
self._websocket = await websockets.connect(uri=uri)
- self._receive_task = self.get_event_loop().create_task(self._receive_task_handler())
+ self._receive_task = self.create_task(self._receive_task_handler())
+ self._transcribe_audio_task = self.create_task(self._transcribe_audio_handler())
+ self._transcribe_model_audio_task = self.create_task(
+ self._transcribe_model_audio_handler()
+ )
config = events.Config.model_validate(
{
"setup": {
@@ -437,12 +450,14 @@ class GeminiMultimodalLiveLLMService(LLMService):
await self._websocket.close()
self._websocket = None
if self._receive_task:
- self._receive_task.cancel()
- try:
- await asyncio.wait_for(self._receive_task, timeout=1.0)
- except asyncio.TimeoutError:
- logger.warning("Timed out waiting for receive task to finish")
+ await self.cancel_task(self._receive_task, timeout=1.0)
self._receive_task = None
+ if self._transcribe_audio_task:
+ await self.cancel_task(self._transcribe_audio_task)
+ self._transcribe_audio_task = None
+ if self._transcribe_model_audio_task:
+ await self.cancel_task(self._transcribe_model_audio_task)
+ self._transcribe_model_audio_task = None
self._disconnecting = False
except Exception as e:
logger.error(f"{self} error disconnecting: {e}")
@@ -450,9 +465,8 @@ class GeminiMultimodalLiveLLMService(LLMService):
async def _ws_send(self, message):
# logger.debug(f"Sending message to websocket: {message}")
try:
- if not self._websocket:
- await self._connect()
- await self._websocket.send(json.dumps(message))
+ if self._websocket:
+ await self._websocket.send(json.dumps(message))
except Exception as e:
if self._disconnecting:
return
@@ -469,32 +483,35 @@ class GeminiMultimodalLiveLLMService(LLMService):
#
async def _receive_task_handler(self):
- try:
- async for message in self._websocket:
- evt = events.parse_server_event(message)
- # logger.debug(f"Received event: {message[:500]}")
- # logger.debug(f"Received event: {evt}")
+ async for message in self._websocket:
+ evt = events.parse_server_event(message)
+ # logger.debug(f"Received event: {message[:500]}")
+ # logger.debug(f"Received event: {evt}")
- if evt.setupComplete:
- await self._handle_evt_setup_complete(evt)
- elif evt.serverContent and evt.serverContent.modelTurn:
- await self._handle_evt_model_turn(evt)
- elif evt.serverContent and evt.serverContent.turnComplete:
- await self._handle_evt_turn_complete(evt)
- elif evt.toolCall:
- await self._handle_evt_tool_call(evt)
+ if evt.setupComplete:
+ await self._handle_evt_setup_complete(evt)
+ elif evt.serverContent and evt.serverContent.modelTurn:
+ await self._handle_evt_model_turn(evt)
+ elif evt.serverContent and evt.serverContent.turnComplete:
+ await self._handle_evt_turn_complete(evt)
+ elif evt.toolCall:
+ await self._handle_evt_tool_call(evt)
+ elif False: # !!! todo: error events?
+ await self._handle_evt_error(evt)
+ # errors are fatal, so exit the receive loop
+ return
+ else:
+ pass
- elif False: # !!! todo: error events?
- await self._handle_evt_error(evt)
- # errors are fatal, so exit the receive loop
- return
+ async def _transcribe_audio_handler(self):
+ while True:
+ audio = await self._transcribe_audio_queue.get()
+ await self._handle_transcribe_user_audio(audio, self._context)
- else:
- pass
- except asyncio.CancelledError:
- logger.debug("websocket receive task cancelled")
- except Exception as e:
- logger.error(f"{self} exception: {e}")
+ async def _transcribe_model_audio_handler(self):
+ while True:
+ audio = await self._transcribe_model_audio_queue.get()
+ await self._handle_transcribe_model_audio(audio, self._context)
#
#
@@ -628,7 +645,7 @@ class GeminiMultimodalLiveLLMService(LLMService):
await self.push_frame(LLMFullResponseStartFrame())
self._bot_text_buffer += text
- await self.push_frame(TextFrame(text=text))
+ await self.push_frame(LLMTextFrame(text=text))
inline_data = part.inlineData
if not inline_data:
@@ -675,7 +692,7 @@ class GeminiMultimodalLiveLLMService(LLMService):
self._bot_text_buffer = ""
if audio and self._transcribe_model_audio and self._context:
- asyncio.create_task(self._handle_transcribe_model_audio(audio, self._context))
+ await self._transcribe_model_audio_queue.put(audio)
elif text:
await self.push_frame(LLMFullResponseEndFrame())
diff --git a/src/pipecat/services/gladia.py b/src/pipecat/services/gladia.py
index 3487be5ae..e240252a2 100644
--- a/src/pipecat/services/gladia.py
+++ b/src/pipecat/services/gladia.py
@@ -180,7 +180,7 @@ class GladiaSTTService(STTService):
await super().start(frame)
response = await self._setup_gladia()
self._websocket = await websockets.connect(response["url"])
- self._receive_task = self.get_event_loop().create_task(self._receive_task_handler())
+ self._receive_task = self.create_task(self._receive_task_handler())
async def stop(self, frame: EndFrame):
await super().stop(frame)
diff --git a/src/pipecat/services/google/__init__.py b/src/pipecat/services/google/__init__.py
new file mode 100644
index 000000000..1b81980e6
--- /dev/null
+++ b/src/pipecat/services/google/__init__.py
@@ -0,0 +1,2 @@
+from .frames import LLMSearchResponseFrame
+from .google import *
diff --git a/src/pipecat/services/google/frames.py b/src/pipecat/services/google/frames.py
new file mode 100644
index 000000000..700a39a7e
--- /dev/null
+++ b/src/pipecat/services/google/frames.py
@@ -0,0 +1,33 @@
+#
+# Copyright (c) 2024–2025, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+from dataclasses import dataclass, field
+from typing import List, Optional
+
+from pipecat.frames.frames import DataFrame
+
+
+@dataclass
+class LLMSearchResult:
+ text: str
+ confidence: List[float] = field(default_factory=list)
+
+
+@dataclass
+class LLMSearchOrigin:
+ site_uri: Optional[str] = None
+ site_title: Optional[str] = None
+ results: List[LLMSearchResult] = field(default_factory=list)
+
+
+@dataclass
+class LLMSearchResponseFrame(DataFrame):
+ search_result: Optional[str] = None
+ rendered_content: Optional[str] = None
+ origins: List[LLMSearchOrigin] = field(default_factory=list)
+
+ def __str__(self):
+ return f"LLMSearchResponseFrame(search_result={self.search_result}, origins={self.origins})"
diff --git a/src/pipecat/services/google.py b/src/pipecat/services/google/google.py
similarity index 88%
rename from src/pipecat/services/google.py
rename to src/pipecat/services/google/google.py
index 341c51636..c9be2f00a 100644
--- a/src/pipecat/services/google.py
+++ b/src/pipecat/services/google/google.py
@@ -23,9 +23,9 @@ from pipecat.frames.frames import (
LLMFullResponseEndFrame,
LLMFullResponseStartFrame,
LLMMessagesFrame,
+ LLMTextFrame,
LLMUpdateSettingsFrame,
OpenAILLMContextAssistantTimestampFrame,
- TextFrame,
TTSAudioRawFrame,
TTSStartedFrame,
TTSStoppedFrame,
@@ -38,6 +38,7 @@ from pipecat.processors.aggregators.openai_llm_context import (
)
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_services import LLMService, TTSService
+from pipecat.services.google.frames import LLMSearchResponseFrame
from pipecat.services.openai import (
OpenAIAssistantContextAggregator,
OpenAIUserContextAggregator,
@@ -639,6 +640,9 @@ class GoogleLLMService(LLMService):
completion_tokens = 0
total_tokens = 0
+ grounding_metadata = None
+ search_result = ""
+
try:
logger.debug(
# f"Generating chat: {self._system_instruction} | {context.get_messages_for_logging()}"
@@ -698,7 +702,8 @@ class GoogleLLMService(LLMService):
try:
for c in chunk.parts:
if c.text:
- await self.push_frame(TextFrame(c.text))
+ search_result += c.text
+ await self.push_frame(LLMTextFrame(c.text))
elif c.function_call:
logger.debug(f"!!! Function call: {c.function_call}")
args = type(c.function_call).to_dict(c.function_call).get("args", {})
@@ -708,6 +713,63 @@ class GoogleLLMService(LLMService):
function_name=c.function_call.name,
arguments=args,
)
+ # Handle grounding metadata
+ # It seems only the last chunk that we receive may contain this information
+ # If the response doesn't include groundingMetadata, this means the response wasn't grounded.
+ if chunk.candidates:
+ for candidate in chunk.candidates:
+ # logger.debug(f"candidate received: {candidate}")
+ # Extract grounding metadata
+ grounding_metadata = (
+ {
+ "rendered_content": getattr(
+ getattr(candidate, "grounding_metadata", None),
+ "search_entry_point",
+ None,
+ ).rendered_content
+ if hasattr(
+ getattr(candidate, "grounding_metadata", None),
+ "search_entry_point",
+ )
+ else None,
+ "origins": [
+ {
+ "site_uri": getattr(grounding_chunk.web, "uri", None),
+ "site_title": getattr(
+ grounding_chunk.web, "title", None
+ ),
+ "results": [
+ {
+ "text": getattr(
+ grounding_support.segment, "text", ""
+ ),
+ "confidence": getattr(
+ grounding_support, "confidence_scores", None
+ ),
+ }
+ for grounding_support in getattr(
+ getattr(candidate, "grounding_metadata", None),
+ "grounding_supports",
+ [],
+ )
+ if index
+ in getattr(
+ grounding_support, "grounding_chunk_indices", []
+ )
+ ],
+ }
+ for index, grounding_chunk in enumerate(
+ getattr(
+ getattr(candidate, "grounding_metadata", None),
+ "grounding_chunks",
+ [],
+ )
+ )
+ ],
+ }
+ if getattr(candidate, "grounding_metadata", None)
+ else None
+ )
except Exception as e:
# Google LLMs seem to flag safety issues a lot!
if chunk.candidates[0].finish_reason == 3:
@@ -720,6 +782,14 @@ class GoogleLLMService(LLMService):
except Exception as e:
logger.exception(f"{self} exception: {e}")
finally:
+ if grounding_metadata is not None and isinstance(grounding_metadata, dict):
+ llm_search_frame = LLMSearchResponseFrame(
+ search_result=search_result,
+ origins=grounding_metadata["origins"],
+ rendered_content=grounding_metadata["rendered_content"],
+ )
+ await self.push_frame(llm_search_frame)
+
await self.start_llm_usage_metrics(
LLMTokenUsage(
prompt_tokens=prompt_tokens,
diff --git a/src/pipecat/services/lmnt.py b/src/pipecat/services/lmnt.py
index 633c24265..327901423 100644
--- a/src/pipecat/services/lmnt.py
+++ b/src/pipecat/services/lmnt.py
@@ -113,16 +113,13 @@ class LmntTTSService(TTSService, WebsocketService):
async def _connect(self):
await self._connect_websocket()
- self._receive_task = self.get_event_loop().create_task(
- self._receive_task_handler(self.push_error)
- )
+ self._receive_task = self.create_task(self._receive_task_handler(self.push_error))
async def _disconnect(self):
await self._disconnect_websocket()
if self._receive_task:
- self._receive_task.cancel()
- await self._receive_task
+ await self.cancel_task(self._receive_task)
self._receive_task = None
async def _connect_websocket(self):
@@ -186,10 +183,10 @@ class LmntTTSService(TTSService, WebsocketService):
try:
msg = json.loads(message)
if "error" in msg:
- logger.error(f'{self} error: {msg["error"]}')
+ logger.error(f"{self} error: {msg['error']}")
await self.push_frame(TTSStoppedFrame())
await self.stop_all_metrics()
- await self.push_error(ErrorFrame(f'{self} error: {msg["error"]}'))
+ await self.push_error(ErrorFrame(f"{self} error: {msg['error']}"))
return
except json.JSONDecodeError:
logger.error(f"Invalid JSON message: {message}")
diff --git a/src/pipecat/services/openai.py b/src/pipecat/services/openai.py
index 4a70838f4..1c3977b7f 100644
--- a/src/pipecat/services/openai.py
+++ b/src/pipecat/services/openai.py
@@ -25,10 +25,10 @@ from pipecat.frames.frames import (
LLMFullResponseEndFrame,
LLMFullResponseStartFrame,
LLMMessagesFrame,
+ LLMTextFrame,
LLMUpdateSettingsFrame,
OpenAILLMContextAssistantTimestampFrame,
StartInterruptionFrame,
- TextFrame,
TTSAudioRawFrame,
TTSStartedFrame,
TTSStoppedFrame,
@@ -221,7 +221,7 @@ class BaseOpenAILLMService(LLMService):
)
await self.start_llm_usage_metrics(tokens)
- if len(chunk.choices) == 0:
+ if chunk.choices is None or len(chunk.choices) == 0:
continue
await self.stop_ttfb_metrics()
@@ -258,7 +258,7 @@ class BaseOpenAILLMService(LLMService):
# Keep iterating through the response to collect all the argument fragments
arguments += tool_call.function.arguments
elif chunk.choices[0].delta.content:
- await self.push_frame(TextFrame(chunk.choices[0].delta.content))
+ await self.push_frame(LLMTextFrame(chunk.choices[0].delta.content))
# if we got a function name and arguments, check to see if it's a function with
# a registered handler. If so, run the registered callback, save the result to
diff --git a/src/pipecat/services/openai_realtime_beta/context.py b/src/pipecat/services/openai_realtime_beta/context.py
index 91f01d5a1..da287194a 100644
--- a/src/pipecat/services/openai_realtime_beta/context.py
+++ b/src/pipecat/services/openai_realtime_beta/context.py
@@ -6,10 +6,16 @@
import copy
import json
+from typing import Optional
from loguru import logger
-from pipecat.frames.frames import Frame, LLMMessagesUpdateFrame, LLMSetToolsFrame
+from pipecat.frames.frames import (
+ Frame,
+ FunctionCallResultProperties,
+ LLMMessagesUpdateFrame,
+ LLMSetToolsFrame,
+)
from pipecat.processors.aggregators.openai_llm_context import (
OpenAILLMContext,
OpenAILLMContextFrame,
@@ -174,10 +180,13 @@ class OpenAIRealtimeAssistantContextAggregator(OpenAIAssistantContextAggregator)
if not self._function_call_result:
return
+ properties: Optional[FunctionCallResultProperties] = None
+
self._reset()
try:
run_llm = True
frame = self._function_call_result
+ properties = frame.properties
self._function_call_result = None
if frame.result:
# The "tool_call" message from the LLM that triggered the function call
@@ -211,11 +220,20 @@ class OpenAIRealtimeAssistantContextAggregator(OpenAIAssistantContextAggregator)
await self._user_context_aggregator.push_frame(
RealtimeFunctionCallResultFrame(result_frame=frame)
)
- run_llm = frame.run_llm
+ if properties and properties.run_llm is not None:
+ # If the tool call result has a run_llm property, use it
+ run_llm = properties.run_llm
+ else:
+ # Default behavior is to run the LLM if there are no function calls in progress
+ run_llm = not bool(self._function_calls_in_progress)
if run_llm:
await self._user_context_aggregator.push_context_frame()
+ # Emit the on_context_updated callback once the function call result is added to the context
+ if properties and properties.on_context_updated is not None:
+ await properties.on_context_updated()
+
frame = OpenAILLMContextFrame(self._context)
await self.push_frame(frame)
diff --git a/src/pipecat/services/openai_realtime_beta/openai.py b/src/pipecat/services/openai_realtime_beta/openai.py
index 44057606f..a42ed7b9a 100644
--- a/src/pipecat/services/openai_realtime_beta/openai.py
+++ b/src/pipecat/services/openai_realtime_beta/openai.py
@@ -24,11 +24,11 @@ from pipecat.frames.frames import (
LLMFullResponseStartFrame,
LLMMessagesAppendFrame,
LLMSetToolsFrame,
+ LLMTextFrame,
LLMUpdateSettingsFrame,
StartFrame,
StartInterruptionFrame,
StopInterruptionFrame,
- TextFrame,
TranscriptionFrame,
TTSAudioRawFrame,
TTSStartedFrame,
@@ -167,8 +167,11 @@ class OpenAIRealtimeBetaLLMService(LLMService):
either the wall clock time or the actual audio duration to prevent invalid truncation
requests.
"""
+ if not self._current_audio_response:
+ return
+
# if the bot is still speaking, truncate the last message
- if self._current_audio_response:
+ try:
current = self._current_audio_response
self._current_audio_response = None
@@ -179,6 +182,11 @@ class OpenAIRealtimeBetaLLMService(LLMService):
elapsed_ms = int(time.time() * 1000 - current.start_time_ms)
truncate_ms = min(elapsed_ms, audio_duration_ms)
+ logger.trace(
+ f"Truncating audio: duration={audio_duration_ms}ms, "
+ f"elapsed={elapsed_ms}ms, truncate={truncate_ms}ms"
+ )
+
await self.send_client_event(
events.ConversationItemTruncateEvent(
item_id=current.item_id,
@@ -186,6 +194,9 @@ class OpenAIRealtimeBetaLLMService(LLMService):
audio_end_ms=truncate_ms,
)
)
+ except Exception as e:
+ # Log warning and don't re-raise - allow session to continue
+ logger.warning(f"Audio truncation failed (non-fatal): {e}")
#
# frame processing
@@ -266,7 +277,7 @@ class OpenAIRealtimeBetaLLMService(LLMService):
"OpenAI-Beta": "realtime=v1",
},
)
- self._receive_task = self.get_event_loop().create_task(self._receive_task_handler())
+ self._receive_task = self.create_task(self._receive_task_handler())
except Exception as e:
logger.error(f"{self} initialization error: {e}")
self._websocket = None
@@ -280,11 +291,7 @@ class OpenAIRealtimeBetaLLMService(LLMService):
await self._websocket.close()
self._websocket = None
if self._receive_task:
- self._receive_task.cancel()
- try:
- await asyncio.wait_for(self._receive_task, timeout=1.0)
- except asyncio.TimeoutError:
- logger.warning("Timed out waiting for receive task to finish")
+ await self.cancel_task(self._receive_task, timeout=1.0)
self._receive_task = None
self._disconnecting = False
except Exception as e:
@@ -321,40 +328,32 @@ class OpenAIRealtimeBetaLLMService(LLMService):
#
async def _receive_task_handler(self):
- try:
- async for message in self._websocket:
- evt = events.parse_server_event(message)
- if evt.type == "session.created":
- await self._handle_evt_session_created(evt)
- elif evt.type == "session.updated":
- await self._handle_evt_session_updated(evt)
- elif evt.type == "response.audio.delta":
- await self._handle_evt_audio_delta(evt)
- elif evt.type == "response.audio.done":
- await self._handle_evt_audio_done(evt)
- elif evt.type == "conversation.item.created":
- await self._handle_evt_conversation_item_created(evt)
- elif evt.type == "conversation.item.input_audio_transcription.completed":
- await self.handle_evt_input_audio_transcription_completed(evt)
- elif evt.type == "response.done":
- await self._handle_evt_response_done(evt)
- elif evt.type == "input_audio_buffer.speech_started":
- await self._handle_evt_speech_started(evt)
- elif evt.type == "input_audio_buffer.speech_stopped":
- await self._handle_evt_speech_stopped(evt)
- elif evt.type == "response.audio_transcript.delta":
- await self._handle_evt_audio_transcript_delta(evt)
- elif evt.type == "error":
- await self._handle_evt_error(evt)
- # errors are fatal, so exit the receive loop
- return
-
- else:
- pass
- except asyncio.CancelledError:
- logger.debug("websocket receive task cancelled")
- except Exception as e:
- logger.error(f"{self} exception: {e}")
+ async for message in self._websocket:
+ evt = events.parse_server_event(message)
+ if evt.type == "session.created":
+ await self._handle_evt_session_created(evt)
+ elif evt.type == "session.updated":
+ await self._handle_evt_session_updated(evt)
+ elif evt.type == "response.audio.delta":
+ await self._handle_evt_audio_delta(evt)
+ elif evt.type == "response.audio.done":
+ await self._handle_evt_audio_done(evt)
+ elif evt.type == "conversation.item.created":
+ await self._handle_evt_conversation_item_created(evt)
+ elif evt.type == "conversation.item.input_audio_transcription.completed":
+ await self.handle_evt_input_audio_transcription_completed(evt)
+ elif evt.type == "response.done":
+ await self._handle_evt_response_done(evt)
+ elif evt.type == "input_audio_buffer.speech_started":
+ await self._handle_evt_speech_started(evt)
+ elif evt.type == "input_audio_buffer.speech_stopped":
+ await self._handle_evt_speech_stopped(evt)
+ elif evt.type == "response.audio_transcript.delta":
+ await self._handle_evt_audio_transcript_delta(evt)
+ elif evt.type == "error":
+ await self._handle_evt_error(evt)
+ # errors are fatal, so exit the receive loop
+ return
async def _handle_evt_session_created(self, evt):
# session.created is received right after connecting. Send a message
@@ -458,7 +457,7 @@ class OpenAIRealtimeBetaLLMService(LLMService):
async def _handle_evt_audio_transcript_delta(self, evt):
if evt.delta:
- await self.push_frame(TextFrame(evt.delta))
+ await self.push_frame(LLMTextFrame(evt.delta))
async def _handle_evt_speech_started(self, evt):
await self._truncate_current_audio_response()
diff --git a/src/pipecat/services/playht.py b/src/pipecat/services/playht.py
index a511e2456..07e78ae52 100644
--- a/src/pipecat/services/playht.py
+++ b/src/pipecat/services/playht.py
@@ -165,16 +165,13 @@ class PlayHTTTSService(TTSService, WebsocketService):
async def _connect(self):
await self._connect_websocket()
- self._receive_task = self.get_event_loop().create_task(
- self._receive_task_handler(self.push_error)
- )
+ self._receive_task = self.create_task(self._receive_task_handler(self.push_error))
async def _disconnect(self):
await self._disconnect_websocket()
if self._receive_task:
- self._receive_task.cancel()
- await self._receive_task
+ await self.cancel_task(self._receive_task)
self._receive_task = None
async def _connect_websocket(self):
@@ -269,7 +266,7 @@ class PlayHTTTSService(TTSService, WebsocketService):
self._request_id = None
elif "error" in msg:
logger.error(f"{self} error: {msg}")
- await self.push_error(ErrorFrame(f'{self} error: {msg["error"]}'))
+ await self.push_error(ErrorFrame(f"{self} error: {msg['error']}"))
except json.JSONDecodeError:
logger.error(f"Invalid JSON message: {message}")
diff --git a/src/pipecat/services/riva.py b/src/pipecat/services/riva.py
index 662ac21f5..cac1946d5 100644
--- a/src/pipecat/services/riva.py
+++ b/src/pipecat/services/riva.py
@@ -202,8 +202,8 @@ class ParakeetSTTService(STTService):
async def start(self, frame: StartFrame):
await super().start(frame)
- self._thread_task = self.get_event_loop().create_task(self._thread_task_handler())
- self._response_task = self.get_event_loop().create_task(self._response_task_handler())
+ self._thread_task = self.create_task(self._thread_task_handler())
+ self._response_task = self.create_task(self._response_task_handler())
self._response_queue = asyncio.Queue()
async def stop(self, frame: EndFrame):
@@ -215,10 +215,8 @@ class ParakeetSTTService(STTService):
await self._stop_tasks()
async def _stop_tasks(self):
- self._thread_task.cancel()
- await self._thread_task
- self._response_task.cancel()
- await self._response_task
+ await self.cancel_task(self._thread_task)
+ await self.cancel_task(self._response_task)
def _response_handler(self):
responses = self._asr_service.streaming_response_generator(
@@ -238,7 +236,7 @@ class ParakeetSTTService(STTService):
await asyncio.to_thread(self._response_handler)
except asyncio.CancelledError:
self._thread_running = False
- pass
+ raise
async def _handle_response(self, response):
for result in response.results:
@@ -260,11 +258,8 @@ class ParakeetSTTService(STTService):
async def _response_task_handler(self):
while True:
- try:
- response = await self._response_queue.get()
- await self._handle_response(response)
- except asyncio.CancelledError:
- break
+ response = await self._response_queue.get()
+ await self._handle_response(response)
async def run_stt(self, audio: bytes) -> AsyncGenerator[Frame, None]:
await self._queue.put(audio)
diff --git a/src/pipecat/services/simli.py b/src/pipecat/services/simli.py
index be7edfdbd..5a6aecbe4 100644
--- a/src/pipecat/services/simli.py
+++ b/src/pipecat/services/simli.py
@@ -49,45 +49,33 @@ class SimliVideoService(FrameProcessor):
async def _start_connection(self):
await self._simli_client.Initialize()
# Create task to consume and process audio and video
- self._audio_task = asyncio.create_task(self._consume_and_process_audio())
- self._video_task = asyncio.create_task(self._consume_and_process_video())
+ self._audio_task = self.create_task(self._consume_and_process_audio())
+ self._video_task = self.create_task(self._consume_and_process_video())
async def _consume_and_process_audio(self):
- try:
- await self._pipecat_resampler_event.wait()
- async for audio_frame in self._simli_client.getAudioStreamIterator():
- resampled_frames = self._pipecat_resampler.resample(audio_frame)
- for resampled_frame in resampled_frames:
- await self.push_frame(
- TTSAudioRawFrame(
- audio=resampled_frame.to_ndarray().tobytes(),
- sample_rate=self._pipecat_resampler.rate,
- num_channels=1,
- ),
- )
- except Exception as e:
- logger.exception(f"{self} exception: {e}")
- except asyncio.CancelledError:
- pass
+ await self._pipecat_resampler_event.wait()
+ async for audio_frame in self._simli_client.getAudioStreamIterator():
+ resampled_frames = self._pipecat_resampler.resample(audio_frame)
+ for resampled_frame in resampled_frames:
+ await self.push_frame(
+ TTSAudioRawFrame(
+ audio=resampled_frame.to_ndarray().tobytes(),
+ sample_rate=self._pipecat_resampler.rate,
+ num_channels=1,
+ ),
+ )
async def _consume_and_process_video(self):
- try:
- await self._pipecat_resampler_event.wait()
- async for video_frame in self._simli_client.getVideoStreamIterator(
- targetFormat="rgb24"
- ):
- # Process the video frame
- convertedFrame: OutputImageRawFrame = OutputImageRawFrame(
- image=video_frame.to_rgb().to_image().tobytes(),
- size=(video_frame.width, video_frame.height),
- format="RGB",
- )
- convertedFrame.pts = video_frame.pts
- await self.push_frame(convertedFrame)
- except Exception as e:
- logger.exception(f"{self} exception: {e}")
- except asyncio.CancelledError:
- pass
+ await self._pipecat_resampler_event.wait()
+ async for video_frame in self._simli_client.getVideoStreamIterator(targetFormat="rgb24"):
+ # Process the video frame
+ convertedFrame: OutputImageRawFrame = OutputImageRawFrame(
+ image=video_frame.to_rgb().to_image().tobytes(),
+ size=(video_frame.width, video_frame.height),
+ format="RGB",
+ )
+ convertedFrame.pts = video_frame.pts
+ await self.push_frame(convertedFrame)
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
@@ -128,8 +116,6 @@ class SimliVideoService(FrameProcessor):
async def _stop(self):
await self._simli_client.stop()
if self._audio_task:
- self._audio_task.cancel()
- await self._audio_task
+ await self.cancel_task(self._audio_task)
if self._video_task:
- self._video_task.cancel()
- await self._video_task
+ await self.cancel_task(self._video_task)
diff --git a/src/pipecat/services/tavus.py b/src/pipecat/services/tavus.py
index 32752205c..b1dfef9e4 100644
--- a/src/pipecat/services/tavus.py
+++ b/src/pipecat/services/tavus.py
@@ -75,6 +75,14 @@ class TavusVideoService(AIService):
logger.debug(f"TavusVideoService persona grabbed {response_json}")
return response_json["persona_name"]
+ async def stop(self, frame: EndFrame):
+ await super().stop(frame)
+ await self._end_conversation()
+
+ async def cancel(self, frame: CancelFrame):
+ await super().cancel(frame)
+ await self._end_conversation()
+
async def _end_conversation(self) -> None:
url = f"https://tavusapi.com/v2/conversations/{self._conversation_id}/end"
headers = {"Content-Type": "application/json", "x-api-key": self._api_key}
@@ -88,7 +96,7 @@ class TavusVideoService(AIService):
if not done:
audio = resample_audio(audio, original_sample_rate, 16000)
audio_base64 = base64.b64encode(audio).decode("utf-8")
- logger.trace(f"TavusVideoService sending {len(audio)} bytes")
+ logger.trace(f"{self}: sending {len(audio)} bytes")
await self._send_audio_message(audio_base64, done=done)
async def process_frame(self, frame: Frame, direction: FrameDirection):
@@ -105,8 +113,6 @@ class TavusVideoService(AIService):
await self.stop_processing_metrics()
elif isinstance(frame, StartInterruptionFrame):
await self._send_interrupt_message()
- elif isinstance(frame, (EndFrame, CancelFrame)):
- await self._end_conversation()
else:
await self.push_frame(frame, direction)
diff --git a/src/pipecat/services/websocket_service.py b/src/pipecat/services/websocket_service.py
index 365f5a7c8..7e480f2b7 100644
--- a/src/pipecat/services/websocket_service.py
+++ b/src/pipecat/services/websocket_service.py
@@ -85,10 +85,6 @@ class WebsocketService(ABC):
await self._receive_messages()
logger.debug(f"{self} connection established successfully")
retry_count = 0 # Reset counter on successful message receive
-
- except asyncio.CancelledError:
- break
-
except Exception as e:
retry_count += 1
if retry_count >= MAX_RETRIES:
diff --git a/src/pipecat/tests/__init__.py b/src/pipecat/tests/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/src/pipecat/tests/utils.py b/src/pipecat/tests/utils.py
new file mode 100644
index 000000000..3b6a0d009
--- /dev/null
+++ b/src/pipecat/tests/utils.py
@@ -0,0 +1,120 @@
+#
+# Copyright (c) 2024-2025 Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import asyncio
+from dataclasses import dataclass
+from typing import Awaitable, Callable, Sequence, Tuple
+
+from pipecat.clocks.system_clock import SystemClock
+from pipecat.frames.frames import (
+ ControlFrame,
+ Frame,
+ HeartbeatFrame,
+ StartFrame,
+)
+from pipecat.observers.base_observer import BaseObserver
+from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
+
+
+@dataclass
+class EndTestFrame(ControlFrame):
+ pass
+
+
+class HeartbeatsObserver(BaseObserver):
+ def __init__(
+ self,
+ *,
+ target: FrameProcessor,
+ heartbeat_callback: Callable[[FrameProcessor, HeartbeatFrame], Awaitable[None]],
+ ):
+ self._target = target
+ self._callback = heartbeat_callback
+
+ async def on_push_frame(
+ self,
+ src: FrameProcessor,
+ dst: FrameProcessor,
+ frame: Frame,
+ direction: FrameDirection,
+ timestamp: int,
+ ):
+ if src == self._target and isinstance(frame, HeartbeatFrame):
+ await self._callback(self._target, frame)
+
+
+class QueuedFrameProcessor(FrameProcessor):
+ def __init__(self, queue: asyncio.Queue, ignore_start: bool = True):
+ super().__init__()
+ self._queue = queue
+ self._ignore_start = ignore_start
+
+ async def process_frame(self, frame: Frame, direction: FrameDirection):
+ await super().process_frame(frame, direction)
+ if self._ignore_start and isinstance(frame, StartFrame):
+ return
+ await self._queue.put(frame)
+
+
+async def run_test(
+ processor: FrameProcessor,
+ frames_to_send: Sequence[Frame],
+ expected_down_frames: Sequence[type],
+ expected_up_frames: Sequence[type] = [],
+) -> Tuple[Sequence[Frame], Sequence[Frame]]:
+ received_up = asyncio.Queue()
+ received_down = asyncio.Queue()
+ up_processor = QueuedFrameProcessor(received_up)
+ down_processor = QueuedFrameProcessor(received_down)
+
+ up_processor.link(processor)
+ processor.link(down_processor)
+
+ await processor.queue_frame(StartFrame(clock=SystemClock()))
+
+ for frame in frames_to_send:
+ await processor.process_frame(frame, FrameDirection.DOWNSTREAM)
+
+ await processor.queue_frame(EndTestFrame())
+ await processor.queue_frame(EndTestFrame(), FrameDirection.UPSTREAM)
+
+ #
+ # Down frames
+ #
+ received_down_frames: Sequence[Frame] = []
+ running = True
+ while running:
+ frame = await received_down.get()
+ running = not isinstance(frame, EndTestFrame)
+ if running:
+ received_down_frames.append(frame)
+
+ print("received DOWN frames =", received_down_frames)
+
+ assert len(received_down_frames) == len(expected_down_frames)
+
+ for real, expected in zip(received_down_frames, expected_down_frames):
+ assert isinstance(real, expected)
+
+ #
+ # Up frames
+ #
+ received_up_frames: Sequence[Frame] = []
+ running = True
+ while running:
+ frame = await received_up.get()
+ running = not isinstance(frame, EndTestFrame)
+ if running:
+ received_up_frames.append(frame)
+
+ print("received UP frames =", received_up_frames)
+
+ assert len(received_up_frames) == len(expected_up_frames)
+
+ for real, expected in zip(received_up_frames, expected_up_frames):
+ assert isinstance(real, expected)
+
+ return (received_down_frames, received_up_frames)
diff --git a/src/pipecat/transports/base_input.py b/src/pipecat/transports/base_input.py
index b0d6a7953..192101e62 100644
--- a/src/pipecat/transports/base_input.py
+++ b/src/pipecat/transports/base_input.py
@@ -35,7 +35,9 @@ class BaseInputTransport(FrameProcessor):
self._params = params
- self._executor = ThreadPoolExecutor(max_workers=5)
+ # We read audio from a single queue one at a time and we then run VAD in
+ # a thread. Therefore, only one thread should be necessary.
+ self._executor = ThreadPoolExecutor(max_workers=1)
# Task to process incoming audio (VAD) and push audio frames downstream
# if passthrough is enabled.
@@ -48,13 +50,12 @@ class BaseInputTransport(FrameProcessor):
# Create audio input queue and task if needed.
if self._params.audio_in_enabled or self._params.vad_enabled:
self._audio_in_queue = asyncio.Queue()
- self._audio_task = self.get_event_loop().create_task(self._audio_task_handler())
+ self._audio_task = self.create_task(self._audio_task_handler())
async def stop(self, frame: EndFrame):
# Cancel and wait for the audio input task to finish.
if self._audio_task and (self._params.audio_in_enabled or self._params.vad_enabled):
- self._audio_task.cancel()
- await self._audio_task
+ await self.cancel_task(self._audio_task)
self._audio_task = None
# Stop audio filter.
if self._params.audio_in_filter:
@@ -63,8 +64,7 @@ class BaseInputTransport(FrameProcessor):
async def cancel(self, frame: CancelFrame):
# Cancel and wait for the audio input task to finish.
if self._audio_task and (self._params.audio_in_enabled or self._params.vad_enabled):
- self._audio_task.cancel()
- await self._audio_task
+ await self.cancel_task(self._audio_task)
self._audio_task = None
def vad_analyzer(self) -> VADAnalyzer | None:
@@ -138,17 +138,19 @@ class BaseInputTransport(FrameProcessor):
# Audio input
#
- async def _vad_analyze(self, audio_frames: bytes) -> VADState:
+ async def _vad_analyze(self, audio_frame: InputAudioRawFrame) -> VADState:
state = VADState.QUIET
vad_analyzer = self.vad_analyzer()
if vad_analyzer:
+ logger.trace(f"{self}: analyzing VAD on {audio_frame}")
state = await self.get_event_loop().run_in_executor(
- self._executor, vad_analyzer.analyze_audio, audio_frames
+ self._executor, vad_analyzer.analyze_audio, audio_frame.audio
)
+ logger.trace(f"{self}: done analyzing VAD on {audio_frame}")
return state
- async def _handle_vad(self, audio_frames: bytes, vad_state: VADState):
- new_vad_state = await self._vad_analyze(audio_frames)
+ async def _handle_vad(self, audio_frame: InputAudioRawFrame, vad_state: VADState):
+ new_vad_state = await self._vad_analyze(audio_frame)
if (
new_vad_state != vad_state
and new_vad_state != VADState.STARTING
@@ -169,27 +171,22 @@ class BaseInputTransport(FrameProcessor):
async def _audio_task_handler(self):
vad_state: VADState = VADState.QUIET
while True:
- try:
- frame: InputAudioRawFrame = await self._audio_in_queue.get()
+ frame: InputAudioRawFrame = await self._audio_in_queue.get()
- audio_passthrough = True
+ audio_passthrough = True
- # If an audio filter is available, run it before VAD.
- if self._params.audio_in_filter:
- frame.audio = await self._params.audio_in_filter.filter(frame.audio)
+ # If an audio filter is available, run it before VAD.
+ if self._params.audio_in_filter:
+ frame.audio = await self._params.audio_in_filter.filter(frame.audio)
- # Check VAD and push event if necessary. We just care about
- # changes from QUIET to SPEAKING and vice versa.
- if self._params.vad_enabled:
- vad_state = await self._handle_vad(frame.audio, vad_state)
- audio_passthrough = self._params.vad_audio_passthrough
+ # Check VAD and push event if necessary. We just care about
+ # changes from QUIET to SPEAKING and vice versa.
+ if self._params.vad_enabled:
+ vad_state = await self._handle_vad(frame, vad_state)
+ audio_passthrough = self._params.vad_audio_passthrough
- # Push audio downstream if passthrough.
- if audio_passthrough:
- await self.push_frame(frame)
+ # Push audio downstream if passthrough.
+ if audio_passthrough:
+ await self.push_frame(frame)
- self._audio_in_queue.task_done()
- except asyncio.CancelledError:
- break
- except Exception as e:
- logger.exception(f"{self} error reading audio frames: {e}")
+ self._audio_in_queue.task_done()
diff --git a/src/pipecat/transports/base_output.py b/src/pipecat/transports/base_output.py
index ef073e22a..5182fc790 100644
--- a/src/pipecat/transports/base_output.py
+++ b/src/pipecat/transports/base_output.py
@@ -35,6 +35,7 @@ from pipecat.frames.frames import (
)
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.transports.base_transport import TransportParams
+from pipecat.utils.asyncio import wait_for_task
from pipecat.utils.time import nanoseconds_to_seconds
@@ -87,9 +88,9 @@ class BaseOutputTransport(FrameProcessor):
# for these tasks before cancelling the camera and audio tasks below
# because they might be still rendering.
if self._sink_task:
- await self._sink_task
+ await wait_for_task(self._sink_task)
if self._sink_clock_task:
- await self._sink_clock_task
+ await wait_for_task(self._sink_clock_task)
# We can now cancel the camera task.
await self._cancel_camera_task()
@@ -217,22 +218,19 @@ class BaseOutputTransport(FrameProcessor):
#
def _create_sink_tasks(self):
- loop = self.get_event_loop()
self._sink_queue = asyncio.Queue()
- self._sink_task = loop.create_task(self._sink_task_handler())
self._sink_clock_queue = asyncio.PriorityQueue()
- self._sink_clock_task = loop.create_task(self._sink_clock_task_handler())
+ self._sink_task = self.create_task(self._sink_task_handler())
+ self._sink_clock_task = self.create_task(self._sink_clock_task_handler())
async def _cancel_sink_tasks(self):
# Stop sink tasks.
if self._sink_task:
- self._sink_task.cancel()
- await self._sink_task
+ await self.cancel_task(self._sink_task)
self._sink_task = None
# Stop sink clock tasks.
if self._sink_clock_task:
- self._sink_clock_task.cancel()
- await self._sink_clock_task
+ await self.cancel_task(self._sink_clock_task)
self._sink_clock_task = None
async def _sink_frame_handler(self, frame: Frame):
@@ -269,7 +267,7 @@ class BaseOutputTransport(FrameProcessor):
self._sink_clock_queue.task_done()
except asyncio.CancelledError:
- break
+ raise
except Exception as e:
logger.exception(f"{self} error processing sink clock queue: {e}")
@@ -317,49 +315,42 @@ class BaseOutputTransport(FrameProcessor):
return without_mixer(vad_stop_secs)
async def _sink_task_handler(self):
- try:
- async for frame in self._next_frame():
- # Notify the bot started speaking upstream if necessary and that
- # it's actually speaking.
- if isinstance(frame, TTSAudioRawFrame):
- await self._bot_started_speaking()
- await self.push_frame(BotSpeakingFrame())
- await self.push_frame(BotSpeakingFrame(), FrameDirection.UPSTREAM)
+ async for frame in self._next_frame():
+ # Notify the bot started speaking upstream if necessary and that
+ # it's actually speaking.
+ if isinstance(frame, TTSAudioRawFrame):
+ await self._bot_started_speaking()
+ await self.push_frame(BotSpeakingFrame())
+ await self.push_frame(BotSpeakingFrame(), FrameDirection.UPSTREAM)
- # No need to push EndFrame, it's pushed from process_frame().
- if isinstance(frame, EndFrame):
- break
+ # No need to push EndFrame, it's pushed from process_frame().
+ if isinstance(frame, EndFrame):
+ break
- # Handle frame.
- await self._sink_frame_handler(frame)
+ # Handle frame.
+ await self._sink_frame_handler(frame)
- # Also, push frame downstream in case anyone else needs it.
- await self.push_frame(frame)
+ # Also, push frame downstream in case anyone else needs it.
+ await self.push_frame(frame)
- # Send audio.
- if isinstance(frame, OutputAudioRawFrame):
- await self.write_raw_audio_frames(frame.audio)
- except asyncio.CancelledError:
- pass
- except Exception as e:
- logger.exception(f"{self} error writing to microphone: {e}")
+ # Send audio.
+ if isinstance(frame, OutputAudioRawFrame):
+ await self.write_raw_audio_frames(frame.audio)
#
# Camera task
#
def _create_camera_task(self):
- loop = self.get_event_loop()
# Create camera output queue and task if needed.
if self._params.camera_out_enabled:
self._camera_out_queue = asyncio.Queue()
- self._camera_out_task = loop.create_task(self._camera_out_task_handler())
+ self._camera_out_task = self.create_task(self._camera_out_task_handler())
async def _cancel_camera_task(self):
# Stop camera output task.
if self._camera_out_task and self._params.camera_out_enabled:
- self._camera_out_task.cancel()
- await self._camera_out_task
+ await self.cancel_task(self._camera_out_task)
self._camera_out_task = None
async def _draw_image(self, frame: OutputImageRawFrame):
@@ -387,19 +378,14 @@ class BaseOutputTransport(FrameProcessor):
self._camera_out_frame_duration = 1 / self._params.camera_out_framerate
self._camera_out_frame_reset = self._camera_out_frame_duration * 5
while True:
- try:
- if self._params.camera_out_is_live:
- await self._camera_out_is_live_handler()
- elif self._camera_images:
- image = next(self._camera_images)
- await self._draw_image(image)
- await asyncio.sleep(self._camera_out_frame_duration)
- else:
- await asyncio.sleep(self._camera_out_frame_duration)
- except asyncio.CancelledError:
- break
- except Exception as e:
- logger.exception(f"{self} error writing to camera: {e}")
+ if self._params.camera_out_is_live:
+ await self._camera_out_is_live_handler()
+ elif self._camera_images:
+ image = next(self._camera_images)
+ await self._draw_image(image)
+ await asyncio.sleep(self._camera_out_frame_duration)
+ else:
+ await asyncio.sleep(self._camera_out_frame_duration)
async def _camera_out_is_live_handler(self):
image = await self._camera_out_queue.get()
diff --git a/src/pipecat/transports/base_transport.py b/src/pipecat/transports/base_transport.py
index bcc13c4fd..fd14033f2 100644
--- a/src/pipecat/transports/base_transport.py
+++ b/src/pipecat/transports/base_transport.py
@@ -16,6 +16,7 @@ from pipecat.audio.filters.base_audio_filter import BaseAudioFilter
from pipecat.audio.mixers.base_audio_mixer import BaseAudioMixer
from pipecat.audio.vad.vad_analyzer import VADAnalyzer
from pipecat.processors.frame_processor import FrameProcessor
+from pipecat.utils.utils import obj_count, obj_id
class TransportParams(BaseModel):
@@ -46,15 +47,27 @@ class TransportParams(BaseModel):
class BaseTransport(ABC):
def __init__(
self,
- input_name: str | None = None,
- output_name: str | None = None,
- loop: asyncio.AbstractEventLoop | None = None,
+ *,
+ name: Optional[str] = None,
+ input_name: Optional[str] = None,
+ output_name: Optional[str] = None,
+ loop: Optional[asyncio.AbstractEventLoop] = None,
):
+ self._id: int = obj_id()
+ self._name = name or f"{self.__class__.__name__}#{obj_count(self)}"
self._input_name = input_name
self._output_name = output_name
self._loop = loop or asyncio.get_running_loop()
self._event_handlers: dict = {}
+ @property
+ def id(self) -> int:
+ return self._id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
@abstractmethod
def input(self) -> FrameProcessor:
raise NotImplementedError
@@ -89,3 +102,6 @@ class BaseTransport(ABC):
handler(self, *args, **kwargs)
except Exception as e:
logger.exception(f"Exception in event handler {event_name}: {e}")
+
+ def __str__(self):
+ return self.name
diff --git a/src/pipecat/transports/local/audio.py b/src/pipecat/transports/local/audio.py
index c042c3fde..52681900d 100644
--- a/src/pipecat/transports/local/audio.py
+++ b/src/pipecat/transports/local/audio.py
@@ -69,7 +69,9 @@ class LocalAudioOutputTransport(BaseOutputTransport):
def __init__(self, py_audio: pyaudio.PyAudio, params: TransportParams):
super().__init__(params)
- self._executor = ThreadPoolExecutor(max_workers=5)
+ # We only write audio frames from a single task, so only one thread
+ # should be necessary.
+ self._executor = ThreadPoolExecutor(max_workers=1)
self._out_stream = py_audio.open(
format=py_audio.get_format_from_width(2),
@@ -96,6 +98,7 @@ class LocalAudioOutputTransport(BaseOutputTransport):
class LocalAudioTransport(BaseTransport):
def __init__(self, params: TransportParams):
+ super().__init__()
self._params = params
self._pyaudio = pyaudio.PyAudio()
diff --git a/src/pipecat/transports/local/tk.py b/src/pipecat/transports/local/tk.py
index 23c70868c..44ca45b33 100644
--- a/src/pipecat/transports/local/tk.py
+++ b/src/pipecat/transports/local/tk.py
@@ -77,7 +77,9 @@ class TkOutputTransport(BaseOutputTransport):
def __init__(self, tk_root: tk.Tk, py_audio: pyaudio.PyAudio, params: TransportParams):
super().__init__(params)
- self._executor = ThreadPoolExecutor(max_workers=5)
+ # We only write audio frames from a single task, so only one thread
+ # should be necessary.
+ self._executor = ThreadPoolExecutor(max_workers=1)
self._out_stream = py_audio.open(
format=py_audio.get_format_from_width(2),
@@ -125,6 +127,7 @@ class TkOutputTransport(BaseOutputTransport):
class TkLocalTransport(BaseTransport):
def __init__(self, tk_root: tk.Tk, params: TransportParams):
+ super().__init__()
self._tk_root = tk_root
self._params = params
self._pyaudio = pyaudio.PyAudio()
diff --git a/src/pipecat/transports/network/fastapi_websocket.py b/src/pipecat/transports/network/fastapi_websocket.py
index fc5804611..bcfbbc8cd 100644
--- a/src/pipecat/transports/network/fastapi_websocket.py
+++ b/src/pipecat/transports/network/fastapi_websocket.py
@@ -16,6 +16,8 @@ from loguru import logger
from pydantic import BaseModel
from pipecat.frames.frames import (
+ CancelFrame,
+ EndFrame,
Frame,
InputAudioRawFrame,
OutputAudioRawFrame,
@@ -27,6 +29,7 @@ from pipecat.serializers.base_serializer import FrameSerializer, FrameSerializer
from pipecat.transports.base_input import BaseInputTransport
from pipecat.transports.base_output import BaseOutputTransport
from pipecat.transports.base_transport import BaseTransport, TransportParams
+from pipecat.utils.asyncio import cancel_task
try:
from fastapi import WebSocket
@@ -68,11 +71,17 @@ class FastAPIWebsocketInputTransport(BaseInputTransport):
async def start(self, frame: StartFrame):
await super().start(frame)
if self._params.session_timeout:
- self._monitor_websocket_task = self.get_event_loop().create_task(
- self._monitor_websocket()
- )
+ self._monitor_websocket_task = self.create_task(self._monitor_websocket())
await self._callbacks.on_client_connected(self._websocket)
- self._receive_task = self.get_event_loop().create_task(self._receive_messages())
+ self._receive_task = self.create_task(self._receive_messages())
+
+ async def stop(self, frame: EndFrame):
+ await super().stop(frame)
+ await cancel_task(self._receive_task)
+
+ async def cancel(self, frame: CancelFrame):
+ await super().cancel(frame)
+ await cancel_task(self._receive_task)
def _iter_data(self) -> typing.AsyncIterator[bytes | str]:
if self._params.serializer.type == FrameSerializerType.BINARY:
@@ -96,11 +105,8 @@ class FastAPIWebsocketInputTransport(BaseInputTransport):
async def _monitor_websocket(self):
"""Wait for self._params.session_timeout seconds, if the websocket is still open, trigger timeout event."""
- try:
- await asyncio.sleep(self._params.session_timeout)
- await self._callbacks.on_session_timeout(self._websocket)
- except asyncio.CancelledError:
- logger.info(f"Monitoring task cancelled for: {self._websocket}")
+ await asyncio.sleep(self._params.session_timeout)
+ await self._callbacks.on_session_timeout(self._websocket)
class FastAPIWebsocketOutputTransport(BaseOutputTransport):
diff --git a/src/pipecat/transports/network/websocket_server.py b/src/pipecat/transports/network/websocket_server.py
index 9bd344cb1..3eb06dc6b 100644
--- a/src/pipecat/transports/network/websocket_server.py
+++ b/src/pipecat/transports/network/websocket_server.py
@@ -71,17 +71,16 @@ class WebsocketServerInputTransport(BaseInputTransport):
async def start(self, frame: StartFrame):
await super().start(frame)
- self._server_task = self.get_event_loop().create_task(self._server_task_handler())
+ self._server_task = self.create_task(self._server_task_handler())
async def stop(self, frame: EndFrame):
await super().stop(frame)
self._stop_server_event.set()
- await self._server_task
+ await self.wait_for_task(self._server_task)
async def cancel(self, frame: CancelFrame):
await super().cancel(frame)
- self._stop_server_event.set()
- await self._server_task
+ await self.cancel_task(self._server_task)
async def _server_task_handler(self):
logger.info(f"Starting websocket server on {self._host}:{self._port}")
@@ -131,6 +130,7 @@ class WebsocketServerInputTransport(BaseInputTransport):
await self._callbacks.on_session_timeout(websocket)
except asyncio.CancelledError:
logger.info(f"Monitoring task cancelled for: {websocket.remote_address}")
+ raise
class WebsocketServerOutputTransport(BaseOutputTransport):
diff --git a/src/pipecat/transports/services/daily.py b/src/pipecat/transports/services/daily.py
index d3a56d5a7..099b78aa8 100644
--- a/src/pipecat/transports/services/daily.py
+++ b/src/pipecat/transports/services/daily.py
@@ -27,6 +27,7 @@ from pipecat.audio.vad.vad_analyzer import VADAnalyzer, VADParams
from pipecat.frames.frames import (
CancelFrame,
EndFrame,
+ ErrorFrame,
Frame,
InputAudioRawFrame,
InterimTranscriptionFrame,
@@ -45,6 +46,7 @@ from pipecat.transcriptions.language import Language
from pipecat.transports.base_input import BaseInputTransport
from pipecat.transports.base_output import BaseOutputTransport
from pipecat.transports.base_transport import BaseTransport, TransportParams
+from pipecat.utils.asyncio import cancel_task, create_task
try:
from daily import CallClient, Daily, EventHandler
@@ -138,7 +140,6 @@ class DailyCallbacks(BaseModel):
on_dialout_stopped: Callable[[Any], Awaitable[None]]
on_dialout_error: Callable[[Any], Awaitable[None]]
on_dialout_warning: Callable[[Any], Awaitable[None]]
- on_first_participant_joined: Callable[[Mapping[str, Any]], Awaitable[None]]
on_participant_joined: Callable[[Mapping[str, Any]], Awaitable[None]]
on_participant_left: Callable[[Mapping[str, Any], str], Awaitable[None]]
on_participant_updated: Callable[[Mapping[str, Any]], Awaitable[None]]
@@ -179,6 +180,7 @@ class DailyTransportClient(EventHandler):
params: DailyParams,
callbacks: DailyCallbacks,
loop: asyncio.AbstractEventLoop,
+ transport_name: str,
):
super().__init__()
@@ -192,17 +194,20 @@ class DailyTransportClient(EventHandler):
self._params: DailyParams = params
self._callbacks = callbacks
self._loop = loop
+ self._transport_name = transport_name
self._participant_id: str = ""
self._video_renderers = {}
self._transcription_ids = []
self._transcription_status = None
- self._other_participant_has_joined = False
self._joined = False
+ self._joined_event = asyncio.Event()
self._leave_counter = 0
- self._executor = ThreadPoolExecutor(max_workers=5)
+ # We use the executor to cleanup the client. We just do it from one
+ # place, so only one thread is really needed.
+ self._executor = ThreadPoolExecutor(max_workers=1)
self._client: CallClient = CallClient(event_handler=self)
@@ -215,7 +220,11 @@ class DailyTransportClient(EventHandler):
# future) we will deadlock because completions use event handlers (which
# are holding the GIL).
self._callback_queue = asyncio.Queue()
- self._callback_task = self._loop.create_task(self._callback_task_handler())
+ self._callback_task = create_task(
+ self._loop,
+ self._callback_task_handler(),
+ f"{self._transport_name}::DailyTransportClient::callback_task",
+ )
self._camera: VirtualCameraDevice | None = None
if self._params.camera_out_enabled:
@@ -343,6 +352,8 @@ class DailyTransportClient(EventHandler):
await self._start_transcription()
await self._callbacks.on_joined(data)
+
+ self._joined_event.set()
else:
error_msg = f"Error joining {self._room_url}: {error}"
logger.error(error_msg)
@@ -431,6 +442,7 @@ class DailyTransportClient(EventHandler):
return
self._joined = False
+ self._joined_event.clear()
logger.info(f"Leaving {self._room_url}")
@@ -466,9 +478,12 @@ class DailyTransportClient(EventHandler):
return await asyncio.wait_for(future, timeout=10)
async def cleanup(self):
+ if self._callback_task:
+ await cancel_task(self._callback_task)
+ self._callback_task = None
+ # Make sure we don't block the event loop in case `client.release()`
+ # takes extra time.
await self._loop.run_in_executor(self._executor, self._cleanup)
- self._callback_task.cancel()
- await self._callback_task
def _cleanup(self):
if self._client:
@@ -616,19 +631,9 @@ class DailyTransportClient(EventHandler):
self._call_async_callback(self._callbacks.on_dialout_warning, data)
def on_participant_joined(self, participant):
- id = participant["id"]
- logger.info(f"Participant joined {id}")
-
- if not self._other_participant_has_joined:
- self._other_participant_has_joined = True
- self._call_async_callback(self._callbacks.on_first_participant_joined, participant)
-
self._call_async_callback(self._callbacks.on_participant_joined, participant)
def on_participant_left(self, participant, reason):
- id = participant["id"]
- logger.info(f"Participant left {id}")
-
self._call_async_callback(self._callbacks.on_participant_left, participant, reason)
def on_participant_updated(self, participant):
@@ -682,11 +687,10 @@ class DailyTransportClient(EventHandler):
async def _callback_task_handler(self):
while True:
- try:
- (callback, *args) = await self._callback_queue.get()
- await callback(*args)
- except asyncio.CancelledError:
- break
+ # Wait to process any callback until we are joined.
+ await self._joined_event.wait()
+ (callback, *args) = await self._callback_queue.get()
+ await callback(*args)
class DailyInputTransport(BaseInputTransport):
@@ -716,7 +720,7 @@ class DailyInputTransport(BaseInputTransport):
# Create audio task. It reads audio frames from Daily and push them
# internally for VAD processing.
if self._params.audio_in_enabled or self._params.vad_enabled:
- self._audio_in_task = self.get_event_loop().create_task(self._audio_in_task_handler())
+ self._audio_in_task = self.create_task(self._audio_in_task_handler())
async def stop(self, frame: EndFrame):
# Parent stop.
@@ -725,8 +729,7 @@ class DailyInputTransport(BaseInputTransport):
await self._client.leave()
# Stop audio thread.
if self._audio_in_task and (self._params.audio_in_enabled or self._params.vad_enabled):
- self._audio_in_task.cancel()
- await self._audio_in_task
+ await self.cancel_task(self._audio_in_task)
self._audio_in_task = None
async def cancel(self, frame: CancelFrame):
@@ -736,8 +739,7 @@ class DailyInputTransport(BaseInputTransport):
await self._client.leave()
# Stop audio thread.
if self._audio_in_task and (self._params.audio_in_enabled or self._params.vad_enabled):
- self._audio_in_task.cancel()
- await self._audio_in_task
+ await self.cancel_task(self._audio_in_task)
self._audio_in_task = None
async def cleanup(self):
@@ -774,12 +776,9 @@ class DailyInputTransport(BaseInputTransport):
async def _audio_in_task_handler(self):
while True:
- try:
- frame = await self._client.read_next_audio_frame()
- if frame:
- await self.push_audio_frame(frame)
- except asyncio.CancelledError:
- break
+ frame = await self._client.read_next_audio_frame()
+ if frame:
+ await self.push_audio_frame(frame)
#
# Camera in
@@ -896,7 +895,6 @@ class DailyTransport(BaseTransport):
on_dialout_stopped=self._on_dialout_stopped,
on_dialout_error=self._on_dialout_error,
on_dialout_warning=self._on_dialout_warning,
- on_first_participant_joined=self._on_first_participant_joined,
on_participant_joined=self._on_participant_joined,
on_participant_left=self._on_participant_left,
on_participant_updated=self._on_participant_updated,
@@ -908,15 +906,18 @@ class DailyTransport(BaseTransport):
self._params = params
self._client = DailyTransportClient(
- room_url, token, bot_name, params, callbacks, self._loop
+ room_url, token, bot_name, params, callbacks, self._loop, self.name
)
self._input: DailyInputTransport | None = None
self._output: DailyOutputTransport | None = None
+ self._other_participant_has_joined = False
+
# Register supported handlers. The user will only be able to register
# these handlers.
self._register_event_handler("on_joined")
self._register_event_handler("on_left")
+ self._register_event_handler("on_error")
self._register_event_handler("on_app_message")
self._register_event_handler("on_call_state_updated")
self._register_event_handler("on_dialin_connected")
@@ -1031,9 +1032,17 @@ class DailyTransport(BaseTransport):
await self._call_event_handler("on_left")
async def _on_error(self, error):
- # TODO(aleix): Report error to input/output transports. The one managing
- # the client should report the error.
- pass
+ await self._call_event_handler("on_error", error)
+ # Push error frame to notify the pipeline
+ error_frame = ErrorFrame(error)
+
+ if self._input:
+ await self._input.push_error(error_frame)
+ elif self._output:
+ await self._output.push_error(error_frame)
+ else:
+ logger.error("Both input and output are None while trying to push error")
+ raise RuntimeError("No valid input or output channel to push error")
async def _on_app_message(self, message: Any, sender: str):
if self._input:
@@ -1110,17 +1119,23 @@ class DailyTransport(BaseTransport):
await self._call_event_handler("on_dialout_warning", data)
async def _on_participant_joined(self, participant):
+ id = participant["id"]
+ logger.info(f"Participant joined {id}")
+
+ if not self._other_participant_has_joined:
+ self._other_participant_has_joined = True
+ await self._call_event_handler("on_first_participant_joined", participant)
+
await self._call_event_handler("on_participant_joined", participant)
async def _on_participant_left(self, participant, reason):
+ id = participant["id"]
+ logger.info(f"Participant left {id}")
await self._call_event_handler("on_participant_left", participant, reason)
async def _on_participant_updated(self, participant):
await self._call_event_handler("on_participant_updated", participant)
- async def _on_first_participant_joined(self, participant):
- await self._call_event_handler("on_first_participant_joined", participant)
-
async def _on_transcription_message(self, message):
await self._call_event_handler("on_transcription_message", message)
diff --git a/src/pipecat/transports/services/helpers/daily_rest.py b/src/pipecat/transports/services/helpers/daily_rest.py
index 65b6a1d62..9becacebc 100644
--- a/src/pipecat/transports/services/helpers/daily_rest.py
+++ b/src/pipecat/transports/services/helpers/daily_rest.py
@@ -33,6 +33,19 @@ class DailyRoomSipParams(BaseModel):
num_endpoints: int = 1
+class RecordingsBucketConfig(BaseModel):
+ """Configuration for storing Daily recordings in a custom S3 bucket.
+
+ Refer to the Daily API documentation for more information:
+ https://docs.daily.co/guides/products/live-streaming-recording/storing-recordings-in-a-custom-s3-bucket
+ """
+
+ bucket_name: str
+ bucket_region: str
+ assume_role_arn: str
+ allow_api_access: bool = False
+
+
class DailyRoomProperties(BaseModel, extra="allow"):
"""Properties for configuring a Daily room.
@@ -43,6 +56,8 @@ class DailyRoomProperties(BaseModel, extra="allow"):
enable_emoji_reactions: Whether emoji reactions are enabled
eject_at_room_exp: Whether to remove participants when room expires
enable_dialout: Whether SIP dial-out is enabled
+ enable_recording: Recording settings ('cloud', 'local', 'raw-tracks')
+ geo: Geographic region for room
max_participants: Maximum number of participants allowed in the room
sip: SIP configuration parameters
sip_uri: SIP URI information returned by Daily
@@ -57,7 +72,10 @@ class DailyRoomProperties(BaseModel, extra="allow"):
enable_emoji_reactions: bool = False
eject_at_room_exp: bool = True
enable_dialout: Optional[bool] = None
+ enable_recording: Optional[Literal["cloud", "local", "raw-tracks"]] = None
+ geo: Optional[str] = None
max_participants: Optional[int] = None
+ recordings_bucket: Optional[RecordingsBucketConfig] = None
sip: Optional[DailyRoomSipParams] = None
sip_uri: Optional[dict] = None
start_video_off: bool = False
@@ -111,6 +129,84 @@ class DailyRoomObject(BaseModel):
config: DailyRoomProperties
+class DailyMeetingTokenProperties(BaseModel):
+ """Properties for configuring a Daily meeting token.
+
+ Refer to the Daily API documentation for more information:
+ https://docs.daily.co/reference/rest-api/meeting-tokens/create-meeting-token#properties
+ """
+
+ room_name: Optional[str] = Field(
+ default=None,
+ description="The room for which this token is valid. If not set, the token is valid for all rooms in your domain. You should always set room_name if using this token to control meeting access.",
+ )
+
+ eject_at_token_exp: Optional[bool] = Field(
+ default=None,
+ description="If `true`, the user will be ejected from the room when the token expires. Defaults to `false`.",
+ )
+ eject_after_elapsed: Optional[int] = Field(
+ default=None,
+ description="The number of seconds after which the user will be ejected from the room. If not provided, the user will not be ejected based on elapsed time.",
+ )
+
+ nbf: Optional[int] = Field(
+ default=None,
+ description="Not before. This is a unix timestamp (seconds since the epoch.) Users cannot join a meeting in with this token before this time.",
+ )
+
+ exp: Optional[int] = Field(
+ default=None,
+ description="Expiration time (unix timestamp in seconds). We strongly recommend setting this value for security. If not set, the token will not expire. Refer docs for more info.",
+ )
+ is_owner: Optional[bool] = Field(
+ default=None,
+ description="If `true`, the token will grant owner privileges in the room. Defaults to `false`.",
+ )
+ user_name: Optional[str] = Field(
+ default=None,
+ description="The name of the user. This will be added to the token payload.",
+ )
+ user_id: Optional[str] = Field(
+ default=None,
+ description="A unique identifier for the user. This will be added to the token payload.",
+ )
+ enable_screenshare: Optional[bool] = Field(
+ default=None,
+ description="If `true`, the user will be able to share their screen. Defaults to `true`.",
+ )
+ start_video_off: Optional[bool] = Field(
+ default=None,
+ description="If `true`, the user's video will be turned off when they join the room. Defaults to `false`.",
+ )
+ start_audio_off: Optional[bool] = Field(
+ default=None,
+ description="If `true`, the user's audio will be turned off when they join the room. Defaults to `false`.",
+ )
+ enable_recording: Optional[Literal["cloud", "local", "raw-tracks"]] = Field(
+ default=None,
+ description="Recording settings for the token. Must be one of `cloud`, `local` or `raw-tracks`.",
+ )
+ enable_prejoin_ui: Optional[bool] = Field(
+ default=None,
+ description="If `true`, the user will see the prejoin UI before joining the room.",
+ )
+ start_cloud_recording: Optional[bool] = Field(
+ default=None,
+ description="Start cloud recording when the user joins the room. This can be used to always record and archive meetings, for example in a customer support context.",
+ )
+
+
+class DailyMeetingTokenParams(BaseModel):
+ """Parameters for creating a Daily meeting token.
+
+ Refer to the Daily API documentation for more information:
+ https://docs.daily.co/reference/rest-api/meeting-tokens/create-meeting-token#body-params
+ """
+
+ properties: DailyMeetingTokenProperties = Field(default_factory=DailyMeetingTokenProperties)
+
+
class DailyRESTHelper:
"""Helper class for interacting with Daily's REST API.
@@ -129,6 +225,7 @@ class DailyRESTHelper:
daily_api_url: str = "https://api.daily.co/v1",
aiohttp_session: aiohttp.ClientSession,
):
+ """Initialize the Daily REST helper."""
self.daily_api_key = daily_api_key
self.daily_api_url = daily_api_url
self.aiohttp_session = aiohttp_session
@@ -169,7 +266,7 @@ class DailyRESTHelper:
Exception: If room creation fails or response is invalid
"""
headers = {"Authorization": f"Bearer {self.daily_api_key}"}
- json = {**params.model_dump(exclude_none=True)}
+ json = params.model_dump(exclude_none=True)
async with self.aiohttp_session.post(
f"{self.daily_api_url}/rooms", headers=headers, json=json
) as r:
@@ -187,7 +284,11 @@ class DailyRESTHelper:
return room
async def get_token(
- self, room_url: str, expiry_time: float = 60 * 60, owner: bool = True
+ self,
+ room_url: str,
+ expiry_time: float = 60 * 60,
+ owner: bool = True,
+ params: Optional[DailyMeetingTokenParams] = None,
) -> str:
"""Generate a meeting token for user to join a Daily room.
@@ -195,6 +296,9 @@ class DailyRESTHelper:
room_url: Daily room URL
expiry_time: Token validity duration in seconds (default: 1 hour)
owner: Whether token has owner privileges
+ params: Optional additional token properties. Note that room_name,
+ exp, and is_owner will be set based on the other function
+ parameters regardless of values in params.
Returns:
str: Meeting token
@@ -207,12 +311,25 @@ class DailyRESTHelper:
"No Daily room specified. You must specify a Daily room in order a token to be generated."
)
- expiration: float = time.time() + expiry_time
+ expiration: int = int(time.time() + expiry_time)
room_name = self.get_name_from_url(room_url)
headers = {"Authorization": f"Bearer {self.daily_api_key}"}
- json = {"properties": {"room_name": room_name, "is_owner": owner, "exp": expiration}}
+
+ if params is None:
+ params = DailyMeetingTokenParams(
+ properties=DailyMeetingTokenProperties(
+ room_name=room_name, is_owner=owner, exp=expiration
+ )
+ )
+ else:
+ params.properties.room_name = room_name
+ params.properties.exp = expiration
+ params.properties.is_owner = owner
+
+ json = params.model_dump(exclude_none=True)
+
async with self.aiohttp_session.post(
f"{self.daily_api_url}/meeting-tokens", headers=headers, json=json
) as r:
diff --git a/src/pipecat/transports/services/livekit.py b/src/pipecat/transports/services/livekit.py
index 0b036cecb..eff892ac0 100644
--- a/src/pipecat/transports/services/livekit.py
+++ b/src/pipecat/transports/services/livekit.py
@@ -28,6 +28,7 @@ from pipecat.processors.frame_processor import FrameDirection
from pipecat.transports.base_input import BaseInputTransport
from pipecat.transports.base_output import BaseOutputTransport
from pipecat.transports.base_transport import BaseTransport, TransportParams
+from pipecat.utils.asyncio import create_task
try:
from livekit import rtc
@@ -72,6 +73,7 @@ class LiveKitTransportClient:
params: LiveKitParams,
callbacks: LiveKitCallbacks,
loop: asyncio.AbstractEventLoop,
+ transport_name: str,
):
self._url = url
self._token = token
@@ -79,6 +81,7 @@ class LiveKitTransportClient:
self._params = params
self._callbacks = callbacks
self._loop = loop
+ self._transport_name = transport_name
self._room = rtc.Room(loop=loop)
self._participant_id: str = ""
self._connected = False
@@ -215,10 +218,18 @@ class LiveKitTransportClient:
# Wrapper methods for event handlers
def _on_participant_connected_wrapper(self, participant: rtc.RemoteParticipant):
- asyncio.create_task(self._async_on_participant_connected(participant))
+ create_task(
+ self._loop,
+ self._async_on_participant_connected(participant),
+ f"{self._transport_name}::LiveKitTransportClient::_async_on_participant_connected",
+ )
def _on_participant_disconnected_wrapper(self, participant: rtc.RemoteParticipant):
- asyncio.create_task(self._async_on_participant_disconnected(participant))
+ create_task(
+ self._loop,
+ self._async_on_participant_disconnected(participant),
+ f"{self._transport_name}::LiveKitTransportClient::_async_on_participant_disconnected",
+ )
def _on_track_subscribed_wrapper(
self,
@@ -226,7 +237,11 @@ class LiveKitTransportClient:
publication: rtc.RemoteTrackPublication,
participant: rtc.RemoteParticipant,
):
- asyncio.create_task(self._async_on_track_subscribed(track, publication, participant))
+ create_task(
+ self._loop,
+ self._async_on_track_subscribed(track, publication, participant),
+ f"{self._transport_name}::LiveKitTransportClient::_async_on_track_subscribed",
+ )
def _on_track_unsubscribed_wrapper(
self,
@@ -234,16 +249,32 @@ class LiveKitTransportClient:
publication: rtc.RemoteTrackPublication,
participant: rtc.RemoteParticipant,
):
- asyncio.create_task(self._async_on_track_unsubscribed(track, publication, participant))
+ create_task(
+ self._loop,
+ self._async_on_track_unsubscribed(track, publication, participant),
+ f"{self._transport_name}::LiveKitTransportClient::_async_on_track_unsubscribed",
+ )
def _on_data_received_wrapper(self, data: rtc.DataPacket):
- asyncio.create_task(self._async_on_data_received(data))
+ create_task(
+ self._loop,
+ self._async_on_data_received(data),
+ f"{self._transport_name}::LiveKitTransportClient::_async_on_data_received",
+ )
def _on_connected_wrapper(self):
- asyncio.create_task(self._async_on_connected())
+ create_task(
+ self._loop,
+ self._async_on_connected(),
+ f"{self._transport_name}::LiveKitTransportClient::_async_on_connected",
+ )
def _on_disconnected_wrapper(self):
- asyncio.create_task(self._async_on_disconnected())
+ create_task(
+ self._loop,
+ self._async_on_disconnected(),
+ f"{self._transport_name}::LiveKitTransportClient::_async_on_disconnected",
+ )
# Async methods for event handling
async def _async_on_participant_connected(self, participant: rtc.RemoteParticipant):
@@ -269,7 +300,11 @@ class LiveKitTransportClient:
logger.info(f"Audio track subscribed: {track.sid} from participant {participant.sid}")
self._audio_tracks[participant.sid] = track
audio_stream = rtc.AudioStream(track)
- asyncio.create_task(self._process_audio_stream(audio_stream, participant.sid))
+ create_task(
+ self._loop,
+ self._process_audio_stream(audio_stream, participant.sid),
+ f"{self._transport_name}::LiveKitTransportClient::_process_audio_stream",
+ )
async def _async_on_track_unsubscribed(
self,
@@ -319,23 +354,21 @@ class LiveKitInputTransport(BaseInputTransport):
await super().start(frame)
await self._client.connect()
if self._params.audio_in_enabled or self._params.vad_enabled:
- self._audio_in_task = asyncio.create_task(self._audio_in_task_handler())
+ self._audio_in_task = self.create_task(self._audio_in_task_handler())
logger.info("LiveKitInputTransport started")
async def stop(self, frame: EndFrame):
await super().stop(frame)
await self._client.disconnect()
if self._audio_in_task:
- self._audio_in_task.cancel()
- await self._audio_in_task
+ await self.cancel_task(self._audio_in_task)
logger.info("LiveKitInputTransport stopped")
async def cancel(self, frame: CancelFrame):
await super().cancel(frame)
await self._client.disconnect()
if self._audio_in_task and (self._params.audio_in_enabled or self._params.vad_enabled):
- self._audio_in_task.cancel()
- await self._audio_in_task
+ await self.cancel_task(self._audio_in_task)
def vad_analyzer(self) -> VADAnalyzer | None:
return self._vad_analyzer
@@ -347,22 +380,16 @@ class LiveKitInputTransport(BaseInputTransport):
async def _audio_in_task_handler(self):
logger.info("Audio input task started")
while True:
- try:
- audio_data = await self._client.get_next_audio_frame()
- if audio_data:
- audio_frame_event, participant_id = audio_data
- pipecat_audio_frame = self._convert_livekit_audio_to_pipecat(audio_frame_event)
- input_audio_frame = InputAudioRawFrame(
- audio=pipecat_audio_frame.audio,
- sample_rate=pipecat_audio_frame.sample_rate,
- num_channels=pipecat_audio_frame.num_channels,
- )
- await self.push_audio_frame(input_audio_frame)
- except asyncio.CancelledError:
- logger.info("Audio input task cancelled")
- break
- except Exception as e:
- logger.error(f"Error in audio input task: {e}")
+ audio_data = await self._client.get_next_audio_frame()
+ if audio_data:
+ audio_frame_event, participant_id = audio_data
+ pipecat_audio_frame = self._convert_livekit_audio_to_pipecat(audio_frame_event)
+ input_audio_frame = InputAudioRawFrame(
+ audio=pipecat_audio_frame.audio,
+ sample_rate=pipecat_audio_frame.sample_rate,
+ num_channels=pipecat_audio_frame.num_channels,
+ )
+ await self.push_audio_frame(input_audio_frame)
def _convert_livekit_audio_to_pipecat(
self, audio_frame_event: rtc.AudioFrameEvent
@@ -451,7 +478,7 @@ class LiveKitTransport(BaseTransport):
self._params = params
self._client = LiveKitTransportClient(
- url, token, room_name, self._params, callbacks, self._loop
+ url, token, room_name, self._params, callbacks, self._loop, self.name
)
self._input: LiveKitInputTransport | None = None
self._output: LiveKitOutputTransport | None = None
diff --git a/src/pipecat/utils/asyncio.py b/src/pipecat/utils/asyncio.py
new file mode 100644
index 000000000..d10895758
--- /dev/null
+++ b/src/pipecat/utils/asyncio.py
@@ -0,0 +1,114 @@
+#
+# Copyright (c) 2024–2025, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import asyncio
+from typing import Coroutine, Optional, Set
+
+from loguru import logger
+
+_TASKS: Set[asyncio.Task] = set()
+
+
+def create_task(loop: asyncio.AbstractEventLoop, coroutine: Coroutine, name: str) -> asyncio.Task:
+ """
+ Creates and schedules a new asyncio Task that runs the given coroutine.
+
+ The task is added to a global set of created tasks.
+
+ Args:
+ loop (asyncio.AbstractEventLoop): The event loop to use for creating the task.
+ coroutine (Coroutine): The coroutine to be executed within the task.
+ name (str): The name to assign to the task for identification.
+
+ Returns:
+ asyncio.Task: The created task object.
+ """
+
+ async def run_coroutine():
+ try:
+ await coroutine
+ except asyncio.CancelledError:
+ logger.trace(f"{name}: task cancelled")
+ # Re-raise the exception to ensure the task is cancelled.
+ raise
+ except Exception as e:
+ logger.exception(f"{name}: unexpected exception: {e}")
+
+ task = loop.create_task(run_coroutine())
+ task.set_name(name)
+ _TASKS.add(task)
+ logger.trace(f"{name}: task created")
+ return task
+
+
+async def wait_for_task(task: asyncio.Task, timeout: Optional[float] = None):
+ """Wait for an asyncio.Task to complete with optional timeout handling.
+
+ This function awaits the specified asyncio.Task and handles scenarios for
+ timeouts, cancellations, and other exceptions. It also ensures that the task
+ is removed from the set of registered tasks upon completion or failure.
+
+ Args:
+ task (asyncio.Task): The asyncio Task to wait for.
+ timeout (Optional[float], optional): The maximum number of seconds
+ to wait for the task to complete. If None, waits indefinitely.
+ Defaults to None.
+ """
+ name = task.get_name()
+ try:
+ if timeout:
+ await asyncio.wait_for(task, timeout=timeout)
+ else:
+ await task
+ except asyncio.TimeoutError:
+ logger.warning(f"{name}: timed out waiting for task to finish")
+ except asyncio.CancelledError:
+ logger.trace(f"{name}: unexpected task cancellation (maybe Ctrl-C?)")
+ except Exception as e:
+ logger.exception(f"{name}: unexpected exception while stopping task: {e}")
+ finally:
+ try:
+ _TASKS.remove(task)
+ except KeyError as e:
+ logger.trace(f"{name}: unable to remove task (already removed?): {e}")
+
+
+async def cancel_task(task: asyncio.Task, timeout: Optional[float] = None):
+ """Cancels the given asyncio Task and awaits its completion with an
+ optional timeout.
+
+ This function removes the task from the set of registered tasks upon
+ completion or failure.
+
+ Args:
+ task (asyncio.Task): The task to be cancelled.
+ timeout (Optional[float]): The optional timeout in seconds to wait for the task to cancel.
+
+ """
+ name = task.get_name()
+ task.cancel()
+ try:
+ if timeout:
+ await asyncio.wait_for(task, timeout=timeout)
+ else:
+ await task
+ except asyncio.TimeoutError:
+ logger.warning(f"{name}: timed out waiting for task to cancel")
+ except asyncio.CancelledError:
+ # Here are sure the task is cancelled properly.
+ pass
+ except Exception as e:
+ logger.exception(f"{name}: unexpected exception while cancelling task: {e}")
+ finally:
+ try:
+ _TASKS.remove(task)
+ except KeyError as e:
+ logger.trace(f"{name}: unable to remove task (already removed?): {e}")
+
+
+def current_tasks() -> Set[asyncio.Task]:
+ """Returns the list of currently created/registered tasks."""
+ return _TASKS
diff --git a/src/pipecat/utils/string.py b/src/pipecat/utils/string.py
index c901586f7..c269c5e57 100644
--- a/src/pipecat/utils/string.py
+++ b/src/pipecat/utils/string.py
@@ -14,7 +14,7 @@ ENDOFSENTENCE_PATTERN_STR = r"""
(? MyFrame -> processor1"
- )
-
- @unittest.skip("FIXME: This test is failing")
- def test_log_frame_to_sink(self):
- frame = Mock(__class__=Mock(__name__="MyFrame"))
- self.pipeline._log_frame(frame, depth=3)
- self.pipeline._logger.debug.assert_called_once_with(
- "MyClass processor2 -> MyFrame -> sink"
- )
-
- @unittest.skip("FIXME: This test is failing")
- def test_log_frame_repeated_log(self):
- frame = Mock(__class__=Mock(__name__="MyFrame"))
- self.pipeline._log_frame(frame, depth=2)
- self.pipeline._logger.debug.assert_called_once_with(
- "MyClass processor1 -> MyFrame -> processor2"
- )
- self.pipeline._log_frame(frame, depth=2)
- self.pipeline._logger.debug.assert_called_with("MyClass ... repeated")
-
- @unittest.skip("FIXME: This test is failing")
- def test_log_frame_reset_repeated_log(self):
- frame1 = Mock(__class__=Mock(__name__="MyFrame1"))
- frame2 = Mock(__class__=Mock(__name__="MyFrame2"))
- self.pipeline._log_frame(frame1, depth=2)
- self.pipeline._logger.debug.assert_called_once_with(
- "MyClass processor1 -> MyFrame1 -> processor2"
- )
- self.pipeline._log_frame(frame1, depth=2)
- self.pipeline._logger.debug.assert_called_with("MyClass ... repeated")
- self.pipeline._log_frame(frame2, depth=2)
- self.pipeline._logger.debug.assert_called_with(
- "MyClass processor1 -> MyFrame2 -> processor2"
- )
+ await task.queue_frame(TextFrame(text="Hello!"))
+ try:
+ await asyncio.wait_for(task.run(), timeout=1.0)
+ except asyncio.TimeoutError:
+ pass
+ assert heartbeats_counter == expected_heartbeats
diff --git a/tests/test_protobuf_serializer.py b/tests/test_protobuf_serializer.py
index 7f9841622..88fd7867e 100644
--- a/tests/test_protobuf_serializer.py
+++ b/tests/test_protobuf_serializer.py
@@ -1,6 +1,16 @@
+#
+# Copyright (c) 2024-2025 Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
import unittest
-from pipecat.frames.frames import AudioRawFrame, TextFrame, TranscriptionFrame
+from pipecat.frames.frames import (
+ OutputAudioRawFrame,
+ TextFrame,
+ TranscriptionFrame,
+)
from pipecat.serializers.protobuf import ProtobufFrameSerializer
@@ -8,22 +18,19 @@ class TestProtobufFrameSerializer(unittest.IsolatedAsyncioTestCase):
def setUp(self):
self.serializer = ProtobufFrameSerializer()
- @unittest.skip("FIXME: This test is failing")
async def test_roundtrip(self):
text_frame = TextFrame(text="hello world")
frame = self.serializer.deserialize(self.serializer.serialize(text_frame))
- self.assertEqual(frame, TextFrame(text="hello world"))
+ self.assertEqual(text_frame, frame)
transcription_frame = TranscriptionFrame(
- text="Hello there!", participantId="123", timestamp="2021-01-01"
+ text="Hello there!", user_id="123", timestamp="2021-01-01"
)
frame = self.serializer.deserialize(self.serializer.serialize(transcription_frame))
self.assertEqual(frame, transcription_frame)
- audio_frame = AudioRawFrame(data=b"1234567890")
+ audio_frame = OutputAudioRawFrame(audio=b"1234567890", sample_rate=16000, num_channels=1)
frame = self.serializer.deserialize(self.serializer.serialize(audio_frame))
- self.assertEqual(frame, audio_frame)
-
-
-if __name__ == "__main__":
- unittest.main()
+ self.assertEqual(frame.audio, audio_frame.audio)
+ self.assertEqual(frame.sample_rate, audio_frame.sample_rate)
+ self.assertEqual(frame.num_channels, audio_frame.num_channels)
diff --git a/tests/test_ai_services.py b/tests/test_utils_string.py
similarity index 63%
rename from tests/test_ai_services.py
rename to tests/test_utils_string.py
index 13aa20467..aa135a639 100644
--- a/tests/test_ai_services.py
+++ b/tests/test_utils_string.py
@@ -1,28 +1,15 @@
+#
+# Copyright (c) 2024-2025 Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
import unittest
-from typing import AsyncGenerator
-from pipecat.frames.frames import EndFrame, Frame, TextFrame
-from pipecat.services.ai_services import AIService, match_endofsentence
+from pipecat.utils.string import match_endofsentence
-class SimpleAIService(AIService):
- async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
- yield frame
-
-
-class TestBaseAIService(unittest.IsolatedAsyncioTestCase):
- async def test_simple_processing(self):
- service = SimpleAIService()
-
- input_frames = [TextFrame("hello"), EndFrame()]
-
- output_frames = []
- for input_frame in input_frames:
- async for output_frame in service.process_frame(input_frame):
- output_frames.append(output_frame)
-
- self.assertEqual(input_frames, output_frames)
-
+class TestUtilsString(unittest.IsolatedAsyncioTestCase):
async def test_endofsentence(self):
assert match_endofsentence("This is a sentence.")
assert match_endofsentence("This is a sentence! ")
@@ -52,6 +39,13 @@ class TestBaseAIService(unittest.IsolatedAsyncioTestCase):
assert match_endofsentence(i)
assert not match_endofsentence("你好，")
-
-if __name__ == "__main__":
- unittest.main()
+ async def test_endofsentence_hi(self):
+ hindi_sentences = [
+ "हैलो।",
+ "हैलो！",
+ "आप खाये हैं？",
+ "सुरक्षा पहले।",
+ ]
+ for i in hindi_sentences:
+ assert match_endofsentence(i)
+ assert not match_endofsentence("हैलो，")
diff --git a/tests/test_websocket_transport.py b/tests/test_websocket_transport.py
index b24caa5b9..c918acd8a 100644
--- a/tests/test_websocket_transport.py
+++ b/tests/test_websocket_transport.py
@@ -1,3 +1,9 @@
+#
+# Copyright (c) 2024-2025 Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
# import asyncio
# import unittest
# from unittest.mock import AsyncMock, patch, Mock
-[](https://pypi.org/project/pipecat-ai) [](https://docs.pipecat.ai) [](https://discord.gg/pipecat) 
-