diff --git a/CHANGELOG.md b/CHANGELOG.md
index 466b3f28b..f153dda13 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,10 +9,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- Added logging and improved error handling to help diagnose and prevent potential
+ Pipeline freezes.
+
+- Introduce task watchdog timers. Watchdog timers are used to detect if a
+ Pipecat task is taking longer than expected (by default 5 seconds). It is
+ possible to change the default watchdog timer timeout by using the
+ `watchdog_timeout` constructor argument when creating a `PipelineTask`. With
+ watchdog timers it is also possible to log how long each processing step is
+ taking (e.g. processing an element from a queue inside a task). This is done
+ with the `enable_watchdog_logging` constructor argument when creating a
+ `PipelineTask.` It is also possible to control these two values per each frame
+ processor. That is, you can set set `enable_watchdog_logging` and
+ `watchdog_timeout` when creating any frame processor through their constructor
+ arguments. Finally, you can also set these values per task. So, if you are
+ writing a frame processor that creates multiple tasks and you only want to
+ enable logging for one of them, you can do so by passing the same argument
+ names to the `FrameProcessor.create_task()` function. Note that watchdog
+ timers only work with Pipecat tasks but not if you use `asycio.create_task()`
+ or similar.
+
- Added `lexicon_names` parameter to `AWSPollyTTSService.InputParams`.
- Added reconnection logic and audio buffer management to `GladiaSTTService`.
+- The `TurnTrackingObserver` now ends a turn upon observing an `EndFrame` or
+ `CancelFrame`.
+
- Added Polish support to `AWSTranscribeSTTService`.
- Added new frames `FrameProcessorPauseFrame` and `FrameProcessorResumeFrame`
@@ -29,8 +52,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`LLMAssistantContextAggregator` that exposes whether a function call is in
progress.
+- Added `SambaNovaLLMService` which provides llm api integration with an
+ OpenAI-compatible interface.
+
+- Added `SambaNovaTTSService` which provides speech-to-text functionality using
+ SambaNovas's (whisper) API.
+
+- Add fundational examples for function calling and transcription
+ `14s-function-calling-sambanova.py`, `13g-sambanova-transcription.py`
+
### Changed
+- `HeartbeatFrame`s are now control frames. This will make it easier to detect
+ pipeline freezes. Previously, heartbeat frames were system frames which meant
+ they were not get queued with other frames, making it difficult to detect
+ pipeline stalls.
+
+- Updated `OpenAIRealtimeBetaLLMService` to accept `language` in the
+ `InputAudioTranscription` class for all models.
+
+- Updated the default model for `OpenAIRealtimeBetaLLMService` to
+ `gpt-4o-realtime-preview-2025-06-03`.
+
- The `PipelineParams` arg `allow_interruptions` now defaults to `True`.
- `TavusTransport` and `TavusVideoService` now send audio to Tavus using WebRTC
@@ -41,6 +84,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- Fixed an event loop blocking issue when using `SentryMetrics`.
+
+- Fixed an issue in `FastAPIWebsocketClient` to ensure proper disconnection
+ when the websocket is already closed.
+
+- Fixed an issue where the `UserStoppedSpeakingFrame` was not received if the
+ transport was not receiving new audio frames.
+
+- Fixed an edge case where if the user interrupted the bot but no new aggregation
+ was received, the bot would not resume speaking.
+
+- Fixed an issue with `TelnyxFrameSerializer` where it would throw an exception
+ when the user hung up the call.
+
+- Fixed an issue with `ElevenLabsTTSService` where the context was not being
+ closed.
+
- Fixed function calling in `AWSNovaSonicLLMService`.
- Fixed an issue that would cause multiple `PipelineTask.on_idle_timeout`
diff --git a/README.md b/README.md
index 7ec4c6000..2d14f37c9 100644
--- a/README.md
+++ b/README.md
@@ -53,8 +53,8 @@ You can connect to Pipecat from any platform using our official SDKs:
| Category | Services |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| Speech-to-Text | [AssemblyAI](https://docs.pipecat.ai/server/services/stt/assemblyai), [AWS](https://docs.pipecat.ai/server/services/stt/aws), [Azure](https://docs.pipecat.ai/server/services/stt/azure), [Cartesia](https://docs.pipecat.ai/server/services/stt/cartesia), [Deepgram](https://docs.pipecat.ai/server/services/stt/deepgram), [Fal Wizper](https://docs.pipecat.ai/server/services/stt/fal), [Gladia](https://docs.pipecat.ai/server/services/stt/gladia), [Google](https://docs.pipecat.ai/server/services/stt/google), [Groq (Whisper)](https://docs.pipecat.ai/server/services/stt/groq), [OpenAI (Whisper)](https://docs.pipecat.ai/server/services/stt/openai), [Parakeet (NVIDIA)](https://docs.pipecat.ai/server/services/stt/parakeet), [Ultravox](https://docs.pipecat.ai/server/services/stt/ultravox), [Whisper](https://docs.pipecat.ai/server/services/stt/whisper) |
-| LLMs | [Anthropic](https://docs.pipecat.ai/server/services/llm/anthropic), [AWS](https://docs.pipecat.ai/server/services/llm/aws), [Azure](https://docs.pipecat.ai/server/services/llm/azure), [Cerebras](https://docs.pipecat.ai/server/services/llm/cerebras), [DeepSeek](https://docs.pipecat.ai/server/services/llm/deepseek), [Fireworks AI](https://docs.pipecat.ai/server/services/llm/fireworks), [Gemini](https://docs.pipecat.ai/server/services/llm/gemini), [Grok](https://docs.pipecat.ai/server/services/llm/grok), [Groq](https://docs.pipecat.ai/server/services/llm/groq), [NVIDIA NIM](https://docs.pipecat.ai/server/services/llm/nim), [Ollama](https://docs.pipecat.ai/server/services/llm/ollama), [OpenAI](https://docs.pipecat.ai/server/services/llm/openai), [OpenRouter](https://docs.pipecat.ai/server/services/llm/openrouter), [Perplexity](https://docs.pipecat.ai/server/services/llm/perplexity), [Qwen](https://docs.pipecat.ai/server/services/llm/qwen), [Together AI](https://docs.pipecat.ai/server/services/llm/together) |
+| Speech-to-Text | [AssemblyAI](https://docs.pipecat.ai/server/services/stt/assemblyai), [AWS](https://docs.pipecat.ai/server/services/stt/aws), [Azure](https://docs.pipecat.ai/server/services/stt/azure), [Cartesia](https://docs.pipecat.ai/server/services/stt/cartesia), [Deepgram](https://docs.pipecat.ai/server/services/stt/deepgram), [Fal Wizper](https://docs.pipecat.ai/server/services/stt/fal), [Gladia](https://docs.pipecat.ai/server/services/stt/gladia), [Google](https://docs.pipecat.ai/server/services/stt/google), [Groq (Whisper)](https://docs.pipecat.ai/server/services/stt/groq), [OpenAI (Whisper)](https://docs.pipecat.ai/server/services/stt/openai), [Parakeet (NVIDIA)](https://docs.pipecat.ai/server/services/stt/parakeet), [SambaNova (Whisper)](https://docs.pipecat.ai/server/services/stt/sambanova) [Ultravox](https://docs.pipecat.ai/server/services/stt/ultravox), [Whisper](https://docs.pipecat.ai/server/services/stt/whisper) |
+| LLMs | [Anthropic](https://docs.pipecat.ai/server/services/llm/anthropic), [AWS](https://docs.pipecat.ai/server/services/llm/aws), [Azure](https://docs.pipecat.ai/server/services/llm/azure), [Cerebras](https://docs.pipecat.ai/server/services/llm/cerebras), [DeepSeek](https://docs.pipecat.ai/server/services/llm/deepseek), [Fireworks AI](https://docs.pipecat.ai/server/services/llm/fireworks), [Gemini](https://docs.pipecat.ai/server/services/llm/gemini), [Grok](https://docs.pipecat.ai/server/services/llm/grok), [Groq](https://docs.pipecat.ai/server/services/llm/groq), [NVIDIA NIM](https://docs.pipecat.ai/server/services/llm/nim), [Ollama](https://docs.pipecat.ai/server/services/llm/ollama), [OpenAI](https://docs.pipecat.ai/server/services/llm/openai), [OpenRouter](https://docs.pipecat.ai/server/services/llm/openrouter), [Perplexity](https://docs.pipecat.ai/server/services/llm/perplexity), [Qwen](https://docs.pipecat.ai/server/services/llm/qwen), [SambaNova](https://docs.pipecat.ai/server/services/llm/sambanova) [Together AI](https://docs.pipecat.ai/server/services/llm/together) |
| Text-to-Speech | [AWS](https://docs.pipecat.ai/server/services/tts/aws), [Azure](https://docs.pipecat.ai/server/services/tts/azure), [Cartesia](https://docs.pipecat.ai/server/services/tts/cartesia), [Deepgram](https://docs.pipecat.ai/server/services/tts/deepgram), [ElevenLabs](https://docs.pipecat.ai/server/services/tts/elevenlabs), [FastPitch (NVIDIA)](https://docs.pipecat.ai/server/services/tts/fastpitch), [Fish](https://docs.pipecat.ai/server/services/tts/fish), [Google](https://docs.pipecat.ai/server/services/tts/google), [LMNT](https://docs.pipecat.ai/server/services/tts/lmnt), [MiniMax](https://docs.pipecat.ai/server/services/tts/minimax), [Neuphonic](https://docs.pipecat.ai/server/services/tts/neuphonic), [OpenAI](https://docs.pipecat.ai/server/services/tts/openai), [Piper](https://docs.pipecat.ai/server/services/tts/piper), [PlayHT](https://docs.pipecat.ai/server/services/tts/playht), [Rime](https://docs.pipecat.ai/server/services/tts/rime), [Sarvam](https://docs.pipecat.ai/server/services/tts/sarvam), [XTTS](https://docs.pipecat.ai/server/services/tts/xtts) |
| Speech-to-Speech | [AWS Nova Sonic](https://docs.pipecat.ai/server/services/s2s/aws), [Gemini Multimodal Live](https://docs.pipecat.ai/server/services/s2s/gemini), [OpenAI Realtime](https://docs.pipecat.ai/server/services/s2s/openai) |
| Transport | [Daily (WebRTC)](https://docs.pipecat.ai/server/services/transport/daily), [FastAPI Websocket](https://docs.pipecat.ai/server/services/transport/fastapi-websocket), [SmallWebRTCTransport](https://docs.pipecat.ai/server/services/transport/small-webrtc), [WebSocket Server](https://docs.pipecat.ai/server/services/transport/websocket-server), Local |
diff --git a/docs/api/requirements.txt b/docs/api/requirements.txt
index a77ff1084..d783b33e8 100644
--- a/docs/api/requirements.txt
+++ b/docs/api/requirements.txt
@@ -42,6 +42,7 @@ pipecat-ai[openai]
pipecat-ai[qwen]
pipecat-ai[remote-smart-turn]
# pipecat-ai[riva] # Mocked
+pipecat-ai[sambanova]
pipecat-ai[silero]
pipecat-ai[simli]
pipecat-ai[soundfile]
diff --git a/dot-env.template b/dot-env.template
index 20d73b3ad..f4fd43eea 100644
--- a/dot-env.template
+++ b/dot-env.template
@@ -107,4 +107,10 @@ MINIMAX_API_KEY=...
MINIMAX_GROUP_ID=...
# Sarvam AI
-SARVAM_API_KEY=...
\ No newline at end of file
+SARVAM_API_KEY=...
+
+# SambaNova
+SAMBANOVA_API_KEY=...
+
+# Sentry
+SENTRY_DSN=...
diff --git a/examples/foundational/07s-interruptible-google-audio-in.py b/examples/foundational/07s-interruptible-google-audio-in.py
index c10ca02d0..9a7aa24b1 100644
--- a/examples/foundational/07s-interruptible-google-audio-in.py
+++ b/examples/foundational/07s-interruptible-google-audio-in.py
@@ -8,8 +8,8 @@ import argparse
import os
from dataclasses import dataclass
-import google.ai.generativelanguage as glm
from dotenv import load_dotenv
+from google.genai.types import Content, Part
from loguru import logger
from pipecat.audio.vad.silero import SileroVADAnalyzer
@@ -164,9 +164,7 @@ class TanscriptionContextFixup(FrameProcessor):
and last_part.inline_data
and last_part.inline_data.mime_type == "audio/wav"
):
- self._context.messages[-2] = glm.Content(
- role="user", parts=[glm.Part(text=self._transcript)]
- )
+ self._context.messages[-2] = Content(role="user", parts=[Part(text=self._transcript)])
def add_transcript_back_to_inference_output(self):
if not self._transcript:
diff --git a/examples/foundational/13g-sambanova-transcription.py b/examples/foundational/13g-sambanova-transcription.py
new file mode 100644
index 000000000..bff8aa4d4
--- /dev/null
+++ b/examples/foundational/13g-sambanova-transcription.py
@@ -0,0 +1,108 @@
+#
+# Copyright (c) 2024–2025, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import argparse
+import os
+import time
+
+from dotenv import load_dotenv
+from loguru import logger
+
+from pipecat.audio.vad.silero import SileroVADAnalyzer
+from pipecat.audio.vad.vad_analyzer import VADParams
+from pipecat.frames.frames import Frame, TranscriptionFrame, UserStoppedSpeakingFrame
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.runner import PipelineRunner
+from pipecat.pipeline.task import PipelineParams, PipelineTask
+from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
+from pipecat.services.sambanova.stt import SambaNovaSTTService
+from pipecat.transports.base_transport import BaseTransport, TransportParams
+from pipecat.transports.network.fastapi_websocket import FastAPIWebsocketParams
+from pipecat.transports.services.daily import DailyParams
+
+load_dotenv(override=True)
+
+
+STOP_SECS = 2.0
+
+
+class TranscriptionLogger(FrameProcessor):
+ """Measures transcription latency.
+
+ Uses the (intentionally) long STOP_SECS parameter to give the transcription time to finish,
+ then outputs the timing between when the VAD first classified audio input as not-speech and
+ the delivery of the last transcription frame.
+ """
+
+ def __init__(self):
+ super().__init__()
+ self._last_transcription_time = time.time()
+
+ async def process_frame(self, frame: Frame, direction: FrameDirection):
+ await super().process_frame(frame, direction)
+
+ if isinstance(frame, UserStoppedSpeakingFrame):
+ logger.debug(
+ f"Transcription latency: {(STOP_SECS - (time.time() - self._last_transcription_time)):.2f}"
+ )
+
+ if isinstance(frame, TranscriptionFrame):
+ self._last_transcription_time = time.time()
+
+
+# We store functions so objects (e.g. SileroVADAnalyzer) don't get
+# instantiated. The function will be called when the desired transport gets
+# selected.
+transport_params = {
+ "daily": lambda: DailyParams(
+ audio_in_enabled=True,
+ vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=STOP_SECS)),
+ ),
+ "twilio": lambda: FastAPIWebsocketParams(
+ audio_in_enabled=True,
+ vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=STOP_SECS)),
+ ),
+ "webrtc": lambda: TransportParams(
+ audio_in_enabled=True,
+ vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=STOP_SECS)),
+ ),
+}
+
+
+async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_sigint: bool):
+ logger.info(f"Starting bot")
+
+ stt = SambaNovaSTTService(
+ model="Whisper-Large-v3",
+ api_key=os.getenv("SAMBANOVA_API_KEY"),
+ )
+
+ tl = TranscriptionLogger()
+
+ pipeline = Pipeline([transport.input(), stt, tl])
+
+ task = PipelineTask(
+ pipeline,
+ params=PipelineParams(
+ enable_metrics=True,
+ enable_usage_metrics=True,
+ ),
+ )
+
+ @transport.event_handler("on_client_disconnected")
+ async def on_client_disconnected(transport, client):
+ logger.info(f"Client disconnected")
+ await task.cancel()
+
+ runner = PipelineRunner(handle_sigint=handle_sigint)
+
+ await runner.run(task)
+
+
+if __name__ == "__main__":
+ from pipecat.examples.run import main
+
+ main(run_example, transport_params=transport_params)
diff --git a/examples/foundational/14s-function-calling-sambanova.py b/examples/foundational/14s-function-calling-sambanova.py
new file mode 100644
index 000000000..dd11af527
--- /dev/null
+++ b/examples/foundational/14s-function-calling-sambanova.py
@@ -0,0 +1,152 @@
+#
+# Copyright (c) 2024–2025, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import argparse
+import os
+
+from dotenv import load_dotenv
+from loguru import logger
+
+from pipecat.adapters.schemas.function_schema import FunctionSchema
+from pipecat.adapters.schemas.tools_schema import ToolsSchema
+from pipecat.audio.vad.silero import SileroVADAnalyzer
+from pipecat.frames.frames import TTSSpeakFrame
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.runner import PipelineRunner
+from pipecat.pipeline.task import PipelineParams, PipelineTask
+from pipecat.processors.aggregators.llm_response import LLMUserAggregatorParams
+from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
+from pipecat.services.cartesia.tts import CartesiaTTSService
+from pipecat.services.llm_service import FunctionCallParams
+from pipecat.services.sambanova.llm import SambaNovaLLMService
+from pipecat.services.sambanova.stt import SambaNovaSTTService
+from pipecat.transports.base_transport import BaseTransport, TransportParams
+from pipecat.transports.network.fastapi_websocket import FastAPIWebsocketParams
+from pipecat.transports.services.daily import DailyParams
+
+load_dotenv(override=True)
+
+
+async def fetch_weather_from_api(params: FunctionCallParams):
+ await params.result_callback({"conditions": "nice", "temperature": "75"})
+
+
+# We store functions so objects (e.g. SileroVADAnalyzer) don't get
+# instantiated. The function will be called when the desired transport gets
+# selected.
+transport_params = {
+ "daily": lambda: DailyParams(
+ audio_in_enabled=True,
+ audio_out_enabled=True,
+ vad_analyzer=SileroVADAnalyzer(),
+ ),
+ "twilio": lambda: FastAPIWebsocketParams(
+ audio_in_enabled=True,
+ audio_out_enabled=True,
+ vad_analyzer=SileroVADAnalyzer(),
+ ),
+ "webrtc": lambda: TransportParams(
+ audio_in_enabled=True,
+ audio_out_enabled=True,
+ vad_analyzer=SileroVADAnalyzer(),
+ ),
+}
+
+
+async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_sigint: bool):
+ logger.info(f"Starting bot")
+
+ stt = SambaNovaSTTService(
+ model="Whisper-Large-v3",
+ api_key=os.getenv("SAMBANOVA_API_KEY"),
+ )
+
+ tts = CartesiaTTSService(
+ api_key=os.getenv("CARTESIA_API_KEY"),
+ voice_id="71a7ad14-091c-4e8e-a314-022ece01c121", # British Reading Lady
+ )
+
+ llm = SambaNovaLLMService(
+ api_key=os.getenv("SAMBANOVA_API_KEY"),
+ model="Llama-4-Maverick-17B-128E-Instruct",
+ )
+ # You can also register a function_name of None to get all functions
+ # sent to the same callback with an additional function_name parameter.
+ llm.register_function("get_current_weather", fetch_weather_from_api)
+
+ @llm.event_handler("on_function_calls_started")
+ async def on_function_calls_started(service, function_calls):
+ await tts.queue_frame(TTSSpeakFrame("Let me check on that."))
+
+ weather_function = FunctionSchema(
+ name="get_current_weather",
+ description="Get the current weather",
+ properties={
+ "location": {
+ "type": "string",
+ "description": "The city and state, e.g. San Francisco, CA",
+ },
+ "format": {
+ "type": "string",
+ "enum": ["celsius", "fahrenheit"],
+ "description": "The temperature unit to use. Infer this from the user's location.",
+ },
+ },
+ required=["location"],
+ )
+ tools = ToolsSchema(standard_tools=[weather_function])
+ messages = [
+ {
+ "role": "system",
+ "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
+ },
+ ]
+
+ context = OpenAILLMContext(messages, tools)
+ context_aggregator = llm.create_context_aggregator(
+ context, user_params=LLMUserAggregatorParams(aggregation_timeout=0.05)
+ )
+
+ pipeline = Pipeline(
+ [
+ transport.input(),
+ stt,
+ context_aggregator.user(),
+ llm,
+ tts,
+ transport.output(),
+ context_aggregator.assistant(),
+ ]
+ )
+
+ task = PipelineTask(
+ pipeline,
+ params=PipelineParams(
+ enable_metrics=True,
+ enable_usage_metrics=True,
+ ),
+ )
+
+ @transport.event_handler("on_client_connected")
+ async def on_client_connected(transport, client):
+ logger.info(f"Client connected")
+ # Kick off the conversation.
+ await task.queue_frames([context_aggregator.user().get_context_frame()])
+
+ @transport.event_handler("on_client_disconnected")
+ async def on_client_disconnected(transport, client):
+ logger.info(f"Client disconnected")
+ await task.cancel()
+
+ runner = PipelineRunner(handle_sigint=handle_sigint)
+
+ await runner.run(task)
+
+
+if __name__ == "__main__":
+ from pipecat.examples.run import main
+
+ main(run_example, transport_params=transport_params)
diff --git a/examples/foundational/22d-natural-conversation-gemini-audio.py b/examples/foundational/22d-natural-conversation-gemini-audio.py
index b8e21266f..0d053febf 100644
--- a/examples/foundational/22d-natural-conversation-gemini-audio.py
+++ b/examples/foundational/22d-natural-conversation-gemini-audio.py
@@ -9,8 +9,8 @@ import asyncio
import os
import time
-import google.ai.generativelanguage as glm
from dotenv import load_dotenv
+from google.genai.types import Content, Part
from loguru import logger
from pipecat.audio.vad.silero import SileroVADAnalyzer
@@ -611,9 +611,7 @@ class OutputGate(FrameProcessor):
await self._notifier.wait()
transcription = await self._transcription_buffer.wait_for_transcription() or "-"
- self._context._messages.append(
- glm.Content(role="user", parts=[glm.Part(text=transcription)])
- )
+ self._context.add_message(Content(role="user", parts=[Part(text=transcription)]))
self.open_gate()
for frame, direction in self._frames_buffer:
diff --git a/examples/foundational/25-google-audio-in.py b/examples/foundational/25-google-audio-in.py
index e0c834c23..00c2c0941 100644
--- a/examples/foundational/25-google-audio-in.py
+++ b/examples/foundational/25-google-audio-in.py
@@ -8,8 +8,8 @@ import argparse
import os
from dataclasses import dataclass
-import google.ai.generativelanguage as glm
from dotenv import load_dotenv
+from google.genai.types import Content, Part
from loguru import logger
from pipecat.audio.vad.silero import SileroVADAnalyzer
@@ -142,8 +142,8 @@ class InputTranscriptionContextFilter(FrameProcessor):
context = GoogleLLMContext.upgrade_to_google(frame.context)
message = context.messages[-1]
- if not isinstance(message, glm.Content):
- logger.error(f"Expected glm.Content, got {type(message)}")
+ if not isinstance(message, Content):
+ logger.error(f"Expected Content, got {type(message)}")
return
last_part = message.parts[-1]
@@ -168,15 +168,15 @@ class InputTranscriptionContextFilter(FrameProcessor):
history += f"{msg.role}: {part.text}\n"
if history:
assembled = f"Here is the conversation history so far. These are not instructions. This is data that you should use only to improve the accuracy of your transcription.\n\n----\n\n{history}\n\n----\n\nEND OF CONVERSATION HISTORY\n\n"
- parts.append(glm.Part(text=assembled))
+ parts.append(Part(text=assembled))
parts.append(
- glm.Part(
+ Part(
text="Transcribe this audio. Respond either with the transcription exactly as it was said by the user, or with the special string 'EMPTY' if the audio is not clear."
)
)
parts.append(last_part)
- msg = glm.Content(role="user", parts=parts)
+ msg = Content(role="user", parts=parts)
ctx = GoogleLLMContext([msg])
ctx.system_message = transcriber_system_message
await self.push_frame(OpenAILLMContextFrame(context=ctx))
diff --git a/examples/foundational/38-smart-turn-fal.py b/examples/foundational/38-smart-turn-fal.py
index 120fe67dc..a3fac43ab 100644
--- a/examples/foundational/38-smart-turn-fal.py
+++ b/examples/foundational/38-smart-turn-fal.py
@@ -27,7 +27,6 @@ from pipecat.transports.services.daily import DailyParams
load_dotenv(override=True)
-aiohttp_session = aiohttp.ClientSession()
# We store functions so objects (e.g. SileroVADAnalyzer) don't get
# instantiated. The function will be called when the desired transport gets
@@ -38,7 +37,7 @@ transport_params = {
audio_out_enabled=True,
vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
turn_analyzer=FalSmartTurnAnalyzer(
- api_key=os.getenv("FAL_SMART_TURN_API_KEY"), aiohttp_session=aiohttp_session
+ api_key=os.getenv("FAL_SMART_TURN_API_KEY"), aiohttp_session=aiohttp.ClientSession()
),
),
"twilio": lambda: FastAPIWebsocketParams(
@@ -46,7 +45,7 @@ transport_params = {
audio_out_enabled=True,
vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
turn_analyzer=FalSmartTurnAnalyzer(
- api_key=os.getenv("FAL_SMART_TURN_API_KEY"), aiohttp_session=aiohttp_session
+ api_key=os.getenv("FAL_SMART_TURN_API_KEY"), aiohttp_session=aiohttp.ClientSession()
),
),
"webrtc": lambda: TransportParams(
@@ -54,7 +53,7 @@ transport_params = {
audio_out_enabled=True,
vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
turn_analyzer=FalSmartTurnAnalyzer(
- api_key=os.getenv("FAL_SMART_TURN_API_KEY"), aiohttp_session=aiohttp_session
+ api_key=os.getenv("FAL_SMART_TURN_API_KEY"), aiohttp_session=aiohttp.ClientSession()
),
),
}
@@ -118,8 +117,6 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si
await runner.run(task)
- await aiohttp_session.close()
-
if __name__ == "__main__":
from pipecat.examples.run import main
diff --git a/examples/foundational/39a-mcp-run-sse.py b/examples/foundational/39a-mcp-run-sse.py
index 8b43b3022..fbbcbcc6a 100644
--- a/examples/foundational/39a-mcp-run-sse.py
+++ b/examples/foundational/39a-mcp-run-sse.py
@@ -9,6 +9,7 @@ import os
from dotenv import load_dotenv
from loguru import logger
+from mcp.client.session_group import SseServerParameters
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.pipeline.pipeline import Pipeline
@@ -63,7 +64,7 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si
try:
# https://docs.mcp.run/integrating/tutorials/mcp-run-sse-openai-agents/
- mcp = MCPClient(server_params=os.getenv("MCP_RUN_SSE_URL"))
+ mcp = MCPClient(server_params=SseServerParameters(url=os.getenv("MCP_RUN_SSE_URL")))
except Exception as e:
logger.error(f"error setting up mcp")
logger.exception("error trace:")
diff --git a/examples/foundational/39b-multiple-mcp.py b/examples/foundational/39b-multiple-mcp.py
index 3ba90d580..7d1396834 100644
--- a/examples/foundational/39b-multiple-mcp.py
+++ b/examples/foundational/39b-multiple-mcp.py
@@ -15,6 +15,7 @@ import aiohttp
from dotenv import load_dotenv
from loguru import logger
from mcp import StdioServerParameters
+from mcp.client.session_group import SseServerParameters
from PIL import Image
from pipecat.adapters.schemas.tools_schema import ToolsSchema
@@ -149,7 +150,7 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si
# https://docs.mcp.run/integrating/tutorials/mcp-run-sse-openai-agents/
# ie. "https://www.mcp.run/api/mcp/sse?..."
# ensure the profile has a tool or few installed
- mcp_run = MCPClient(server_params=os.getenv("MCP_RUN_SSE_URL"))
+ mcp_run = MCPClient(server_params=SseServerParameters(url=os.getenv("MCP_RUN_SSE_URL")))
except Exception as e:
logger.error(f"error setting up mcp.run")
logger.exception("error trace:")
diff --git a/examples/freeze-test/README.md b/examples/freeze-test/README.md
new file mode 100644
index 000000000..4bbc8deae
--- /dev/null
+++ b/examples/freeze-test/README.md
@@ -0,0 +1,59 @@
+# Freeze Test Client
+
+The purpose of this example is to create an environment for testing the bot and try to create freezing conditions.
+
+### Approach 1: Server-Side Testing with `SimulateFreezeInput`
+
+- Utilize only the bot `freeze_test_bot.py` with the `SimulateFreezeInput` processor. This input continuously injects frames, simulating user speech interruptions at random intervals.
+- This approach excludes the use of input transport and speech-to-text (STT) functionalities.
+
+### Approach 2: Server-Side with TypeScript Client
+
+- Combine server-side operations with a TypeScript client.
+- The client initially records a segment of audio, e.g., 5–10 seconds long. It can be anything.
+- After that, it replays this recorded audio to the server at random intervals, mimicking user input interruptions.
+- This helps testing interruptions in the pipeline as if real users were interacting with the bot.
+
+## Setup
+
+Follow these steps to set up and run the Freeze Test Client:
+
+1. **Run the Bot Server**
+ - Set up and activate your virtual environment:
+ ```bash
+ python3 -m venv venv
+ source venv/bin/activate # On Windows: venv\Scripts\activate
+ ```
+
+ - Install dependencies:
+ ```bash
+ pip install -r requirements.txt
+ ```
+
+ - Create your `.env` file and set your env vars:
+ ```bash
+ cp env.example .env
+ ```
+
+ - Run the server:
+ ```bash
+ python freeze_test_bot.py
+ ```
+
+2. **Navigate to the Client Directory**
+ ```bash
+ cd client
+ ```
+
+3. **Install Dependencies**
+ ```bash
+ npm install
+ ```
+
+4. **Run the Client Application**
+ ```bash
+ npm run dev
+ ```
+
+5. **Access the Client in Your Browser**
+ Visit [http://localhost:5173](http://localhost:5173) to interact with the Freeze Test Client.
diff --git a/examples/freeze-test/client/index.html b/examples/freeze-test/client/index.html
new file mode 100644
index 000000000..843ade882
--- /dev/null
+++ b/examples/freeze-test/client/index.html
@@ -0,0 +1,43 @@
+
+
+
+
+
+
+ AI Chatbot
+
+
+
+
+
+
+ Transport: Disconnected
+
+
+
+
+
+
+
+
+ Playing audio:
+
+
+
+
+
+
+
+
+
+
+
Debug Info
+
+
+
+
+
+
+
+
+
diff --git a/examples/freeze-test/client/package-lock.json b/examples/freeze-test/client/package-lock.json
new file mode 100644
index 000000000..f8157d4e1
--- /dev/null
+++ b/examples/freeze-test/client/package-lock.json
@@ -0,0 +1,1770 @@
+{
+ "name": "client",
+ "version": "1.0.0",
+ "lockfileVersion": 3,
+ "requires": true,
+ "packages": {
+ "": {
+ "name": "client",
+ "version": "1.0.0",
+ "license": "ISC",
+ "dependencies": {
+ "@pipecat-ai/client-js": "^0.4.0",
+ "@pipecat-ai/websocket-transport": "^0.4.1",
+ "protobufjs": "^7.4.0"
+ },
+ "devDependencies": {
+ "@types/node": "^22.15.30",
+ "@types/protobufjs": "^6.0.0",
+ "@vitejs/plugin-react-swc": "^3.10.1",
+ "typescript": "^5.8.3",
+ "vite": "^6.3.5"
+ }
+ },
+ "node_modules/@babel/runtime": {
+ "version": "7.27.6",
+ "resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.27.6.tgz",
+ "integrity": "sha512-vbavdySgbTTrmFE+EsiqUTzlOr5bzlnJtUv9PynGCAKvfQqjIXbvFdumPM/GxMDfyuGMJaJAU6TO4zc1Jf1i8Q==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=6.9.0"
+ }
+ },
+ "node_modules/@bufbuild/protobuf": {
+ "version": "2.5.2",
+ "resolved": "https://registry.npmjs.org/@bufbuild/protobuf/-/protobuf-2.5.2.tgz",
+ "integrity": "sha512-foZ7qr0IsUBjzWIq+SuBLfdQCpJ1j8cTuNNT4owngTHoN5KsJb8L9t65fzz7SCeSWzescoOil/0ldqiL041ABg==",
+ "license": "(Apache-2.0 AND BSD-3-Clause)"
+ },
+ "node_modules/@bufbuild/protoplugin": {
+ "version": "2.5.2",
+ "resolved": "https://registry.npmjs.org/@bufbuild/protoplugin/-/protoplugin-2.5.2.tgz",
+ "integrity": "sha512-7d/NUae/ugs/qgHEYOwkVWGDE3Bf/xjuGviVFs38+MLRdwiHNTiuvzPVwuIPo/1wuZCZn3Nax1cg1owLuY72xw==",
+ "license": "Apache-2.0",
+ "dependencies": {
+ "@bufbuild/protobuf": "2.5.2",
+ "@typescript/vfs": "^1.5.2",
+ "typescript": "5.4.5"
+ }
+ },
+ "node_modules/@bufbuild/protoplugin/node_modules/typescript": {
+ "version": "5.4.5",
+ "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.4.5.tgz",
+ "integrity": "sha512-vcI4UpRgg81oIRUFwR0WSIHKt11nJ7SAVlYNIu+QpqeyXP+gpQJy/Z4+F0aGxSE4MqwjyXvW/TzgkLAx2AGHwQ==",
+ "license": "Apache-2.0",
+ "bin": {
+ "tsc": "bin/tsc",
+ "tsserver": "bin/tsserver"
+ },
+ "engines": {
+ "node": ">=14.17"
+ }
+ },
+ "node_modules/@daily-co/daily-js": {
+ "version": "0.79.0",
+ "resolved": "https://registry.npmjs.org/@daily-co/daily-js/-/daily-js-0.79.0.tgz",
+ "integrity": "sha512-Ii/Zi6cfTl2EZBpX8msRPNkkCHcajA+ErXpbN2Xe2KySd1Nb4IzC/QWJlSl9VA9pIlYPQicRTDoZnoym/0uEAw==",
+ "license": "BSD-2-Clause",
+ "dependencies": {
+ "@babel/runtime": "^7.12.5",
+ "@sentry/browser": "^8.33.1",
+ "bowser": "^2.8.1",
+ "dequal": "^2.0.3",
+ "events": "^3.1.0"
+ },
+ "engines": {
+ "node": ">=10.0.0"
+ }
+ },
+ "node_modules/@esbuild/aix-ppc64": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.25.5.tgz",
+ "integrity": "sha512-9o3TMmpmftaCMepOdA5k/yDw8SfInyzWWTjYTFCX3kPSDJMROQTb8jg+h9Cnwnmm1vOzvxN7gIfB5V2ewpjtGA==",
+ "cpu": [
+ "ppc64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "aix"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/android-arm": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.25.5.tgz",
+ "integrity": "sha512-AdJKSPeEHgi7/ZhuIPtcQKr5RQdo6OO2IL87JkianiMYMPbCtot9fxPbrMiBADOWWm3T2si9stAiVsGbTQFkbA==",
+ "cpu": [
+ "arm"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "android"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/android-arm64": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.25.5.tgz",
+ "integrity": "sha512-VGzGhj4lJO+TVGV1v8ntCZWJktV7SGCs3Pn1GRWI1SBFtRALoomm8k5E9Pmwg3HOAal2VDc2F9+PM/rEY6oIDg==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "android"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/android-x64": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.25.5.tgz",
+ "integrity": "sha512-D2GyJT1kjvO//drbRT3Hib9XPwQeWd9vZoBJn+bu/lVsOZ13cqNdDeqIF/xQ5/VmWvMduP6AmXvylO/PIc2isw==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "android"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/darwin-arm64": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.25.5.tgz",
+ "integrity": "sha512-GtaBgammVvdF7aPIgH2jxMDdivezgFu6iKpmT+48+F8Hhg5J/sfnDieg0aeG/jfSvkYQU2/pceFPDKlqZzwnfQ==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/darwin-x64": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.25.5.tgz",
+ "integrity": "sha512-1iT4FVL0dJ76/q1wd7XDsXrSW+oLoquptvh4CLR4kITDtqi2e/xwXwdCVH8hVHU43wgJdsq7Gxuzcs6Iq/7bxQ==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/freebsd-arm64": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.25.5.tgz",
+ "integrity": "sha512-nk4tGP3JThz4La38Uy/gzyXtpkPW8zSAmoUhK9xKKXdBCzKODMc2adkB2+8om9BDYugz+uGV7sLmpTYzvmz6Sw==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "freebsd"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/freebsd-x64": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.25.5.tgz",
+ "integrity": "sha512-PrikaNjiXdR2laW6OIjlbeuCPrPaAl0IwPIaRv+SMV8CiM8i2LqVUHFC1+8eORgWyY7yhQY+2U2fA55mBzReaw==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "freebsd"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/linux-arm": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.25.5.tgz",
+ "integrity": "sha512-cPzojwW2okgh7ZlRpcBEtsX7WBuqbLrNXqLU89GxWbNt6uIg78ET82qifUy3W6OVww6ZWobWub5oqZOVtwolfw==",
+ "cpu": [
+ "arm"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/linux-arm64": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.25.5.tgz",
+ "integrity": "sha512-Z9kfb1v6ZlGbWj8EJk9T6czVEjjq2ntSYLY2cw6pAZl4oKtfgQuS4HOq41M/BcoLPzrUbNd+R4BXFyH//nHxVg==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/linux-ia32": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.25.5.tgz",
+ "integrity": "sha512-sQ7l00M8bSv36GLV95BVAdhJ2QsIbCuCjh/uYrWiMQSUuV+LpXwIqhgJDcvMTj+VsQmqAHL2yYaasENvJ7CDKA==",
+ "cpu": [
+ "ia32"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/linux-loong64": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.25.5.tgz",
+ "integrity": "sha512-0ur7ae16hDUC4OL5iEnDb0tZHDxYmuQyhKhsPBV8f99f6Z9KQM02g33f93rNH5A30agMS46u2HP6qTdEt6Q1kg==",
+ "cpu": [
+ "loong64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/linux-mips64el": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.25.5.tgz",
+ "integrity": "sha512-kB/66P1OsHO5zLz0i6X0RxlQ+3cu0mkxS3TKFvkb5lin6uwZ/ttOkP3Z8lfR9mJOBk14ZwZ9182SIIWFGNmqmg==",
+ "cpu": [
+ "mips64el"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/linux-ppc64": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.25.5.tgz",
+ "integrity": "sha512-UZCmJ7r9X2fe2D6jBmkLBMQetXPXIsZjQJCjgwpVDz+YMcS6oFR27alkgGv3Oqkv07bxdvw7fyB71/olceJhkQ==",
+ "cpu": [
+ "ppc64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/linux-riscv64": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.25.5.tgz",
+ "integrity": "sha512-kTxwu4mLyeOlsVIFPfQo+fQJAV9mh24xL+y+Bm6ej067sYANjyEw1dNHmvoqxJUCMnkBdKpvOn0Ahql6+4VyeA==",
+ "cpu": [
+ "riscv64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/linux-s390x": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.25.5.tgz",
+ "integrity": "sha512-K2dSKTKfmdh78uJ3NcWFiqyRrimfdinS5ErLSn3vluHNeHVnBAFWC8a4X5N+7FgVE1EjXS1QDZbpqZBjfrqMTQ==",
+ "cpu": [
+ "s390x"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/linux-x64": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.25.5.tgz",
+ "integrity": "sha512-uhj8N2obKTE6pSZ+aMUbqq+1nXxNjZIIjCjGLfsWvVpy7gKCOL6rsY1MhRh9zLtUtAI7vpgLMK6DxjO8Qm9lJw==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/netbsd-arm64": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.25.5.tgz",
+ "integrity": "sha512-pwHtMP9viAy1oHPvgxtOv+OkduK5ugofNTVDilIzBLpoWAM16r7b/mxBvfpuQDpRQFMfuVr5aLcn4yveGvBZvw==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "netbsd"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/netbsd-x64": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.25.5.tgz",
+ "integrity": "sha512-WOb5fKrvVTRMfWFNCroYWWklbnXH0Q5rZppjq0vQIdlsQKuw6mdSihwSo4RV/YdQ5UCKKvBy7/0ZZYLBZKIbwQ==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "netbsd"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/openbsd-arm64": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.25.5.tgz",
+ "integrity": "sha512-7A208+uQKgTxHd0G0uqZO8UjK2R0DDb4fDmERtARjSHWxqMTye4Erz4zZafx7Di9Cv+lNHYuncAkiGFySoD+Mw==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "openbsd"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/openbsd-x64": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.25.5.tgz",
+ "integrity": "sha512-G4hE405ErTWraiZ8UiSoesH8DaCsMm0Cay4fsFWOOUcz8b8rC6uCvnagr+gnioEjWn0wC+o1/TAHt+It+MpIMg==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "openbsd"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/sunos-x64": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.25.5.tgz",
+ "integrity": "sha512-l+azKShMy7FxzY0Rj4RCt5VD/q8mG/e+mDivgspo+yL8zW7qEwctQ6YqKX34DTEleFAvCIUviCFX1SDZRSyMQA==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "sunos"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/win32-arm64": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.25.5.tgz",
+ "integrity": "sha512-O2S7SNZzdcFG7eFKgvwUEZ2VG9D/sn/eIiz8XRZ1Q/DO5a3s76Xv0mdBzVM5j5R639lXQmPmSo0iRpHqUUrsxw==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "win32"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/win32-ia32": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.25.5.tgz",
+ "integrity": "sha512-onOJ02pqs9h1iMJ1PQphR+VZv8qBMQ77Klcsqv9CNW2w6yLqoURLcgERAIurY6QE63bbLuqgP9ATqajFLK5AMQ==",
+ "cpu": [
+ "ia32"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "win32"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@esbuild/win32-x64": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.25.5.tgz",
+ "integrity": "sha512-TXv6YnJ8ZMVdX+SXWVBo/0p8LTcrUYngpWjvm91TMjjBQii7Oz11Lw5lbDV5Y0TzuhSJHwiH4hEtC1I42mMS0g==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "win32"
+ ],
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@pipecat-ai/client-js": {
+ "version": "0.4.0",
+ "resolved": "https://registry.npmjs.org/@pipecat-ai/client-js/-/client-js-0.4.0.tgz",
+ "integrity": "sha512-O2EgCqt2cAmp21Z6dXz88zgW845HcsfE//qZghaKOt0Z8xPbhidbVbuOX5iajrYgGRqlnXInYiJ9nN2zY6CUJw==",
+ "license": "BSD-2-Clause",
+ "dependencies": {
+ "@types/events": "^3.0.3",
+ "clone-deep": "^4.0.1",
+ "events": "^3.3.0",
+ "typed-emitter": "^2.1.0",
+ "uuid": "^10.0.0"
+ }
+ },
+ "node_modules/@pipecat-ai/websocket-transport": {
+ "version": "0.4.1",
+ "resolved": "https://registry.npmjs.org/@pipecat-ai/websocket-transport/-/websocket-transport-0.4.1.tgz",
+ "integrity": "sha512-/qdMz1IGV+rJ0qi4UE84XKVZu2VqyIh9J7RgNkzS8nEZiUVwaclrVMjKFgwPqwqKi3ik3h2oucPa/u+8s7Tleg==",
+ "license": "BSD-2-Clause",
+ "dependencies": {
+ "@daily-co/daily-js": "^0.79.0",
+ "@protobuf-ts/plugin": "^2.11.0",
+ "@protobuf-ts/runtime": "^2.11.0"
+ },
+ "peerDependencies": {
+ "@pipecat-ai/client-js": "~0.4.0"
+ }
+ },
+ "node_modules/@protobuf-ts/plugin": {
+ "version": "2.11.0",
+ "resolved": "https://registry.npmjs.org/@protobuf-ts/plugin/-/plugin-2.11.0.tgz",
+ "integrity": "sha512-Y+p4Axrk3thxws4BVSIO+x4CKWH2c8k3K+QPrp6Oq8agdsXPL/uwsMTIdpTdXIzTaUEZFASJL9LU56pob5GTHg==",
+ "license": "Apache-2.0",
+ "dependencies": {
+ "@bufbuild/protobuf": "^2.4.0",
+ "@bufbuild/protoplugin": "^2.4.0",
+ "@protobuf-ts/protoc": "^2.11.0",
+ "@protobuf-ts/runtime": "^2.11.0",
+ "@protobuf-ts/runtime-rpc": "^2.11.0",
+ "typescript": "^3.9"
+ },
+ "bin": {
+ "protoc-gen-dump": "bin/protoc-gen-dump",
+ "protoc-gen-ts": "bin/protoc-gen-ts"
+ }
+ },
+ "node_modules/@protobuf-ts/plugin/node_modules/typescript": {
+ "version": "3.9.10",
+ "resolved": "https://registry.npmjs.org/typescript/-/typescript-3.9.10.tgz",
+ "integrity": "sha512-w6fIxVE/H1PkLKcCPsFqKE7Kv7QUwhU8qQY2MueZXWx5cPZdwFupLgKK3vntcK98BtNHZtAF4LA/yl2a7k8R6Q==",
+ "license": "Apache-2.0",
+ "bin": {
+ "tsc": "bin/tsc",
+ "tsserver": "bin/tsserver"
+ },
+ "engines": {
+ "node": ">=4.2.0"
+ }
+ },
+ "node_modules/@protobuf-ts/protoc": {
+ "version": "2.11.0",
+ "resolved": "https://registry.npmjs.org/@protobuf-ts/protoc/-/protoc-2.11.0.tgz",
+ "integrity": "sha512-GYfmv1rjZ/7MWzUqMszhdXiuoa4Js/j6zCbcxFmeThBBUhbrXdPU42vY+QVCHL9PvAMXO+wEhUfPWYdd1YgnlA==",
+ "license": "Apache-2.0",
+ "bin": {
+ "protoc": "protoc.js"
+ }
+ },
+ "node_modules/@protobuf-ts/runtime": {
+ "version": "2.11.0",
+ "resolved": "https://registry.npmjs.org/@protobuf-ts/runtime/-/runtime-2.11.0.tgz",
+ "integrity": "sha512-DfpRpUiNvPC3Kj48CmlU4HaIEY1Myh++PIumMmohBAk8/k0d2CkxYxJfPyUAxfuUfl97F4AvuCu1gXmfOG7OJQ==",
+ "license": "(Apache-2.0 AND BSD-3-Clause)"
+ },
+ "node_modules/@protobuf-ts/runtime-rpc": {
+ "version": "2.11.0",
+ "resolved": "https://registry.npmjs.org/@protobuf-ts/runtime-rpc/-/runtime-rpc-2.11.0.tgz",
+ "integrity": "sha512-g/oMPym5LjVyCc3nlQc6cHer0R3CyleBos4p7CjRNzdKuH/FlRXzfQYo6EN5uv8vLtn7zEK9Cy4YBKvHStIaag==",
+ "license": "Apache-2.0",
+ "dependencies": {
+ "@protobuf-ts/runtime": "^2.11.0"
+ }
+ },
+ "node_modules/@protobufjs/aspromise": {
+ "version": "1.1.2",
+ "resolved": "https://registry.npmjs.org/@protobufjs/aspromise/-/aspromise-1.1.2.tgz",
+ "integrity": "sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ==",
+ "license": "BSD-3-Clause"
+ },
+ "node_modules/@protobufjs/base64": {
+ "version": "1.1.2",
+ "resolved": "https://registry.npmjs.org/@protobufjs/base64/-/base64-1.1.2.tgz",
+ "integrity": "sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg==",
+ "license": "BSD-3-Clause"
+ },
+ "node_modules/@protobufjs/codegen": {
+ "version": "2.0.4",
+ "resolved": "https://registry.npmjs.org/@protobufjs/codegen/-/codegen-2.0.4.tgz",
+ "integrity": "sha512-YyFaikqM5sH0ziFZCN3xDC7zeGaB/d0IUb9CATugHWbd1FRFwWwt4ld4OYMPWu5a3Xe01mGAULCdqhMlPl29Jg==",
+ "license": "BSD-3-Clause"
+ },
+ "node_modules/@protobufjs/eventemitter": {
+ "version": "1.1.0",
+ "resolved": "https://registry.npmjs.org/@protobufjs/eventemitter/-/eventemitter-1.1.0.tgz",
+ "integrity": "sha512-j9ednRT81vYJ9OfVuXG6ERSTdEL1xVsNgqpkxMsbIabzSo3goCjDIveeGv5d03om39ML71RdmrGNjG5SReBP/Q==",
+ "license": "BSD-3-Clause"
+ },
+ "node_modules/@protobufjs/fetch": {
+ "version": "1.1.0",
+ "resolved": "https://registry.npmjs.org/@protobufjs/fetch/-/fetch-1.1.0.tgz",
+ "integrity": "sha512-lljVXpqXebpsijW71PZaCYeIcE5on1w5DlQy5WH6GLbFryLUrBD4932W/E2BSpfRJWseIL4v/KPgBFxDOIdKpQ==",
+ "license": "BSD-3-Clause",
+ "dependencies": {
+ "@protobufjs/aspromise": "^1.1.1",
+ "@protobufjs/inquire": "^1.1.0"
+ }
+ },
+ "node_modules/@protobufjs/float": {
+ "version": "1.0.2",
+ "resolved": "https://registry.npmjs.org/@protobufjs/float/-/float-1.0.2.tgz",
+ "integrity": "sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ==",
+ "license": "BSD-3-Clause"
+ },
+ "node_modules/@protobufjs/inquire": {
+ "version": "1.1.0",
+ "resolved": "https://registry.npmjs.org/@protobufjs/inquire/-/inquire-1.1.0.tgz",
+ "integrity": "sha512-kdSefcPdruJiFMVSbn801t4vFK7KB/5gd2fYvrxhuJYg8ILrmn9SKSX2tZdV6V+ksulWqS7aXjBcRXl3wHoD9Q==",
+ "license": "BSD-3-Clause"
+ },
+ "node_modules/@protobufjs/path": {
+ "version": "1.1.2",
+ "resolved": "https://registry.npmjs.org/@protobufjs/path/-/path-1.1.2.tgz",
+ "integrity": "sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA==",
+ "license": "BSD-3-Clause"
+ },
+ "node_modules/@protobufjs/pool": {
+ "version": "1.1.0",
+ "resolved": "https://registry.npmjs.org/@protobufjs/pool/-/pool-1.1.0.tgz",
+ "integrity": "sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw==",
+ "license": "BSD-3-Clause"
+ },
+ "node_modules/@protobufjs/utf8": {
+ "version": "1.1.0",
+ "resolved": "https://registry.npmjs.org/@protobufjs/utf8/-/utf8-1.1.0.tgz",
+ "integrity": "sha512-Vvn3zZrhQZkkBE8LSuW3em98c0FwgO4nxzv6OdSxPKJIEKY2bGbHn+mhGIPerzI4twdxaP8/0+06HBpwf345Lw==",
+ "license": "BSD-3-Clause"
+ },
+ "node_modules/@rolldown/pluginutils": {
+ "version": "1.0.0-beta.9",
+ "resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0-beta.9.tgz",
+ "integrity": "sha512-e9MeMtVWo186sgvFFJOPGy7/d2j2mZhLJIdVW0C/xDluuOvymEATqz6zKsP0ZmXGzQtqlyjz5sC1sYQUoJG98w==",
+ "dev": true,
+ "license": "MIT"
+ },
+ "node_modules/@rollup/rollup-android-arm-eabi": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm-eabi/-/rollup-android-arm-eabi-4.42.0.tgz",
+ "integrity": "sha512-gldmAyS9hpj+H6LpRNlcjQWbuKUtb94lodB9uCz71Jm+7BxK1VIOo7y62tZZwxhA7j1ylv/yQz080L5WkS+LoQ==",
+ "cpu": [
+ "arm"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "android"
+ ]
+ },
+ "node_modules/@rollup/rollup-android-arm64": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm64/-/rollup-android-arm64-4.42.0.tgz",
+ "integrity": "sha512-bpRipfTgmGFdCZDFLRvIkSNO1/3RGS74aWkJJTFJBH7h3MRV4UijkaEUeOMbi9wxtxYmtAbVcnMtHTPBhLEkaw==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "android"
+ ]
+ },
+ "node_modules/@rollup/rollup-darwin-arm64": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-arm64/-/rollup-darwin-arm64-4.42.0.tgz",
+ "integrity": "sha512-JxHtA081izPBVCHLKnl6GEA0w3920mlJPLh89NojpU2GsBSB6ypu4erFg/Wx1qbpUbepn0jY4dVWMGZM8gplgA==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "darwin"
+ ]
+ },
+ "node_modules/@rollup/rollup-darwin-x64": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-x64/-/rollup-darwin-x64-4.42.0.tgz",
+ "integrity": "sha512-rv5UZaWVIJTDMyQ3dCEK+m0SAn6G7H3PRc2AZmExvbDvtaDc+qXkei0knQWcI3+c9tEs7iL/4I4pTQoPbNL2SA==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "darwin"
+ ]
+ },
+ "node_modules/@rollup/rollup-freebsd-arm64": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-arm64/-/rollup-freebsd-arm64-4.42.0.tgz",
+ "integrity": "sha512-fJcN4uSGPWdpVmvLuMtALUFwCHgb2XiQjuECkHT3lWLZhSQ3MBQ9pq+WoWeJq2PrNxr9rPM1Qx+IjyGj8/c6zQ==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "freebsd"
+ ]
+ },
+ "node_modules/@rollup/rollup-freebsd-x64": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-x64/-/rollup-freebsd-x64-4.42.0.tgz",
+ "integrity": "sha512-CziHfyzpp8hJpCVE/ZdTizw58gr+m7Y2Xq5VOuCSrZR++th2xWAz4Nqk52MoIIrV3JHtVBhbBsJcAxs6NammOQ==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "freebsd"
+ ]
+ },
+ "node_modules/@rollup/rollup-linux-arm-gnueabihf": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-gnueabihf/-/rollup-linux-arm-gnueabihf-4.42.0.tgz",
+ "integrity": "sha512-UsQD5fyLWm2Fe5CDM7VPYAo+UC7+2Px4Y+N3AcPh/LdZu23YcuGPegQly++XEVaC8XUTFVPscl5y5Cl1twEI4A==",
+ "cpu": [
+ "arm"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ]
+ },
+ "node_modules/@rollup/rollup-linux-arm-musleabihf": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-musleabihf/-/rollup-linux-arm-musleabihf-4.42.0.tgz",
+ "integrity": "sha512-/i8NIrlgc/+4n1lnoWl1zgH7Uo0XK5xK3EDqVTf38KvyYgCU/Rm04+o1VvvzJZnVS5/cWSd07owkzcVasgfIkQ==",
+ "cpu": [
+ "arm"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ]
+ },
+ "node_modules/@rollup/rollup-linux-arm64-gnu": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-gnu/-/rollup-linux-arm64-gnu-4.42.0.tgz",
+ "integrity": "sha512-eoujJFOvoIBjZEi9hJnXAbWg+Vo1Ov8n/0IKZZcPZ7JhBzxh2A+2NFyeMZIRkY9iwBvSjloKgcvnjTbGKHE44Q==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ]
+ },
+ "node_modules/@rollup/rollup-linux-arm64-musl": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-musl/-/rollup-linux-arm64-musl-4.42.0.tgz",
+ "integrity": "sha512-/3NrcOWFSR7RQUQIuZQChLND36aTU9IYE4j+TB40VU78S+RA0IiqHR30oSh6P1S9f9/wVOenHQnacs/Byb824g==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ]
+ },
+ "node_modules/@rollup/rollup-linux-loongarch64-gnu": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-loongarch64-gnu/-/rollup-linux-loongarch64-gnu-4.42.0.tgz",
+ "integrity": "sha512-O8AplvIeavK5ABmZlKBq9/STdZlnQo7Sle0LLhVA7QT+CiGpNVe197/t8Aph9bhJqbDVGCHpY2i7QyfEDDStDg==",
+ "cpu": [
+ "loong64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ]
+ },
+ "node_modules/@rollup/rollup-linux-powerpc64le-gnu": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-powerpc64le-gnu/-/rollup-linux-powerpc64le-gnu-4.42.0.tgz",
+ "integrity": "sha512-6Qb66tbKVN7VyQrekhEzbHRxXXFFD8QKiFAwX5v9Xt6FiJ3BnCVBuyBxa2fkFGqxOCSGGYNejxd8ht+q5SnmtA==",
+ "cpu": [
+ "ppc64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ]
+ },
+ "node_modules/@rollup/rollup-linux-riscv64-gnu": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-gnu/-/rollup-linux-riscv64-gnu-4.42.0.tgz",
+ "integrity": "sha512-KQETDSEBamQFvg/d8jajtRwLNBlGc3aKpaGiP/LvEbnmVUKlFta1vqJqTrvPtsYsfbE/DLg5CC9zyXRX3fnBiA==",
+ "cpu": [
+ "riscv64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ]
+ },
+ "node_modules/@rollup/rollup-linux-riscv64-musl": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-musl/-/rollup-linux-riscv64-musl-4.42.0.tgz",
+ "integrity": "sha512-qMvnyjcU37sCo/tuC+JqeDKSuukGAd+pVlRl/oyDbkvPJ3awk6G6ua7tyum02O3lI+fio+eM5wsVd66X0jQtxw==",
+ "cpu": [
+ "riscv64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ]
+ },
+ "node_modules/@rollup/rollup-linux-s390x-gnu": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-s390x-gnu/-/rollup-linux-s390x-gnu-4.42.0.tgz",
+ "integrity": "sha512-I2Y1ZUgTgU2RLddUHXTIgyrdOwljjkmcZ/VilvaEumtS3Fkuhbw4p4hgHc39Ypwvo2o7sBFNl2MquNvGCa55Iw==",
+ "cpu": [
+ "s390x"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ]
+ },
+ "node_modules/@rollup/rollup-linux-x64-gnu": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-gnu/-/rollup-linux-x64-gnu-4.42.0.tgz",
+ "integrity": "sha512-Gfm6cV6mj3hCUY8TqWa63DB8Mx3NADoFwiJrMpoZ1uESbK8FQV3LXkhfry+8bOniq9pqY1OdsjFWNsSbfjPugw==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ]
+ },
+ "node_modules/@rollup/rollup-linux-x64-musl": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-musl/-/rollup-linux-x64-musl-4.42.0.tgz",
+ "integrity": "sha512-g86PF8YZ9GRqkdi0VoGlcDUb4rYtQKyTD1IVtxxN4Hpe7YqLBShA7oHMKU6oKTCi3uxwW4VkIGnOaH/El8de3w==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ]
+ },
+ "node_modules/@rollup/rollup-win32-arm64-msvc": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-arm64-msvc/-/rollup-win32-arm64-msvc-4.42.0.tgz",
+ "integrity": "sha512-+axkdyDGSp6hjyzQ5m1pgcvQScfHnMCcsXkx8pTgy/6qBmWVhtRVlgxjWwDp67wEXXUr0x+vD6tp5W4x6V7u1A==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "win32"
+ ]
+ },
+ "node_modules/@rollup/rollup-win32-ia32-msvc": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-ia32-msvc/-/rollup-win32-ia32-msvc-4.42.0.tgz",
+ "integrity": "sha512-F+5J9pelstXKwRSDq92J0TEBXn2nfUrQGg+HK1+Tk7VOL09e0gBqUHugZv7SW4MGrYj41oNCUe3IKCDGVlis2g==",
+ "cpu": [
+ "ia32"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "win32"
+ ]
+ },
+ "node_modules/@rollup/rollup-win32-x64-msvc": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-msvc/-/rollup-win32-x64-msvc-4.42.0.tgz",
+ "integrity": "sha512-LpHiJRwkaVz/LqjHjK8LCi8osq7elmpwujwbXKNW88bM8eeGxavJIKKjkjpMHAh/2xfnrt1ZSnhTv41WYUHYmA==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "win32"
+ ]
+ },
+ "node_modules/@sentry-internal/browser-utils": {
+ "version": "8.55.0",
+ "resolved": "https://registry.npmjs.org/@sentry-internal/browser-utils/-/browser-utils-8.55.0.tgz",
+ "integrity": "sha512-ROgqtQfpH/82AQIpESPqPQe0UyWywKJsmVIqi3c5Fh+zkds5LUxnssTj3yNd1x+kxaPDVB023jAP+3ibNgeNDw==",
+ "license": "MIT",
+ "dependencies": {
+ "@sentry/core": "8.55.0"
+ },
+ "engines": {
+ "node": ">=14.18"
+ }
+ },
+ "node_modules/@sentry-internal/feedback": {
+ "version": "8.55.0",
+ "resolved": "https://registry.npmjs.org/@sentry-internal/feedback/-/feedback-8.55.0.tgz",
+ "integrity": "sha512-cP3BD/Q6pquVQ+YL+rwCnorKuTXiS9KXW8HNKu4nmmBAyf7urjs+F6Hr1k9MXP5yQ8W3yK7jRWd09Yu6DHWOiw==",
+ "license": "MIT",
+ "dependencies": {
+ "@sentry/core": "8.55.0"
+ },
+ "engines": {
+ "node": ">=14.18"
+ }
+ },
+ "node_modules/@sentry-internal/replay": {
+ "version": "8.55.0",
+ "resolved": "https://registry.npmjs.org/@sentry-internal/replay/-/replay-8.55.0.tgz",
+ "integrity": "sha512-roCDEGkORwolxBn8xAKedybY+Jlefq3xYmgN2fr3BTnsXjSYOPC7D1/mYqINBat99nDtvgFvNfRcZPiwwZ1hSw==",
+ "license": "MIT",
+ "dependencies": {
+ "@sentry-internal/browser-utils": "8.55.0",
+ "@sentry/core": "8.55.0"
+ },
+ "engines": {
+ "node": ">=14.18"
+ }
+ },
+ "node_modules/@sentry-internal/replay-canvas": {
+ "version": "8.55.0",
+ "resolved": "https://registry.npmjs.org/@sentry-internal/replay-canvas/-/replay-canvas-8.55.0.tgz",
+ "integrity": "sha512-nIkfgRWk1091zHdu4NbocQsxZF1rv1f7bbp3tTIlZYbrH62XVZosx5iHAuZG0Zc48AETLE7K4AX9VGjvQj8i9w==",
+ "license": "MIT",
+ "dependencies": {
+ "@sentry-internal/replay": "8.55.0",
+ "@sentry/core": "8.55.0"
+ },
+ "engines": {
+ "node": ">=14.18"
+ }
+ },
+ "node_modules/@sentry/browser": {
+ "version": "8.55.0",
+ "resolved": "https://registry.npmjs.org/@sentry/browser/-/browser-8.55.0.tgz",
+ "integrity": "sha512-1A31mCEWCjaMxJt6qGUK+aDnLDcK6AwLAZnqpSchNysGni1pSn1RWSmk9TBF8qyTds5FH8B31H480uxMPUJ7Cw==",
+ "license": "MIT",
+ "dependencies": {
+ "@sentry-internal/browser-utils": "8.55.0",
+ "@sentry-internal/feedback": "8.55.0",
+ "@sentry-internal/replay": "8.55.0",
+ "@sentry-internal/replay-canvas": "8.55.0",
+ "@sentry/core": "8.55.0"
+ },
+ "engines": {
+ "node": ">=14.18"
+ }
+ },
+ "node_modules/@sentry/core": {
+ "version": "8.55.0",
+ "resolved": "https://registry.npmjs.org/@sentry/core/-/core-8.55.0.tgz",
+ "integrity": "sha512-6g7jpbefjHYs821Z+EBJ8r4Z7LT5h80YSWRJaylGS4nW5W5Z2KXzpdnyFarv37O7QjauzVC2E+PABmpkw5/JGA==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=14.18"
+ }
+ },
+ "node_modules/@swc/core": {
+ "version": "1.11.31",
+ "resolved": "https://registry.npmjs.org/@swc/core/-/core-1.11.31.tgz",
+ "integrity": "sha512-mAby9aUnKRjMEA7v8cVZS9Ah4duoRBnX7X6r5qrhTxErx+68MoY1TPrVwj/66/SWN3Bl+jijqAqoB8Qx0QE34A==",
+ "dev": true,
+ "hasInstallScript": true,
+ "license": "Apache-2.0",
+ "dependencies": {
+ "@swc/counter": "^0.1.3",
+ "@swc/types": "^0.1.21"
+ },
+ "engines": {
+ "node": ">=10"
+ },
+ "funding": {
+ "type": "opencollective",
+ "url": "https://opencollective.com/swc"
+ },
+ "optionalDependencies": {
+ "@swc/core-darwin-arm64": "1.11.31",
+ "@swc/core-darwin-x64": "1.11.31",
+ "@swc/core-linux-arm-gnueabihf": "1.11.31",
+ "@swc/core-linux-arm64-gnu": "1.11.31",
+ "@swc/core-linux-arm64-musl": "1.11.31",
+ "@swc/core-linux-x64-gnu": "1.11.31",
+ "@swc/core-linux-x64-musl": "1.11.31",
+ "@swc/core-win32-arm64-msvc": "1.11.31",
+ "@swc/core-win32-ia32-msvc": "1.11.31",
+ "@swc/core-win32-x64-msvc": "1.11.31"
+ },
+ "peerDependencies": {
+ "@swc/helpers": ">=0.5.17"
+ },
+ "peerDependenciesMeta": {
+ "@swc/helpers": {
+ "optional": true
+ }
+ }
+ },
+ "node_modules/@swc/core-darwin-arm64": {
+ "version": "1.11.31",
+ "resolved": "https://registry.npmjs.org/@swc/core-darwin-arm64/-/core-darwin-arm64-1.11.31.tgz",
+ "integrity": "sha512-NTEaYOts0OGSbJZc0O74xsji+64JrF1stmBii6D5EevWEtrY4wlZhm8SiP/qPrOB+HqtAihxWIukWkP2aSdGSQ==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "Apache-2.0 AND MIT",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "engines": {
+ "node": ">=10"
+ }
+ },
+ "node_modules/@swc/core-darwin-x64": {
+ "version": "1.11.31",
+ "resolved": "https://registry.npmjs.org/@swc/core-darwin-x64/-/core-darwin-x64-1.11.31.tgz",
+ "integrity": "sha512-THSGaSwT96JwXDwuXQ6yFBbn+xDMdyw7OmBpnweAWsh5DhZmQkALEm1DgdQO3+rrE99MkmzwAfclc0UmYro/OA==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "Apache-2.0 AND MIT",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "engines": {
+ "node": ">=10"
+ }
+ },
+ "node_modules/@swc/core-linux-arm-gnueabihf": {
+ "version": "1.11.31",
+ "resolved": "https://registry.npmjs.org/@swc/core-linux-arm-gnueabihf/-/core-linux-arm-gnueabihf-1.11.31.tgz",
+ "integrity": "sha512-laKtQFnW7KHgE57Hx32os2SNAogcuIDxYE+3DYIOmDMqD7/1DCfJe6Rln2N9WcOw6HuDbDpyQavIwZNfSAa8vQ==",
+ "cpu": [
+ "arm"
+ ],
+ "dev": true,
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": ">=10"
+ }
+ },
+ "node_modules/@swc/core-linux-arm64-gnu": {
+ "version": "1.11.31",
+ "resolved": "https://registry.npmjs.org/@swc/core-linux-arm64-gnu/-/core-linux-arm64-gnu-1.11.31.tgz",
+ "integrity": "sha512-T+vGw9aPE1YVyRxRr1n7NAdkbgzBzrXCCJ95xAZc/0+WUwmL77Z+js0J5v1KKTRxw4FvrslNCOXzMWrSLdwPSA==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "Apache-2.0 AND MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": ">=10"
+ }
+ },
+ "node_modules/@swc/core-linux-arm64-musl": {
+ "version": "1.11.31",
+ "resolved": "https://registry.npmjs.org/@swc/core-linux-arm64-musl/-/core-linux-arm64-musl-1.11.31.tgz",
+ "integrity": "sha512-Mztp5NZkyd5MrOAG+kl+QSn0lL4Uawd4CK4J7wm97Hs44N9DHGIG5nOz7Qve1KZo407Y25lTxi/PqzPKHo61zQ==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "Apache-2.0 AND MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": ">=10"
+ }
+ },
+ "node_modules/@swc/core-linux-x64-gnu": {
+ "version": "1.11.31",
+ "resolved": "https://registry.npmjs.org/@swc/core-linux-x64-gnu/-/core-linux-x64-gnu-1.11.31.tgz",
+ "integrity": "sha512-DDVE0LZcXOWwOqFU1Xi7gdtiUg3FHA0vbGb3trjWCuI1ZtDZHEQYL4M3/2FjqKZtIwASrDvO96w91okZbXhvMg==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "Apache-2.0 AND MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": ">=10"
+ }
+ },
+ "node_modules/@swc/core-linux-x64-musl": {
+ "version": "1.11.31",
+ "resolved": "https://registry.npmjs.org/@swc/core-linux-x64-musl/-/core-linux-x64-musl-1.11.31.tgz",
+ "integrity": "sha512-mJA1MzPPRIfaBUHZi0xJQ4vwL09MNWDeFtxXb0r4Yzpf0v5Lue9ymumcBPmw/h6TKWms+Non4+TDquAsweuKSw==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "Apache-2.0 AND MIT",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": ">=10"
+ }
+ },
+ "node_modules/@swc/core-win32-arm64-msvc": {
+ "version": "1.11.31",
+ "resolved": "https://registry.npmjs.org/@swc/core-win32-arm64-msvc/-/core-win32-arm64-msvc-1.11.31.tgz",
+ "integrity": "sha512-RdtakUkNVAb/FFIMw3LnfNdlH1/ep6KgiPDRlmyUfd0WdIQ3OACmeBegEFNFTzi7gEuzy2Yxg4LWf4IUVk8/bg==",
+ "cpu": [
+ "arm64"
+ ],
+ "dev": true,
+ "license": "Apache-2.0 AND MIT",
+ "optional": true,
+ "os": [
+ "win32"
+ ],
+ "engines": {
+ "node": ">=10"
+ }
+ },
+ "node_modules/@swc/core-win32-ia32-msvc": {
+ "version": "1.11.31",
+ "resolved": "https://registry.npmjs.org/@swc/core-win32-ia32-msvc/-/core-win32-ia32-msvc-1.11.31.tgz",
+ "integrity": "sha512-hErXdCGsg7swWdG1fossuL8542I59xV+all751mYlBoZ8kOghLSKObGQTkBbuNvc0sUKWfWg1X0iBuIhAYar+w==",
+ "cpu": [
+ "ia32"
+ ],
+ "dev": true,
+ "license": "Apache-2.0 AND MIT",
+ "optional": true,
+ "os": [
+ "win32"
+ ],
+ "engines": {
+ "node": ">=10"
+ }
+ },
+ "node_modules/@swc/core-win32-x64-msvc": {
+ "version": "1.11.31",
+ "resolved": "https://registry.npmjs.org/@swc/core-win32-x64-msvc/-/core-win32-x64-msvc-1.11.31.tgz",
+ "integrity": "sha512-5t7SGjUBMMhF9b5j17ml/f/498kiBJNf4vZFNM421UGUEETdtjPN9jZIuQrowBkoFGJTCVL/ECM4YRtTH30u/A==",
+ "cpu": [
+ "x64"
+ ],
+ "dev": true,
+ "license": "Apache-2.0 AND MIT",
+ "optional": true,
+ "os": [
+ "win32"
+ ],
+ "engines": {
+ "node": ">=10"
+ }
+ },
+ "node_modules/@swc/counter": {
+ "version": "0.1.3",
+ "resolved": "https://registry.npmjs.org/@swc/counter/-/counter-0.1.3.tgz",
+ "integrity": "sha512-e2BR4lsJkkRlKZ/qCHPw9ZaSxc0MVUd7gtbtaB7aMvHeJVYe8sOB8DBZkP2DtISHGSku9sCK6T6cnY0CtXrOCQ==",
+ "dev": true,
+ "license": "Apache-2.0"
+ },
+ "node_modules/@swc/types": {
+ "version": "0.1.22",
+ "resolved": "https://registry.npmjs.org/@swc/types/-/types-0.1.22.tgz",
+ "integrity": "sha512-D13mY/ZA4PPEFSy6acki9eBT/3WgjMoRqNcdpIvjaYLQ44Xk5BdaL7UkDxAh6Z9UOe7tCCp67BVmZCojYp9owg==",
+ "dev": true,
+ "license": "Apache-2.0",
+ "dependencies": {
+ "@swc/counter": "^0.1.3"
+ }
+ },
+ "node_modules/@types/estree": {
+ "version": "1.0.7",
+ "resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.7.tgz",
+ "integrity": "sha512-w28IoSUCJpidD/TGviZwwMJckNESJZXFu7NBZ5YJ4mEUnNraUn9Pm8HSZm/jDF1pDWYKspWE7oVphigUPRakIQ==",
+ "dev": true,
+ "license": "MIT"
+ },
+ "node_modules/@types/events": {
+ "version": "3.0.3",
+ "resolved": "https://registry.npmjs.org/@types/events/-/events-3.0.3.tgz",
+ "integrity": "sha512-trOc4AAUThEz9hapPtSd7wf5tiQKvTtu5b371UxXdTuqzIh0ArcRspRP0i0Viu+LXstIQ1z96t1nsPxT9ol01g==",
+ "license": "MIT"
+ },
+ "node_modules/@types/node": {
+ "version": "22.15.30",
+ "resolved": "https://registry.npmjs.org/@types/node/-/node-22.15.30.tgz",
+ "integrity": "sha512-6Q7lr06bEHdlfplU6YRbgG1SFBdlsfNC4/lX+SkhiTs0cpJkOElmWls8PxDFv4yY/xKb8Y6SO0OmSX4wgqTZbA==",
+ "license": "MIT",
+ "dependencies": {
+ "undici-types": "~6.21.0"
+ }
+ },
+ "node_modules/@types/protobufjs": {
+ "version": "6.0.0",
+ "resolved": "https://registry.npmjs.org/@types/protobufjs/-/protobufjs-6.0.0.tgz",
+ "integrity": "sha512-A27RDExpAf3rdDjIrHKiJK6x8kqqJ4CmoChwtipfhVAn1p7+wviQFFP7dppn8FslSbHtQeVPvi8wNKkDjSYjHw==",
+ "deprecated": "This is a stub types definition for protobufjs (https://github.com/dcodeIO/ProtoBuf.js). protobufjs provides its own type definitions, so you don't need @types/protobufjs installed!",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "protobufjs": "*"
+ }
+ },
+ "node_modules/@typescript/vfs": {
+ "version": "1.6.1",
+ "resolved": "https://registry.npmjs.org/@typescript/vfs/-/vfs-1.6.1.tgz",
+ "integrity": "sha512-JwoxboBh7Oz1v38tPbkrZ62ZXNHAk9bJ7c9x0eI5zBfBnBYGhURdbnh7Z4smN/MV48Y5OCcZb58n972UtbazsA==",
+ "license": "MIT",
+ "dependencies": {
+ "debug": "^4.1.1"
+ },
+ "peerDependencies": {
+ "typescript": "*"
+ }
+ },
+ "node_modules/@vitejs/plugin-react-swc": {
+ "version": "3.10.1",
+ "resolved": "https://registry.npmjs.org/@vitejs/plugin-react-swc/-/plugin-react-swc-3.10.1.tgz",
+ "integrity": "sha512-FmQvN3yZGyD9XW6IyxE86Kaa/DnxSsrDQX1xCR1qojNpBLaUop+nLYFvhCkJsq8zOupNjCRA9jyhPGOJsSkutA==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "@rolldown/pluginutils": "1.0.0-beta.9",
+ "@swc/core": "^1.11.22"
+ },
+ "peerDependencies": {
+ "vite": "^4 || ^5 || ^6"
+ }
+ },
+ "node_modules/bowser": {
+ "version": "2.11.0",
+ "resolved": "https://registry.npmjs.org/bowser/-/bowser-2.11.0.tgz",
+ "integrity": "sha512-AlcaJBi/pqqJBIQ8U9Mcpc9i8Aqxn88Skv5d+xBX006BY5u8N3mGLHa5Lgppa7L/HfwgwLgZ6NYs+Ag6uUmJRA==",
+ "license": "MIT"
+ },
+ "node_modules/clone-deep": {
+ "version": "4.0.1",
+ "resolved": "https://registry.npmjs.org/clone-deep/-/clone-deep-4.0.1.tgz",
+ "integrity": "sha512-neHB9xuzh/wk0dIHweyAXv2aPGZIVk3pLMe+/RNzINf17fe0OG96QroktYAUm7SM1PBnzTabaLboqqxDyMU+SQ==",
+ "license": "MIT",
+ "dependencies": {
+ "is-plain-object": "^2.0.4",
+ "kind-of": "^6.0.2",
+ "shallow-clone": "^3.0.0"
+ },
+ "engines": {
+ "node": ">=6"
+ }
+ },
+ "node_modules/debug": {
+ "version": "4.4.1",
+ "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.1.tgz",
+ "integrity": "sha512-KcKCqiftBJcZr++7ykoDIEwSa3XWowTfNPo92BYxjXiyYEVrUQh2aLyhxBCwww+heortUFxEJYcRzosstTEBYQ==",
+ "license": "MIT",
+ "dependencies": {
+ "ms": "^2.1.3"
+ },
+ "engines": {
+ "node": ">=6.0"
+ },
+ "peerDependenciesMeta": {
+ "supports-color": {
+ "optional": true
+ }
+ }
+ },
+ "node_modules/dequal": {
+ "version": "2.0.3",
+ "resolved": "https://registry.npmjs.org/dequal/-/dequal-2.0.3.tgz",
+ "integrity": "sha512-0je+qPKHEMohvfRTCEo3CrPG6cAzAYgmzKyxRiYSSDkS6eGJdyVJm7WaYA5ECaAD9wLB2T4EEeymA5aFVcYXCA==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=6"
+ }
+ },
+ "node_modules/esbuild": {
+ "version": "0.25.5",
+ "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.25.5.tgz",
+ "integrity": "sha512-P8OtKZRv/5J5hhz0cUAdu/cLuPIKXpQl1R9pZtvmHWQvrAUVd0UNIPT4IB4W3rNOqVO0rlqHmCIbSwxh/c9yUQ==",
+ "dev": true,
+ "hasInstallScript": true,
+ "license": "MIT",
+ "bin": {
+ "esbuild": "bin/esbuild"
+ },
+ "engines": {
+ "node": ">=18"
+ },
+ "optionalDependencies": {
+ "@esbuild/aix-ppc64": "0.25.5",
+ "@esbuild/android-arm": "0.25.5",
+ "@esbuild/android-arm64": "0.25.5",
+ "@esbuild/android-x64": "0.25.5",
+ "@esbuild/darwin-arm64": "0.25.5",
+ "@esbuild/darwin-x64": "0.25.5",
+ "@esbuild/freebsd-arm64": "0.25.5",
+ "@esbuild/freebsd-x64": "0.25.5",
+ "@esbuild/linux-arm": "0.25.5",
+ "@esbuild/linux-arm64": "0.25.5",
+ "@esbuild/linux-ia32": "0.25.5",
+ "@esbuild/linux-loong64": "0.25.5",
+ "@esbuild/linux-mips64el": "0.25.5",
+ "@esbuild/linux-ppc64": "0.25.5",
+ "@esbuild/linux-riscv64": "0.25.5",
+ "@esbuild/linux-s390x": "0.25.5",
+ "@esbuild/linux-x64": "0.25.5",
+ "@esbuild/netbsd-arm64": "0.25.5",
+ "@esbuild/netbsd-x64": "0.25.5",
+ "@esbuild/openbsd-arm64": "0.25.5",
+ "@esbuild/openbsd-x64": "0.25.5",
+ "@esbuild/sunos-x64": "0.25.5",
+ "@esbuild/win32-arm64": "0.25.5",
+ "@esbuild/win32-ia32": "0.25.5",
+ "@esbuild/win32-x64": "0.25.5"
+ }
+ },
+ "node_modules/events": {
+ "version": "3.3.0",
+ "resolved": "https://registry.npmjs.org/events/-/events-3.3.0.tgz",
+ "integrity": "sha512-mQw+2fkQbALzQ7V0MY0IqdnXNOeTtP4r0lN9z7AAawCXgqea7bDii20AYrIBrFd/Hx0M2Ocz6S111CaFkUcb0Q==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.8.x"
+ }
+ },
+ "node_modules/fdir": {
+ "version": "6.4.5",
+ "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.4.5.tgz",
+ "integrity": "sha512-4BG7puHpVsIYxZUbiUE3RqGloLaSSwzYie5jvasC4LWuBWzZawynvYouhjbQKw2JuIGYdm0DzIxl8iVidKlUEw==",
+ "dev": true,
+ "license": "MIT",
+ "peerDependencies": {
+ "picomatch": "^3 || ^4"
+ },
+ "peerDependenciesMeta": {
+ "picomatch": {
+ "optional": true
+ }
+ }
+ },
+ "node_modules/fsevents": {
+ "version": "2.3.3",
+ "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz",
+ "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==",
+ "dev": true,
+ "hasInstallScript": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "engines": {
+ "node": "^8.16.0 || ^10.6.0 || >=11.0.0"
+ }
+ },
+ "node_modules/is-plain-object": {
+ "version": "2.0.4",
+ "resolved": "https://registry.npmjs.org/is-plain-object/-/is-plain-object-2.0.4.tgz",
+ "integrity": "sha512-h5PpgXkWitc38BBMYawTYMWJHFZJVnBquFE57xFpjB8pJFiF6gZ+bU+WyI/yqXiFR5mdLsgYNaPe8uao6Uv9Og==",
+ "license": "MIT",
+ "dependencies": {
+ "isobject": "^3.0.1"
+ },
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/isobject": {
+ "version": "3.0.1",
+ "resolved": "https://registry.npmjs.org/isobject/-/isobject-3.0.1.tgz",
+ "integrity": "sha512-WhB9zCku7EGTj/HQQRz5aUQEUeoQZH2bWcltRErOpymJ4boYE6wL9Tbr23krRPSZ+C5zqNSrSw+Cc7sZZ4b7vg==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/kind-of": {
+ "version": "6.0.3",
+ "resolved": "https://registry.npmjs.org/kind-of/-/kind-of-6.0.3.tgz",
+ "integrity": "sha512-dcS1ul+9tmeD95T+x28/ehLgd9mENa3LsvDTtzm3vyBEO7RPptvAD+t44WVXaUjTBRcrpFeFlC8WCruUR456hw==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/long": {
+ "version": "5.2.4",
+ "resolved": "https://registry.npmjs.org/long/-/long-5.2.4.tgz",
+ "integrity": "sha512-qtzLbJE8hq7VabR3mISmVGtoXP8KGc2Z/AT8OuqlYD7JTR3oqrgwdjnk07wpj1twXxYmgDXgoKVWUG/fReSzHg==",
+ "license": "Apache-2.0"
+ },
+ "node_modules/ms": {
+ "version": "2.1.3",
+ "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
+ "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==",
+ "license": "MIT"
+ },
+ "node_modules/nanoid": {
+ "version": "3.3.11",
+ "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.11.tgz",
+ "integrity": "sha512-N8SpfPUnUp1bK+PMYW8qSWdl9U+wwNWI4QKxOYDy9JAro3WMX7p2OeVRF9v+347pnakNevPmiHhNmZ2HbFA76w==",
+ "dev": true,
+ "funding": [
+ {
+ "type": "github",
+ "url": "https://github.com/sponsors/ai"
+ }
+ ],
+ "license": "MIT",
+ "bin": {
+ "nanoid": "bin/nanoid.cjs"
+ },
+ "engines": {
+ "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1"
+ }
+ },
+ "node_modules/picocolors": {
+ "version": "1.1.1",
+ "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz",
+ "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==",
+ "dev": true,
+ "license": "ISC"
+ },
+ "node_modules/picomatch": {
+ "version": "4.0.2",
+ "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.2.tgz",
+ "integrity": "sha512-M7BAV6Rlcy5u+m6oPhAPFgJTzAioX/6B0DxyvDlo9l8+T3nLKbrczg2WLUyzd45L8RqfUMyGPzekbMvX2Ldkwg==",
+ "dev": true,
+ "license": "MIT",
+ "engines": {
+ "node": ">=12"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/jonschlinkert"
+ }
+ },
+ "node_modules/postcss": {
+ "version": "8.5.4",
+ "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.4.tgz",
+ "integrity": "sha512-QSa9EBe+uwlGTFmHsPKokv3B/oEMQZxfqW0QqNCyhpa6mB1afzulwn8hihglqAb2pOw+BJgNlmXQ8la2VeHB7w==",
+ "dev": true,
+ "funding": [
+ {
+ "type": "opencollective",
+ "url": "https://opencollective.com/postcss/"
+ },
+ {
+ "type": "tidelift",
+ "url": "https://tidelift.com/funding/github/npm/postcss"
+ },
+ {
+ "type": "github",
+ "url": "https://github.com/sponsors/ai"
+ }
+ ],
+ "license": "MIT",
+ "dependencies": {
+ "nanoid": "^3.3.11",
+ "picocolors": "^1.1.1",
+ "source-map-js": "^1.2.1"
+ },
+ "engines": {
+ "node": "^10 || ^12 || >=14"
+ }
+ },
+ "node_modules/protobufjs": {
+ "version": "7.4.0",
+ "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-7.4.0.tgz",
+ "integrity": "sha512-mRUWCc3KUU4w1jU8sGxICXH/gNS94DvI1gxqDvBzhj1JpcsimQkYiOJfwsPUykUI5ZaspFbSgmBLER8IrQ3tqw==",
+ "hasInstallScript": true,
+ "license": "BSD-3-Clause",
+ "dependencies": {
+ "@protobufjs/aspromise": "^1.1.2",
+ "@protobufjs/base64": "^1.1.2",
+ "@protobufjs/codegen": "^2.0.4",
+ "@protobufjs/eventemitter": "^1.1.0",
+ "@protobufjs/fetch": "^1.1.0",
+ "@protobufjs/float": "^1.0.2",
+ "@protobufjs/inquire": "^1.1.0",
+ "@protobufjs/path": "^1.1.2",
+ "@protobufjs/pool": "^1.1.0",
+ "@protobufjs/utf8": "^1.1.0",
+ "@types/node": ">=13.7.0",
+ "long": "^5.0.0"
+ },
+ "engines": {
+ "node": ">=12.0.0"
+ }
+ },
+ "node_modules/rollup": {
+ "version": "4.42.0",
+ "resolved": "https://registry.npmjs.org/rollup/-/rollup-4.42.0.tgz",
+ "integrity": "sha512-LW+Vse3BJPyGJGAJt1j8pWDKPd73QM8cRXYK1IxOBgL2AGLu7Xd2YOW0M2sLUBCkF5MshXXtMApyEAEzMVMsnw==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "@types/estree": "1.0.7"
+ },
+ "bin": {
+ "rollup": "dist/bin/rollup"
+ },
+ "engines": {
+ "node": ">=18.0.0",
+ "npm": ">=8.0.0"
+ },
+ "optionalDependencies": {
+ "@rollup/rollup-android-arm-eabi": "4.42.0",
+ "@rollup/rollup-android-arm64": "4.42.0",
+ "@rollup/rollup-darwin-arm64": "4.42.0",
+ "@rollup/rollup-darwin-x64": "4.42.0",
+ "@rollup/rollup-freebsd-arm64": "4.42.0",
+ "@rollup/rollup-freebsd-x64": "4.42.0",
+ "@rollup/rollup-linux-arm-gnueabihf": "4.42.0",
+ "@rollup/rollup-linux-arm-musleabihf": "4.42.0",
+ "@rollup/rollup-linux-arm64-gnu": "4.42.0",
+ "@rollup/rollup-linux-arm64-musl": "4.42.0",
+ "@rollup/rollup-linux-loongarch64-gnu": "4.42.0",
+ "@rollup/rollup-linux-powerpc64le-gnu": "4.42.0",
+ "@rollup/rollup-linux-riscv64-gnu": "4.42.0",
+ "@rollup/rollup-linux-riscv64-musl": "4.42.0",
+ "@rollup/rollup-linux-s390x-gnu": "4.42.0",
+ "@rollup/rollup-linux-x64-gnu": "4.42.0",
+ "@rollup/rollup-linux-x64-musl": "4.42.0",
+ "@rollup/rollup-win32-arm64-msvc": "4.42.0",
+ "@rollup/rollup-win32-ia32-msvc": "4.42.0",
+ "@rollup/rollup-win32-x64-msvc": "4.42.0",
+ "fsevents": "~2.3.2"
+ }
+ },
+ "node_modules/rxjs": {
+ "version": "7.8.1",
+ "resolved": "https://registry.npmjs.org/rxjs/-/rxjs-7.8.1.tgz",
+ "integrity": "sha512-AA3TVj+0A2iuIoQkWEK/tqFjBq2j+6PO6Y0zJcvzLAFhEFIO3HL0vls9hWLncZbAAbK0mar7oZ4V079I/qPMxg==",
+ "license": "Apache-2.0",
+ "optional": true,
+ "dependencies": {
+ "tslib": "^2.1.0"
+ }
+ },
+ "node_modules/shallow-clone": {
+ "version": "3.0.1",
+ "resolved": "https://registry.npmjs.org/shallow-clone/-/shallow-clone-3.0.1.tgz",
+ "integrity": "sha512-/6KqX+GVUdqPuPPd2LxDDxzX6CAbjJehAAOKlNpqqUpAqPM6HeL8f+o3a+JsyGjn2lv0WY8UsTgUJjU9Ok55NA==",
+ "license": "MIT",
+ "dependencies": {
+ "kind-of": "^6.0.2"
+ },
+ "engines": {
+ "node": ">=8"
+ }
+ },
+ "node_modules/source-map-js": {
+ "version": "1.2.1",
+ "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz",
+ "integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==",
+ "dev": true,
+ "license": "BSD-3-Clause",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/tinyglobby": {
+ "version": "0.2.14",
+ "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.14.tgz",
+ "integrity": "sha512-tX5e7OM1HnYr2+a2C/4V0htOcSQcoSTH9KgJnVvNm5zm/cyEWKJ7j7YutsH9CxMdtOkkLFy2AHrMci9IM8IPZQ==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "fdir": "^6.4.4",
+ "picomatch": "^4.0.2"
+ },
+ "engines": {
+ "node": ">=12.0.0"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/SuperchupuDev"
+ }
+ },
+ "node_modules/tslib": {
+ "version": "2.8.1",
+ "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz",
+ "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==",
+ "license": "0BSD",
+ "optional": true
+ },
+ "node_modules/typed-emitter": {
+ "version": "2.1.0",
+ "resolved": "https://registry.npmjs.org/typed-emitter/-/typed-emitter-2.1.0.tgz",
+ "integrity": "sha512-g/KzbYKbH5C2vPkaXGu8DJlHrGKHLsM25Zg9WuC9pMGfuvT+X25tZQWo5fK1BjBm8+UrVE9LDCvaY0CQk+fXDA==",
+ "license": "MIT",
+ "optionalDependencies": {
+ "rxjs": "*"
+ }
+ },
+ "node_modules/typescript": {
+ "version": "5.8.3",
+ "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.8.3.tgz",
+ "integrity": "sha512-p1diW6TqL9L07nNxvRMM7hMMw4c5XOo/1ibL4aAIGmSAt9slTE1Xgw5KWuof2uTOvCg9BY7ZRi+GaF+7sfgPeQ==",
+ "license": "Apache-2.0",
+ "bin": {
+ "tsc": "bin/tsc",
+ "tsserver": "bin/tsserver"
+ },
+ "engines": {
+ "node": ">=14.17"
+ }
+ },
+ "node_modules/undici-types": {
+ "version": "6.21.0",
+ "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz",
+ "integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==",
+ "license": "MIT"
+ },
+ "node_modules/uuid": {
+ "version": "10.0.0",
+ "resolved": "https://registry.npmjs.org/uuid/-/uuid-10.0.0.tgz",
+ "integrity": "sha512-8XkAphELsDnEGrDxUOHB3RGvXz6TeuYSGEZBOjtTtPm2lwhGBjLgOzLHB63IUWfBpNucQjND6d3AOudO+H3RWQ==",
+ "funding": [
+ "https://github.com/sponsors/broofa",
+ "https://github.com/sponsors/ctavan"
+ ],
+ "license": "MIT",
+ "bin": {
+ "uuid": "dist/bin/uuid"
+ }
+ },
+ "node_modules/vite": {
+ "version": "6.3.5",
+ "resolved": "https://registry.npmjs.org/vite/-/vite-6.3.5.tgz",
+ "integrity": "sha512-cZn6NDFE7wdTpINgs++ZJ4N49W2vRp8LCKrn3Ob1kYNtOo21vfDoaV5GzBfLU4MovSAB8uNRm4jgzVQZ+mBzPQ==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "esbuild": "^0.25.0",
+ "fdir": "^6.4.4",
+ "picomatch": "^4.0.2",
+ "postcss": "^8.5.3",
+ "rollup": "^4.34.9",
+ "tinyglobby": "^0.2.13"
+ },
+ "bin": {
+ "vite": "bin/vite.js"
+ },
+ "engines": {
+ "node": "^18.0.0 || ^20.0.0 || >=22.0.0"
+ },
+ "funding": {
+ "url": "https://github.com/vitejs/vite?sponsor=1"
+ },
+ "optionalDependencies": {
+ "fsevents": "~2.3.3"
+ },
+ "peerDependencies": {
+ "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0",
+ "jiti": ">=1.21.0",
+ "less": "*",
+ "lightningcss": "^1.21.0",
+ "sass": "*",
+ "sass-embedded": "*",
+ "stylus": "*",
+ "sugarss": "*",
+ "terser": "^5.16.0",
+ "tsx": "^4.8.1",
+ "yaml": "^2.4.2"
+ },
+ "peerDependenciesMeta": {
+ "@types/node": {
+ "optional": true
+ },
+ "jiti": {
+ "optional": true
+ },
+ "less": {
+ "optional": true
+ },
+ "lightningcss": {
+ "optional": true
+ },
+ "sass": {
+ "optional": true
+ },
+ "sass-embedded": {
+ "optional": true
+ },
+ "stylus": {
+ "optional": true
+ },
+ "sugarss": {
+ "optional": true
+ },
+ "terser": {
+ "optional": true
+ },
+ "tsx": {
+ "optional": true
+ },
+ "yaml": {
+ "optional": true
+ }
+ }
+ }
+ }
+}
diff --git a/examples/freeze-test/client/package.json b/examples/freeze-test/client/package.json
new file mode 100644
index 000000000..d2df048f5
--- /dev/null
+++ b/examples/freeze-test/client/package.json
@@ -0,0 +1,26 @@
+{
+ "name": "client",
+ "version": "1.0.0",
+ "main": "index.js",
+ "scripts": {
+ "dev": "vite",
+ "build": "tsc && vite build",
+ "preview": "vite preview"
+ },
+ "keywords": [],
+ "author": "",
+ "license": "ISC",
+ "description": "",
+ "devDependencies": {
+ "@types/node": "^22.15.30",
+ "@types/protobufjs": "^6.0.0",
+ "@vitejs/plugin-react-swc": "^3.10.1",
+ "typescript": "^5.8.3",
+ "vite": "^6.3.5"
+ },
+ "dependencies": {
+ "@pipecat-ai/client-js": "^0.4.0",
+ "@pipecat-ai/websocket-transport": "^0.4.1",
+ "protobufjs": "^7.4.0"
+ }
+}
diff --git a/examples/freeze-test/client/src/app.ts b/examples/freeze-test/client/src/app.ts
new file mode 100644
index 000000000..2e8315539
--- /dev/null
+++ b/examples/freeze-test/client/src/app.ts
@@ -0,0 +1,328 @@
+/**
+ * Copyright (c) 2024–2025, Daily
+ *
+ * SPDX-License-Identifier: BSD 2-Clause License
+ */
+
+/**
+ * RTVI Client Implementation
+ *
+ * This client connects to an RTVI-compatible bot server using WebSocket.
+ *
+ * Requirements:
+ * - A running RTVI bot server (defaults to http://localhost:7860)
+ */
+
+import {
+ RTVIClient,
+ RTVIClientOptions,
+ RTVIEvent,
+} from '@pipecat-ai/client-js';
+import {
+ ProtobufFrameSerializer,
+ WebSocketTransport
+} from "@pipecat-ai/websocket-transport";
+
+class RecordingSerializer extends ProtobufFrameSerializer {
+
+ private lastTimestamp: number | null = null;
+ private recordingAudioToSend: boolean = false;
+ private _recordedAudio: { data: ArrayBuffer; delay: number }[] = [];
+
+ public startRecording() {
+ this.recordingAudioToSend = true;
+ this._recordedAudio = [];
+ this.lastTimestamp = null;
+ }
+
+ public stopRecording() {
+ this.recordingAudioToSend = false;
+ }
+
+ // @ts-ignore
+ serializeAudio(data: ArrayBuffer, sampleRate: number, numChannels: number): Uint8Array | null {
+ if (this.recordingAudioToSend) {
+ const now = Date.now();
+ // Compute delay since last packet
+ const delay = this.lastTimestamp ? now - this.lastTimestamp : 0;
+ this.lastTimestamp = now;
+ // Save audio chunk and delay
+ this._recordedAudio.push({ data, delay });
+ return null;
+ } else {
+ return super.serializeAudio(data, sampleRate, numChannels);
+ }
+ }
+
+ public get recordedAudio() {
+ return this._recordedAudio
+ }
+}
+
+class WebsocketClientApp {
+ private ENABLE_RECORDING_MODE = false
+ private RECORDING_TIME_MS = 10000
+
+ private rtviClient: RTVIClient | null = null;
+ private connectBtn: HTMLButtonElement | null = null;
+ private disconnectBtn: HTMLButtonElement | null = null;
+ private statusSpan: HTMLElement | null = null;
+ private debugLog: HTMLElement | null = null;
+ private botAudio: HTMLAudioElement;
+
+ private declare websocketTransport: WebSocketTransport;
+ private sendRecordedAudio: boolean = false
+ private declare recordingSerializer: RecordingSerializer;
+
+ private playBtn: HTMLButtonElement | null = null;
+ private stopBtn: HTMLButtonElement | null = null;
+
+ constructor() {
+ this.botAudio = document.createElement('audio');
+ this.botAudio.autoplay = true;
+ //this.botAudio.playsInline = true;
+ document.body.appendChild(this.botAudio);
+
+ this.setupDOMElements();
+ this.setupEventListeners();
+ }
+
+ /**
+ * Set up references to DOM elements and create necessary media elements
+ */
+ private setupDOMElements(): void {
+ this.connectBtn = document.getElementById('connect-btn') as HTMLButtonElement;
+ this.disconnectBtn = document.getElementById('disconnect-btn') as HTMLButtonElement;
+ this.statusSpan = document.getElementById('connection-status');
+ this.debugLog = document.getElementById('debug-log');
+ this.playBtn = document.getElementById('play-btn') as HTMLButtonElement;
+ this.stopBtn = document.getElementById('stop-btn') as HTMLButtonElement;
+ }
+
+ /**
+ * Set up event listeners for connect/disconnect buttons
+ */
+ private setupEventListeners(): void {
+ this.connectBtn?.addEventListener('click', () => this.connect());
+ this.disconnectBtn?.addEventListener('click', () => this.disconnect());
+ this.playBtn?.addEventListener('click', () => this.startSendingRecordedAudio());
+ this.stopBtn?.addEventListener('click', () => this.stopSendingRecordedAudio());
+ }
+
+ /**
+ * Add a timestamped message to the debug log
+ */
+ private log(message: string): void {
+ if (!this.debugLog) return;
+ const entry = document.createElement('div');
+ entry.textContent = `${new Date().toISOString()} - ${message}`;
+ if (message.startsWith('User: ')) {
+ entry.style.color = '#2196F3';
+ } else if (message.startsWith('Bot: ')) {
+ entry.style.color = '#4CAF50';
+ }
+ this.debugLog.appendChild(entry);
+ this.debugLog.scrollTop = this.debugLog.scrollHeight;
+ console.log(message);
+ }
+
+ /**
+ * Update the connection status display
+ */
+ private updateStatus(status: string): void {
+ if (this.statusSpan) {
+ this.statusSpan.textContent = status;
+ }
+ this.log(`Status: ${status}`);
+ }
+
+ /**
+ * Check for available media tracks and set them up if present
+ * This is called when the bot is ready or when the transport state changes to ready
+ */
+ setupMediaTracks() {
+ if (!this.rtviClient) return;
+ const tracks = this.rtviClient.tracks();
+ if (tracks.bot?.audio) {
+ this.setupAudioTrack(tracks.bot.audio);
+ }
+ }
+
+ /**
+ * Set up listeners for track events (start/stop)
+ * This handles new tracks being added during the session
+ */
+ setupTrackListeners() {
+ if (!this.rtviClient) return;
+
+ // Listen for new tracks starting
+ this.rtviClient.on(RTVIEvent.TrackStarted, (track, participant) => {
+ // Only handle non-local (bot) tracks
+ if (!participant?.local && track.kind === 'audio') {
+ this.setupAudioTrack(track);
+ }
+ });
+
+ // Listen for tracks stopping
+ this.rtviClient.on(RTVIEvent.TrackStopped, (track, participant) => {
+ this.log(`Track stopped: ${track.kind} from ${participant?.name || 'unknown'}`);
+ });
+ }
+
+ /**
+ * Set up an audio track for playback
+ * Handles both initial setup and track updates
+ */
+ private setupAudioTrack(track: MediaStreamTrack): void {
+ this.log('Setting up audio track');
+ if (this.botAudio.srcObject && "getAudioTracks" in this.botAudio.srcObject) {
+ const oldTrack = this.botAudio.srcObject.getAudioTracks()[0];
+ if (oldTrack?.id === track.id) return;
+ }
+ this.botAudio.srcObject = new MediaStream([track]);
+ }
+
+ /**
+ * Initialize and connect to the bot
+ * This sets up the RTVI client, initializes devices, and establishes the connection
+ */
+ public async connect(): Promise {
+ try {
+ const startTime = Date.now();
+
+ this.recordingSerializer = new RecordingSerializer()
+ const transport = this.ENABLE_RECORDING_MODE ? new WebSocketTransport({serializer: this.recordingSerializer}) : new WebSocketTransport();
+ this.websocketTransport = transport
+
+ const RTVIConfig: RTVIClientOptions = {
+ transport,
+ params: {
+ // The baseURL and endpoint of your bot server that the client will connect to
+ baseUrl: 'http://localhost:7860',
+ endpoints: { connect: '/connect' },
+ },
+ enableMic: true,
+ enableCam: false,
+ callbacks: {
+ onConnected: () => {
+ this.updateStatus('Connected');
+ if (this.connectBtn) this.connectBtn.disabled = true;
+ if (this.disconnectBtn) this.disconnectBtn.disabled = false;
+ },
+ onDisconnected: () => {
+ this.updateStatus('Disconnected');
+ if (this.connectBtn) this.connectBtn.disabled = false;
+ if (this.disconnectBtn) this.disconnectBtn.disabled = true;
+ this.log('Client disconnected');
+ },
+ onBotReady: (data) => {
+ this.log(`Bot ready: ${JSON.stringify(data)}`);
+ this.setupMediaTracks();
+ },
+ onUserTranscript: (data) => {
+ if (data.final) {
+ this.log(`User: ${data.text}`);
+ }
+ },
+ onBotTranscript: (data) => this.log(`Bot: ${data.text}`),
+ onMessageError: (error) => console.error('Message error:', error),
+ onError: (error) => console.error('Error:', error),
+ },
+ }
+ this.rtviClient = new RTVIClient(RTVIConfig);
+ this.setupTrackListeners();
+
+ this.log('Initializing devices...');
+ await this.rtviClient.initDevices();
+
+ this.log('Connecting to bot...');
+ await this.rtviClient.connect();
+
+ const timeTaken = Date.now() - startTime;
+ this.log(`Connection complete, timeTaken: ${timeTaken}`);
+
+ if (this.ENABLE_RECORDING_MODE) {
+ this.log(`Starting to recording the next ${(this.RECORDING_TIME_MS/1000)}s of audio`);
+ this.recordingSerializer.startRecording()
+ await this.sleep(this.RECORDING_TIME_MS)
+ this.recordingSerializer.stopRecording()
+ this.log("Recording stopped");
+ this.rtviClient.enableMic(false)
+ this.startSendingRecordedAudio()
+ }
+ } catch (error) {
+ this.log(`Error connecting: ${(error as Error).message}`);
+ this.updateStatus('Error');
+ // Clean up if there's an error
+ if (this.rtviClient) {
+ try {
+ await this.rtviClient.disconnect();
+ } catch (disconnectError) {
+ this.log(`Error during disconnect: ${disconnectError}`);
+ }
+ }
+ }
+ }
+
+ /**
+ * Disconnect from the bot and clean up media resources
+ */
+ public async disconnect(): Promise {
+ if (this.rtviClient) {
+ try {
+ this.stopSendingRecordedAudio()
+ await this.rtviClient.disconnect();
+ this.rtviClient = null;
+ if (this.botAudio.srcObject && "getAudioTracks" in this.botAudio.srcObject) {
+ this.botAudio.srcObject.getAudioTracks().forEach((track) => track.stop());
+ this.botAudio.srcObject = null;
+ }
+ } catch (error) {
+ this.log(`Error disconnecting: ${(error as Error).message}`);
+ }
+ }
+ }
+
+ private startSendingRecordedAudio() {
+ this.sendRecordedAudio = true
+ if (this.playBtn) this.playBtn.disabled = true;
+ if (this.stopBtn) this.stopBtn.disabled = false;
+ void this.replayAudio()
+ }
+
+ private stopSendingRecordedAudio() {
+ if (this.stopBtn) this.stopBtn.disabled = true;
+ if (this.playBtn) this.playBtn.disabled = false;
+ this.sendRecordedAudio = false
+ }
+
+ private async replayAudio() {
+ if (this.sendRecordedAudio) {
+ this.log("Sending recorded audio")
+ for (const chunk of this.recordingSerializer.recordedAudio) {
+ await this.sleep(chunk.delay);
+ this.websocketTransport.handleUserAudioStream(chunk.data);
+ }
+ const randomDelay = 1000 + Math.random() * (10000 - 500);
+ await this.sleep(randomDelay);
+
+ void this.replayAudio()
+ }
+ }
+
+ private sleep(ms: number): Promise {
+ return new Promise(resolve => setTimeout(resolve, ms));
+ }
+
+}
+
+declare global {
+ interface Window {
+ WebsocketClientApp: typeof WebsocketClientApp;
+ }
+}
+
+window.addEventListener('DOMContentLoaded', () => {
+ window.WebsocketClientApp = WebsocketClientApp;
+ new WebsocketClientApp();
+});
diff --git a/examples/freeze-test/client/src/style.css b/examples/freeze-test/client/src/style.css
new file mode 100644
index 000000000..9c147266e
--- /dev/null
+++ b/examples/freeze-test/client/src/style.css
@@ -0,0 +1,98 @@
+body {
+ margin: 0;
+ padding: 20px;
+ font-family: Arial, sans-serif;
+ background-color: #f0f0f0;
+}
+
+.container {
+ max-width: 1200px;
+ margin: 0 auto;
+}
+
+.status-bar {
+ display: flex;
+ justify-content: space-between;
+ align-items: center;
+ padding: 10px;
+ background-color: #fff;
+ border-radius: 8px;
+ margin-bottom: 20px;
+}
+
+.controls button {
+ padding: 8px 16px;
+ margin-left: 10px;
+ border: none;
+ border-radius: 4px;
+ cursor: pointer;
+}
+
+#connect-btn {
+ background-color: #4caf50;
+ color: white;
+}
+
+#disconnect-btn {
+ background-color: #f44336;
+ color: white;
+}
+
+button:disabled {
+ opacity: 0.5;
+ cursor: not-allowed;
+}
+
+.main-content {
+ background-color: #fff;
+ border-radius: 8px;
+ padding: 20px;
+ margin-bottom: 20px;
+}
+
+.bot-container {
+ display: flex;
+ flex-direction: column;
+ align-items: center;
+}
+
+#bot-video-container {
+ width: 640px;
+ height: 360px;
+ background-color: #e0e0e0;
+ border-radius: 8px;
+ margin: 20px auto;
+ overflow: hidden;
+ display: flex;
+ align-items: center;
+ justify-content: center;
+}
+
+#bot-video-container video {
+ width: 100%;
+ height: 100%;
+ object-fit: cover;
+}
+
+.debug-panel {
+ background-color: #fff;
+ border-radius: 8px;
+ padding: 20px;
+}
+
+.debug-panel h3 {
+ margin: 0 0 10px 0;
+ font-size: 16px;
+ font-weight: bold;
+}
+
+#debug-log {
+ height: 500px;
+ overflow-y: auto;
+ background-color: #f8f8f8;
+ padding: 10px;
+ border-radius: 4px;
+ font-family: monospace;
+ font-size: 12px;
+ line-height: 1.4;
+}
diff --git a/examples/freeze-test/client/tsconfig.json b/examples/freeze-test/client/tsconfig.json
new file mode 100644
index 000000000..c9c555d96
--- /dev/null
+++ b/examples/freeze-test/client/tsconfig.json
@@ -0,0 +1,111 @@
+{
+ "compilerOptions": {
+ /* Visit https://aka.ms/tsconfig to read more about this file */
+
+ /* Projects */
+ // "incremental": true, /* Save .tsbuildinfo files to allow for incremental compilation of projects. */
+ // "composite": true, /* Enable constraints that allow a TypeScript project to be used with project references. */
+ // "tsBuildInfoFile": "./.tsbuildinfo", /* Specify the path to .tsbuildinfo incremental compilation file. */
+ // "disableSourceOfProjectReferenceRedirect": true, /* Disable preferring source files instead of declaration files when referencing composite projects. */
+ // "disableSolutionSearching": true, /* Opt a project out of multi-project reference checking when editing. */
+ // "disableReferencedProjectLoad": true, /* Reduce the number of projects loaded automatically by TypeScript. */
+
+ /* Language and Environment */
+ "target": "es2016", /* Set the JavaScript language version for emitted JavaScript and include compatible library declarations. */
+ // "lib": [], /* Specify a set of bundled library declaration files that describe the target runtime environment. */
+ // "jsx": "preserve", /* Specify what JSX code is generated. */
+ // "experimentalDecorators": true, /* Enable experimental support for legacy experimental decorators. */
+ // "emitDecoratorMetadata": true, /* Emit design-type metadata for decorated declarations in source files. */
+ // "jsxFactory": "", /* Specify the JSX factory function used when targeting React JSX emit, e.g. 'React.createElement' or 'h'. */
+ // "jsxFragmentFactory": "", /* Specify the JSX Fragment reference used for fragments when targeting React JSX emit e.g. 'React.Fragment' or 'Fragment'. */
+ // "jsxImportSource": "", /* Specify module specifier used to import the JSX factory functions when using 'jsx: react-jsx*'. */
+ // "reactNamespace": "", /* Specify the object invoked for 'createElement'. This only applies when targeting 'react' JSX emit. */
+ // "noLib": true, /* Disable including any library files, including the default lib.d.ts. */
+ // "useDefineForClassFields": true, /* Emit ECMAScript-standard-compliant class fields. */
+ // "moduleDetection": "auto", /* Control what method is used to detect module-format JS files. */
+
+ /* Modules */
+ "module": "commonjs", /* Specify what module code is generated. */
+ // "rootDir": "./", /* Specify the root folder within your source files. */
+ // "moduleResolution": "node10", /* Specify how TypeScript looks up a file from a given module specifier. */
+ // "baseUrl": "./", /* Specify the base directory to resolve non-relative module names. */
+ // "paths": {}, /* Specify a set of entries that re-map imports to additional lookup locations. */
+ // "rootDirs": [], /* Allow multiple folders to be treated as one when resolving modules. */
+ // "typeRoots": [], /* Specify multiple folders that act like './node_modules/@types'. */
+ // "types": [], /* Specify type package names to be included without being referenced in a source file. */
+ // "allowUmdGlobalAccess": true, /* Allow accessing UMD globals from modules. */
+ // "moduleSuffixes": [], /* List of file name suffixes to search when resolving a module. */
+ // "allowImportingTsExtensions": true, /* Allow imports to include TypeScript file extensions. Requires '--moduleResolution bundler' and either '--noEmit' or '--emitDeclarationOnly' to be set. */
+ // "rewriteRelativeImportExtensions": true, /* Rewrite '.ts', '.tsx', '.mts', and '.cts' file extensions in relative import paths to their JavaScript equivalent in output files. */
+ // "resolvePackageJsonExports": true, /* Use the package.json 'exports' field when resolving package imports. */
+ // "resolvePackageJsonImports": true, /* Use the package.json 'imports' field when resolving imports. */
+ // "customConditions": [], /* Conditions to set in addition to the resolver-specific defaults when resolving imports. */
+ // "noUncheckedSideEffectImports": true, /* Check side effect imports. */
+ // "resolveJsonModule": true, /* Enable importing .json files. */
+ // "allowArbitraryExtensions": true, /* Enable importing files with any extension, provided a declaration file is present. */
+ // "noResolve": true, /* Disallow 'import's, 'require's or ''s from expanding the number of files TypeScript should add to a project. */
+
+ /* JavaScript Support */
+ // "allowJs": true, /* Allow JavaScript files to be a part of your program. Use the 'checkJS' option to get errors from these files. */
+ // "checkJs": true, /* Enable error reporting in type-checked JavaScript files. */
+ // "maxNodeModuleJsDepth": 1, /* Specify the maximum folder depth used for checking JavaScript files from 'node_modules'. Only applicable with 'allowJs'. */
+
+ /* Emit */
+ // "declaration": true, /* Generate .d.ts files from TypeScript and JavaScript files in your project. */
+ // "declarationMap": true, /* Create sourcemaps for d.ts files. */
+ // "emitDeclarationOnly": true, /* Only output d.ts files and not JavaScript files. */
+ // "sourceMap": true, /* Create source map files for emitted JavaScript files. */
+ // "inlineSourceMap": true, /* Include sourcemap files inside the emitted JavaScript. */
+ // "noEmit": true, /* Disable emitting files from a compilation. */
+ // "outFile": "./", /* Specify a file that bundles all outputs into one JavaScript file. If 'declaration' is true, also designates a file that bundles all .d.ts output. */
+ // "outDir": "./", /* Specify an output folder for all emitted files. */
+ // "removeComments": true, /* Disable emitting comments. */
+ // "importHelpers": true, /* Allow importing helper functions from tslib once per project, instead of including them per-file. */
+ // "downlevelIteration": true, /* Emit more compliant, but verbose and less performant JavaScript for iteration. */
+ // "sourceRoot": "", /* Specify the root path for debuggers to find the reference source code. */
+ // "mapRoot": "", /* Specify the location where debugger should locate map files instead of generated locations. */
+ // "inlineSources": true, /* Include source code in the sourcemaps inside the emitted JavaScript. */
+ // "emitBOM": true, /* Emit a UTF-8 Byte Order Mark (BOM) in the beginning of output files. */
+ // "newLine": "crlf", /* Set the newline character for emitting files. */
+ // "stripInternal": true, /* Disable emitting declarations that have '@internal' in their JSDoc comments. */
+ // "noEmitHelpers": true, /* Disable generating custom helper functions like '__extends' in compiled output. */
+ // "noEmitOnError": true, /* Disable emitting files if any type checking errors are reported. */
+ // "preserveConstEnums": true, /* Disable erasing 'const enum' declarations in generated code. */
+ // "declarationDir": "./", /* Specify the output directory for generated declaration files. */
+
+ /* Interop Constraints */
+ // "isolatedModules": true, /* Ensure that each file can be safely transpiled without relying on other imports. */
+ // "verbatimModuleSyntax": true, /* Do not transform or elide any imports or exports not marked as type-only, ensuring they are written in the output file's format based on the 'module' setting. */
+ // "isolatedDeclarations": true, /* Require sufficient annotation on exports so other tools can trivially generate declaration files. */
+ // "allowSyntheticDefaultImports": true, /* Allow 'import x from y' when a module doesn't have a default export. */
+ "esModuleInterop": true, /* Emit additional JavaScript to ease support for importing CommonJS modules. This enables 'allowSyntheticDefaultImports' for type compatibility. */
+ // "preserveSymlinks": true, /* Disable resolving symlinks to their realpath. This correlates to the same flag in node. */
+ "forceConsistentCasingInFileNames": true, /* Ensure that casing is correct in imports. */
+
+ /* Type Checking */
+ "strict": true, /* Enable all strict type-checking options. */
+ // "noImplicitAny": true, /* Enable error reporting for expressions and declarations with an implied 'any' type. */
+ // "strictNullChecks": true, /* When type checking, take into account 'null' and 'undefined'. */
+ // "strictFunctionTypes": true, /* When assigning functions, check to ensure parameters and the return values are subtype-compatible. */
+ // "strictBindCallApply": true, /* Check that the arguments for 'bind', 'call', and 'apply' methods match the original function. */
+ // "strictPropertyInitialization": true, /* Check for class properties that are declared but not set in the constructor. */
+ // "strictBuiltinIteratorReturn": true, /* Built-in iterators are instantiated with a 'TReturn' type of 'undefined' instead of 'any'. */
+ // "noImplicitThis": true, /* Enable error reporting when 'this' is given the type 'any'. */
+ // "useUnknownInCatchVariables": true, /* Default catch clause variables as 'unknown' instead of 'any'. */
+ // "alwaysStrict": true, /* Ensure 'use strict' is always emitted. */
+ // "noUnusedLocals": true, /* Enable error reporting when local variables aren't read. */
+ // "noUnusedParameters": true, /* Raise an error when a function parameter isn't read. */
+ // "exactOptionalPropertyTypes": true, /* Interpret optional property types as written, rather than adding 'undefined'. */
+ // "noImplicitReturns": true, /* Enable error reporting for codepaths that do not explicitly return in a function. */
+ // "noFallthroughCasesInSwitch": true, /* Enable error reporting for fallthrough cases in switch statements. */
+ // "noUncheckedIndexedAccess": true, /* Add 'undefined' to a type when accessed using an index. */
+ // "noImplicitOverride": true, /* Ensure overriding members in derived classes are marked with an override modifier. */
+ // "noPropertyAccessFromIndexSignature": true, /* Enforces using indexed accessors for keys declared using an indexed type. */
+ // "allowUnusedLabels": true, /* Disable error reporting for unused labels. */
+ // "allowUnreachableCode": true, /* Disable error reporting for unreachable code. */
+
+ /* Completeness */
+ // "skipDefaultLibCheck": true, /* Skip type checking .d.ts files that are included with TypeScript. */
+ "skipLibCheck": true /* Skip type checking all .d.ts files. */
+ }
+}
diff --git a/examples/freeze-test/client/vite.config.js b/examples/freeze-test/client/vite.config.js
new file mode 100644
index 000000000..daf85167d
--- /dev/null
+++ b/examples/freeze-test/client/vite.config.js
@@ -0,0 +1,15 @@
+import { defineConfig } from 'vite';
+import react from '@vitejs/plugin-react-swc';
+
+export default defineConfig({
+ plugins: [react()],
+ server: {
+ proxy: {
+ // Proxy /api requests to the backend server
+ '/connect': {
+ target: 'http://0.0.0.0:7860', // Replace with your backend URL
+ changeOrigin: true,
+ },
+ },
+ },
+});
diff --git a/examples/freeze-test/freeze_test_bot.py b/examples/freeze-test/freeze_test_bot.py
new file mode 100644
index 000000000..5be79ad12
--- /dev/null
+++ b/examples/freeze-test/freeze_test_bot.py
@@ -0,0 +1,309 @@
+#
+# Copyright (c) 2024–2025, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import argparse
+import asyncio
+import os
+from contextlib import asynccontextmanager
+from typing import Any, Dict
+
+import uvicorn
+from dotenv import load_dotenv
+from fastapi import FastAPI, Request, WebSocket
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import RedirectResponse
+from loguru import logger
+from pipecat_ai_small_webrtc_prebuilt.frontend import SmallWebRTCPrebuiltUI
+
+from pipecat.audio.vad.silero import SileroVADAnalyzer
+from pipecat.frames.frames import (
+ CancelFrame,
+ EndFrame,
+ Frame,
+ InterimTranscriptionFrame,
+ LLMFullResponseEndFrame,
+ StartFrame,
+ StartInterruptionFrame,
+ StopFrame,
+ StopInterruptionFrame,
+ TranscriptionFrame,
+ UserStartedSpeakingFrame,
+ UserStoppedSpeakingFrame,
+)
+from pipecat.observers.loggers.debug_log_observer import DebugLogObserver
+from pipecat.pipeline.parallel_pipeline import ParallelPipeline
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.runner import PipelineRunner
+from pipecat.pipeline.task import PipelineParams, PipelineTask
+from pipecat.processors.aggregators.openai_llm_context import (
+ OpenAILLMContext,
+ OpenAILLMContextFrame,
+)
+from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
+from pipecat.processors.frameworks.rtvi import RTVIConfig, RTVIProcessor
+from pipecat.serializers.protobuf import ProtobufFrameSerializer
+from pipecat.services.cartesia.tts import CartesiaTTSService
+from pipecat.services.deepgram.stt import DeepgramSTTService
+from pipecat.services.openai.llm import OpenAILLMService
+from pipecat.transports.network.fastapi_websocket import (
+ FastAPIWebsocketParams,
+ FastAPIWebsocketTransport,
+)
+from pipecat.utils.time import time_now_iso8601
+
+load_dotenv(override=True)
+
+
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+ """Handles FastAPI startup and shutdown."""
+ yield # Run app
+
+
+# Initialize FastAPI app with lifespan manager
+app = FastAPI(lifespan=lifespan)
+
+# Configure CORS to allow requests from any origin
+app.add_middleware(
+ CORSMiddleware,
+ allow_origins=["*"],
+ allow_credentials=True,
+ allow_methods=["*"],
+ allow_headers=["*"],
+)
+
+# Mount the frontend at /
+app.mount("/client", SmallWebRTCPrebuiltUI)
+
+
+class SimulateFreezeInput(FrameProcessor):
+ def __init__(
+ self,
+ **kwargs,
+ ):
+ super().__init__(**kwargs)
+ # Whether we have seen a StartFrame already.
+ self._initialized = False
+ self._send_frames_task = None
+
+ async def process_frame(self, frame: Frame, direction: FrameDirection):
+ await super().process_frame(frame, direction)
+ if isinstance(frame, StartFrame):
+ # Push StartFrame before start(), because we want StartFrame to be
+ # processed by every processor before any other frame is processed.
+ await self.push_frame(frame, direction)
+ await self._start(frame)
+ elif isinstance(frame, CancelFrame):
+ logger.info("SimulateFreezeInput: Received cancel frame")
+ await self._stop()
+ await self.push_frame(frame, direction)
+ elif isinstance(frame, EndFrame):
+ logger.info("SimulateFreezeInput: Received end frame")
+ await self.push_frame(frame, direction)
+ await self._stop()
+ elif isinstance(frame, StopFrame):
+ logger.info("SimulateFreezeInput: Received stop frame")
+ await self.push_frame(frame, direction)
+ await self._stop()
+
+ async def _start(self, frame: StartFrame):
+ if self._initialized:
+ return
+ logger.info(f"Starting SimulateFreezeInput")
+ self._initialized = True
+ if not self._send_frames_task:
+ self._send_frames_task = self.create_task(self._send_frames())
+
+ async def _stop(self):
+ logger.info(f"Stopping SimulateFreezeInput")
+ self._initialized = False
+ if self._send_frames_task:
+ await self.cancel_task(self._send_frames_task)
+ self._send_frames_task = None
+
+ async def _send_user_text(self, text: str):
+ # Emulation as if the user has spoken and the stt transcribed
+ await self.push_frame(UserStartedSpeakingFrame())
+ await self.push_frame(StartInterruptionFrame())
+ await self.push_frame(
+ TranscriptionFrame(
+ text,
+ "",
+ time_now_iso8601(),
+ )
+ )
+ # Need to wait before sending the UserStoppedSpeakingFrame,
+ # otherwise TranscriptionFrame will be processed
+ # later than the UserStoppedSpeakingFrame
+ await asyncio.sleep(0.1)
+ await self.push_frame(UserStoppedSpeakingFrame())
+ await self.push_frame(StopInterruptionFrame())
+
+ async def _send_frames(self):
+ try:
+ i = 0
+ while True:
+ logger.debug("SimulateFreezeInput _send_frames")
+ await self._send_user_text("Tell me a brief history of Brazil!")
+ await asyncio.sleep(3)
+ await self._send_user_text("")
+ break
+ # i += 1
+ # if i >= 5:
+ # break
+ # sleeping 1s before interrupting
+ # wait_time = random.uniform(1, 10)
+ # await asyncio.sleep(wait_time)
+ except Exception as e:
+ logger.error(f"{self} exception receiving data: {e.__class__.__name__} ({e})")
+
+
+async def run_example(websocket_client):
+ logger.info(f"Starting bot")
+
+ # Create a transport using the WebRTC connection
+ transport = FastAPIWebsocketTransport(
+ websocket=websocket_client,
+ params=FastAPIWebsocketParams(
+ audio_in_enabled=True,
+ audio_out_enabled=True,
+ add_wav_header=False,
+ vad_analyzer=SileroVADAnalyzer(),
+ serializer=ProtobufFrameSerializer(),
+ ),
+ )
+
+ freeze = SimulateFreezeInput()
+
+ stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
+
+ tts = CartesiaTTSService(
+ api_key=os.getenv("CARTESIA_API_KEY"),
+ voice_id="71a7ad14-091c-4e8e-a314-022ece01c121", # British Reading Lady
+ )
+
+ llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))
+
+ rtvi = RTVIProcessor(config=RTVIConfig(config=[]))
+
+ messages = [
+ {
+ "role": "system",
+ "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
+ },
+ ]
+
+ context = OpenAILLMContext(messages)
+ context_aggregator = llm.create_context_aggregator(context)
+
+ pipeline = Pipeline(
+ [
+ ParallelPipeline(
+ [
+ freeze,
+ ],
+ [
+ transport.input(),
+ stt,
+ ],
+ ),
+ rtvi,
+ context_aggregator.user(), # User responses
+ llm, # LLM
+ tts, # TTS
+ transport.output(), # Transport bot output
+ context_aggregator.assistant(), # Assistant spoken responses
+ ]
+ )
+
+ task = PipelineTask(
+ pipeline,
+ params=PipelineParams(
+ allow_interruptions=True,
+ enable_metrics=True,
+ enable_usage_metrics=True,
+ report_only_initial_ttfb=True,
+ ),
+ idle_timeout_secs=120,
+ observers=[
+ DebugLogObserver(
+ frame_types={
+ InterimTranscriptionFrame: None,
+ TranscriptionFrame: None,
+ # TTSTextFrame: None,
+ # LLMTextFrame: None,
+ OpenAILLMContextFrame: None,
+ LLMFullResponseEndFrame: None,
+ },
+ exclude_fields={
+ "result",
+ "metadata",
+ "audio",
+ "image",
+ "images",
+ },
+ ),
+ ],
+ )
+
+ @transport.event_handler("on_client_connected")
+ async def on_client_connected(transport, client):
+ logger.info(f"Client connected")
+
+ @rtvi.event_handler("on_client_ready")
+ async def on_client_ready(rtvi):
+ logger.info(f"Client ready")
+ await rtvi.set_bot_ready()
+ # Kick off the conversation.
+ # messages.append({"role": "system", "content": "Please introduce yourself to the user."})
+ # await task.queue_frames([context_aggregator.user().get_context_frame()])
+
+ @transport.event_handler("on_client_disconnected")
+ async def on_client_disconnected(transport, client):
+ logger.info(f"Client disconnected")
+ await task.cancel()
+
+ runner = PipelineRunner(handle_sigint=False)
+
+ await runner.run(task)
+
+
+@app.get("/", include_in_schema=False)
+async def root_redirect():
+ return RedirectResponse(url="/client/")
+
+
+@app.websocket("/ws")
+async def websocket_endpoint(websocket: WebSocket):
+ await websocket.accept()
+ print("WebSocket connection accepted")
+ try:
+ await run_example(websocket)
+ except Exception as e:
+ print(f"Exception in run_bot: {e}")
+
+
+@app.post("/connect")
+async def bot_connect(request: Request) -> Dict[Any, Any]:
+ server_mode = os.getenv("WEBSOCKET_SERVER", "fast_api")
+ if server_mode == "websocket_server":
+ ws_url = "ws://localhost:8765"
+ else:
+ ws_url = "ws://localhost:7860/ws"
+ return {"ws_url": ws_url}
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser(description="Pipecat Bot Runner")
+ parser.add_argument(
+ "--host", default="localhost", help="Host for HTTP server (default: localhost)"
+ )
+ parser.add_argument(
+ "--port", type=int, default=7860, help="Port for HTTP server (default: 7860)"
+ )
+ args = parser.parse_args()
+
+ uvicorn.run(app, host=args.host, port=args.port)
diff --git a/examples/moondream-chatbot/bot.py b/examples/moondream-chatbot/bot.py
index d5b24aec2..dcda37ca5 100644
--- a/examples/moondream-chatbot/bot.py
+++ b/examples/moondream-chatbot/bot.py
@@ -143,6 +143,7 @@ async def main():
DailyParams(
audio_in_enabled=True,
audio_out_enabled=True,
+ video_in_enabled=True,
video_out_enabled=True,
video_out_width=1024,
video_out_height=576,
diff --git a/examples/sentry-metrics/bot.py b/examples/sentry-metrics/bot.py
index 8ff412bb7..44f9a0daa 100644
--- a/examples/sentry-metrics/bot.py
+++ b/examples/sentry-metrics/bot.py
@@ -49,7 +49,7 @@ async def main():
# Initialize Sentry
sentry_sdk.init(
- dsn="your-project-dsn",
+ dsn=os.getenv("SENTRY_DSN"),
traces_sample_rate=1.0,
)
diff --git a/pyproject.toml b/pyproject.toml
index b5c874919..f7c73d49a 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -79,6 +79,7 @@ playht = [ "pyht~=0.1.12", "websockets~=13.1" ]
qwen = []
rime = [ "websockets~=13.1" ]
riva = [ "nvidia-riva-client~=2.19.1" ]
+sambanova = []
sentry = [ "sentry-sdk~=2.23.1" ]
local-smart-turn = [ "coremltools>=8.0", "transformers", "torch==2.5.0", "torchaudio==2.5.0" ]
remote-smart-turn = []
diff --git a/src/pipecat/audio/turn/base_turn_analyzer.py b/src/pipecat/audio/turn/base_turn_analyzer.py
index fd4f18d66..301dbf7bf 100644
--- a/src/pipecat/audio/turn/base_turn_analyzer.py
+++ b/src/pipecat/audio/turn/base_turn_analyzer.py
@@ -78,3 +78,8 @@ class BaseTurnAnalyzer(ABC):
EndOfTurnState: The result of the end of turn analysis.
"""
pass
+
+ @abstractmethod
+ def clear(self):
+ """Reset the turn analyzer to its initial state."""
+ pass
diff --git a/src/pipecat/audio/turn/smart_turn/base_smart_turn.py b/src/pipecat/audio/turn/smart_turn/base_smart_turn.py
index 1bd187aee..0b577028b 100644
--- a/src/pipecat/audio/turn/smart_turn/base_smart_turn.py
+++ b/src/pipecat/audio/turn/smart_turn/base_smart_turn.py
@@ -98,6 +98,9 @@ class BaseSmartTurn(BaseTurnAnalyzer):
logger.debug(f"End of Turn result: {state}")
return state, result
+ def clear(self):
+ self._clear(EndOfTurnState.COMPLETE)
+
def _clear(self, turn_state: EndOfTurnState):
# If the state is still incomplete, keep the _speech_triggered as True
self._speech_triggered = turn_state == EndOfTurnState.INCOMPLETE
diff --git a/src/pipecat/frames/frames.py b/src/pipecat/frames/frames.py
index ec368789d..4b2d934ab 100644
--- a/src/pipecat/frames/frames.py
+++ b/src/pipecat/frames/frames.py
@@ -7,6 +7,7 @@
from dataclasses import dataclass, field
from enum import Enum
from typing import (
+ TYPE_CHECKING,
Any,
Awaitable,
Callable,
@@ -26,6 +27,9 @@ from pipecat.transcriptions.language import Language
from pipecat.utils.time import nanoseconds_to_str
from pipecat.utils.utils import obj_count, obj_id
+if TYPE_CHECKING:
+ from pipecat.processors.frame_processor import FrameProcessor
+
class KeypadEntry(str, Enum):
"""DTMF entries."""
@@ -485,16 +489,6 @@ class FatalErrorFrame(ErrorFrame):
fatal: bool = field(default=True, init=False)
-@dataclass
-class HeartbeatFrame(SystemFrame):
- """This frame is used by the pipeline task as a mechanism to know if the
- pipeline is running properly.
-
- """
-
- timestamp: int
-
-
@dataclass
class EndTaskFrame(SystemFrame):
"""This is used to notify the pipeline task that the pipeline should be
@@ -529,25 +523,25 @@ class StopTaskFrame(SystemFrame):
@dataclass
class FrameProcessorPauseUrgentFrame(SystemFrame):
- """This processor is used to pause frame processing for the given processor
- as fast as possible. Pausing frame processing will keep frames in the
- internal queue which will then be processed when frame processing is resumed
- with `FrameProcessorResumeFrame`.
+ """This frame is used to pause frame processing for the given processor as
+ fast as possible. Pausing frame processing will keep frames in the internal
+ queue which will then be processed when frame processing is resumed with
+ `FrameProcessorResumeFrame`.
"""
- processor: str
+ processor: "FrameProcessor"
@dataclass
class FrameProcessorResumeUrgentFrame(SystemFrame):
- """This processor is used to resume frame processing for the given processor
+ """This frame is used to resume frame processing for the given processor
if it was previously paused as fast as possible. After resuming frame
processing all queued frames will be processed in the order received.
"""
- processor: str
+ processor: "FrameProcessor"
@dataclass
@@ -877,25 +871,37 @@ class StopFrame(ControlFrame):
pass
+@dataclass
+class HeartbeatFrame(ControlFrame):
+ """This frame is used by the pipeline task as a mechanism to know if the
+ pipeline is running properly.
+
+ """
+
+ timestamp: int
+
+
@dataclass
class FrameProcessorPauseFrame(ControlFrame):
- """This processor is used to pause frame processing for the given
+ """This frame is used to pause frame processing for the given
processor. Pausing frame processing will keep frames in the internal queue
which will then be processed when frame processing is resumed with
- `FrameProcessorResumeFrame`."""
+ `FrameProcessorResumeFrame`.
- processor: str
+ """
+
+ processor: "FrameProcessor"
@dataclass
class FrameProcessorResumeFrame(ControlFrame):
- """This processor is used to resume frame processing for the given processor
- if it was previously paused. After resuming frame processing all queued
- frames will be processed in the order received.
+ """This frame is used to resume frame processing for the given processor if
+ it was previously paused. After resuming frame processing all queued frames
+ will be processed in the order received.
"""
- processor: str
+ processor: "FrameProcessor"
@dataclass
diff --git a/src/pipecat/observers/turn_tracking_observer.py b/src/pipecat/observers/turn_tracking_observer.py
index 956e46b55..04b5ad92b 100644
--- a/src/pipecat/observers/turn_tracking_observer.py
+++ b/src/pipecat/observers/turn_tracking_observer.py
@@ -12,6 +12,8 @@ from loguru import logger
from pipecat.frames.frames import (
BotStartedSpeakingFrame,
BotStoppedSpeakingFrame,
+ CancelFrame,
+ EndFrame,
StartFrame,
UserStartedSpeakingFrame,
)
@@ -73,6 +75,8 @@ class TurnTrackingObserver(BaseObserver):
# We only want to end the turn if the bot was previously speaking
elif isinstance(data.frame, BotStoppedSpeakingFrame) and self._is_bot_speaking:
await self._handle_bot_stopped_speaking(data)
+ elif isinstance(data.frame, (EndFrame, CancelFrame)):
+ await self._handle_pipeline_end(data)
def _schedule_turn_end(self, data: FramePushed):
"""Schedule turn end with a timeout."""
@@ -134,6 +138,14 @@ class TurnTrackingObserver(BaseObserver):
# This can happen with HTTP TTS services or function calls
self._schedule_turn_end(data)
+ async def _handle_pipeline_end(self, data: FramePushed):
+ """Handle pipeline end or cancellation by flushing any active turn."""
+ if self._is_turn_active:
+ # Cancel any pending turn end timer
+ self._cancel_turn_end_timer()
+ # End the current turn
+ await self._end_turn(data, was_interrupted=True)
+
async def _start_turn(self, data: FramePushed):
"""Start a new turn."""
self._is_turn_active = True
diff --git a/src/pipecat/pipeline/base_task.py b/src/pipecat/pipeline/base_task.py
index 278709b7e..6cdd88c9b 100644
--- a/src/pipecat/pipeline/base_task.py
+++ b/src/pipecat/pipeline/base_task.py
@@ -6,18 +6,21 @@
import asyncio
from abc import abstractmethod
+from dataclasses import dataclass
from typing import AsyncIterable, Iterable
from pipecat.frames.frames import Frame
from pipecat.utils.base_object import BaseObject
-class BaseTask(BaseObject):
- @abstractmethod
- def set_event_loop(self, loop: asyncio.AbstractEventLoop):
- """Sets the event loop that this task will run on."""
- pass
+@dataclass
+class PipelineTaskParams:
+ """Specific configuration for the pipeline task."""
+ loop: asyncio.AbstractEventLoop
+
+
+class BasePipelineTask(BaseObject):
@abstractmethod
def has_finished(self) -> bool:
"""Indicates whether the tasks has finished. That is, all processors
@@ -40,7 +43,7 @@ class BaseTask(BaseObject):
pass
@abstractmethod
- async def run(self):
+ async def run(self, params: PipelineTaskParams):
"""Starts running the given pipeline."""
pass
diff --git a/src/pipecat/pipeline/parallel_pipeline.py b/src/pipecat/pipeline/parallel_pipeline.py
index c82330107..3a08f44ed 100644
--- a/src/pipecat/pipeline/parallel_pipeline.py
+++ b/src/pipecat/pipeline/parallel_pipeline.py
@@ -202,14 +202,18 @@ class ParallelPipeline(BasePipeline):
async def _process_up_queue(self):
while True:
frame = await self._up_queue.get()
+ self.start_watchdog()
await self._parallel_push_frame(frame, FrameDirection.UPSTREAM)
self._up_queue.task_done()
+ self.reset_watchdog()
async def _process_down_queue(self):
running = True
while running:
frame = await self._down_queue.get()
+ self.start_watchdog()
+
endframe_counter = self._endframe_counter.get(frame.id, 0)
# If we have a counter, decrement it.
@@ -224,3 +228,5 @@ class ParallelPipeline(BasePipeline):
running = not (endframe_counter == 0 and isinstance(frame, EndFrame))
self._down_queue.task_done()
+
+ self.reset_watchdog()
diff --git a/src/pipecat/pipeline/runner.py b/src/pipecat/pipeline/runner.py
index 23c43c06e..b789fc7ba 100644
--- a/src/pipecat/pipeline/runner.py
+++ b/src/pipecat/pipeline/runner.py
@@ -11,6 +11,7 @@ from typing import Optional
from loguru import logger
+from pipecat.pipeline.base_task import PipelineTaskParams
from pipecat.pipeline.task import PipelineTask
from pipecat.utils.base_object import BaseObject
@@ -37,8 +38,8 @@ class PipelineRunner(BaseObject):
async def run(self, task: PipelineTask):
logger.debug(f"Runner {self} started running {task}")
self._tasks[task.name] = task
- task.set_event_loop(self._loop)
- await task.run()
+ params = PipelineTaskParams(loop=self._loop)
+ await task.run(params)
del self._tasks[task.name]
# Cleanup base object.
diff --git a/src/pipecat/pipeline/task.py b/src/pipecat/pipeline/task.py
index 3f13e7eb2..88a47189b 100644
--- a/src/pipecat/pipeline/task.py
+++ b/src/pipecat/pipeline/task.py
@@ -6,7 +6,8 @@
import asyncio
import time
-from typing import Any, AsyncIterable, Dict, Iterable, List, Optional, Sequence, Tuple, Type
+from collections import deque
+from typing import Any, AsyncIterable, Deque, Dict, Iterable, List, Optional, Tuple, Type
from loguru import logger
from pydantic import BaseModel, ConfigDict, Field
@@ -23,6 +24,7 @@ from pipecat.frames.frames import (
ErrorFrame,
Frame,
HeartbeatFrame,
+ InputAudioRawFrame,
LLMFullResponseEndFrame,
MetricsFrame,
StartFrame,
@@ -33,19 +35,22 @@ from pipecat.metrics.metrics import ProcessingMetricsData, TTFBMetricsData
from pipecat.observers.base_observer import BaseObserver
from pipecat.observers.turn_tracking_observer import TurnTrackingObserver
from pipecat.pipeline.base_pipeline import BasePipeline
-from pipecat.pipeline.base_task import BaseTask
+from pipecat.pipeline.base_task import BasePipelineTask, PipelineTaskParams
from pipecat.pipeline.task_observer import TaskObserver
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor, FrameProcessorSetup
-from pipecat.utils.asyncio import BaseTaskManager, TaskManager
+from pipecat.utils.asyncio import WATCHDOG_TIMEOUT, BaseTaskManager, TaskManager, TaskManagerParams
from pipecat.utils.tracing.setup import is_tracing_available
from pipecat.utils.tracing.turn_trace_observer import TurnTraceObserver
HEARTBEAT_SECONDS = 1.0
-HEARTBEAT_MONITOR_SECONDS = HEARTBEAT_SECONDS * 5
+HEARTBEAT_MONITOR_SECONDS = HEARTBEAT_SECONDS * 10
class PipelineParams(BaseModel):
- """Configuration parameters for pipeline execution.
+ """Configuration parameters for pipeline execution. These parameters are
+ usually passed to all frame processors using through `StartFrame`. For other
+ generic pipeline task parameters use `PipelineTask` constructor arguments
+ instead.
Attributes:
allow_interruptions: Whether to allow pipeline interruptions.
@@ -60,6 +65,7 @@ class PipelineParams(BaseModel):
send_initial_empty_metrics: Whether to send initial empty metrics.
start_metadata: Additional metadata for pipeline start.
interruption_strategies: Strategies for bot interruption behavior.
+
"""
model_config = ConfigDict(arbitrary_types_allowed=True)
@@ -71,11 +77,11 @@ class PipelineParams(BaseModel):
enable_metrics: bool = False
enable_usage_metrics: bool = False
heartbeats_period_secs: float = HEARTBEAT_SECONDS
+ interruption_strategies: List[BaseInterruptionStrategy] = Field(default_factory=list)
observers: List[BaseObserver] = Field(default_factory=list)
report_only_initial_ttfb: bool = False
send_initial_empty_metrics: bool = True
start_metadata: Dict[str, Any] = Field(default_factory=dict)
- interruption_strategies: List[BaseInterruptionStrategy] = Field(default_factory=list)
class PipelineTaskSource(FrameProcessor):
@@ -125,7 +131,7 @@ class PipelineTaskSink(FrameProcessor):
await self._down_queue.put(frame)
-class PipelineTask(BaseTask):
+class PipelineTask(BasePipelineTask):
"""Manages the execution of a pipeline, handling frame processing and task lifecycle.
It has a couple of event handlers `on_frame_reached_upstream` and
@@ -172,21 +178,24 @@ class PipelineTask(BaseTask):
Args:
pipeline: The pipeline to execute.
params: Configuration parameters for the pipeline.
- observers: List of observers for monitoring pipeline execution.
- clock: Clock implementation for timing operations.
+ additional_span_attributes: Optional dictionary of attributes to propagate as
+ OpenTelemetry conversation span attributes.
+ cancel_on_idle_timeout: Whether the pipeline task should be cancelled if
+ the idle timeout is reached.
check_dangling_tasks: Whether to check for processors' tasks finishing properly.
+ clock: Clock implementation for timing operations.
+ conversation_id: Optional custom ID for the conversation.
+ enable_tracing: Whether to enable tracing.
+ enable_turn_tracking: Whether to enable turn tracking.
+ enable_watchdog_logging: Whether to print task processing times.
+ idle_timeout_frames: A tuple with the frames that should trigger an idle
+ timeout if not received withing `idle_timeout_seconds`.
idle_timeout_secs: Timeout (in seconds) to consider pipeline idle or
None. If a pipeline is idle the pipeline task will be cancelled
automatically.
- idle_timeout_frames: A tuple with the frames that should trigger an idle
- timeout if not received withing `idle_timeout_seconds`.
- cancel_on_idle_timeout: Whether the pipeline task should be cancelled if
- the idle timeout is reached.
- enable_turn_tracking: Whether to enable turn tracking.
- enable_turn_tracing: Whether to enable turn tracing.
- conversation_id: Optional custom ID for the conversation.
- additional_span_attributes: Optional dictionary of attributes to propagate as
- OpenTelemetry conversation span attributes.
+ observers: List of observers for monitoring pipeline execution.
+ watchdog_timeout_secs: Watchdog timer timeout (in seconds). A warning
+ will be logged if the watchdog timer is not reset before this timeout.
"""
def __init__(
@@ -194,33 +203,37 @@ class PipelineTask(BaseTask):
pipeline: BasePipeline,
*,
params: Optional[PipelineParams] = None,
- observers: Optional[List[BaseObserver]] = None,
- clock: Optional[BaseClock] = None,
- task_manager: Optional[BaseTaskManager] = None,
+ additional_span_attributes: Optional[dict] = None,
+ cancel_on_idle_timeout: bool = True,
check_dangling_tasks: bool = True,
- idle_timeout_secs: Optional[float] = 300,
+ clock: Optional[BaseClock] = None,
+ conversation_id: Optional[str] = None,
+ enable_tracing: bool = False,
+ enable_turn_tracking: bool = True,
+ enable_watchdog_logging: bool = False,
idle_timeout_frames: Tuple[Type[Frame], ...] = (
BotSpeakingFrame,
LLMFullResponseEndFrame,
),
- cancel_on_idle_timeout: bool = True,
- enable_turn_tracking: bool = True,
- enable_tracing: bool = False,
- conversation_id: Optional[str] = None,
- additional_span_attributes: Optional[dict] = None,
+ idle_timeout_secs: Optional[float] = 300,
+ observers: Optional[List[BaseObserver]] = None,
+ task_manager: Optional[BaseTaskManager] = None,
+ watchdog_timeout_secs: float = WATCHDOG_TIMEOUT,
):
super().__init__()
self._pipeline = pipeline
- self._clock = clock or SystemClock()
self._params = params or PipelineParams()
- self._check_dangling_tasks = check_dangling_tasks
- self._idle_timeout_secs = idle_timeout_secs
- self._idle_timeout_frames = idle_timeout_frames
- self._cancel_on_idle_timeout = cancel_on_idle_timeout
- self._enable_turn_tracking = enable_turn_tracking
- self._enable_tracing = enable_tracing and is_tracing_available()
- self._conversation_id = conversation_id
self._additional_span_attributes = additional_span_attributes or {}
+ self._cancel_on_idle_timeout = cancel_on_idle_timeout
+ self._check_dangling_tasks = check_dangling_tasks
+ self._clock = clock or SystemClock()
+ self._conversation_id = conversation_id
+ self._enable_tracing = enable_tracing and is_tracing_available()
+ self._enable_turn_tracking = enable_turn_tracking
+ self._enable_watchdog_logging = enable_watchdog_logging
+ self._idle_timeout_frames = idle_timeout_frames
+ self._idle_timeout_secs = idle_timeout_secs
+ self._watchdog_timeout_secs = watchdog_timeout_secs
if self._params.observers:
import warnings
@@ -322,9 +335,6 @@ class PipelineTask(BaseTask):
async def remove_observer(self, observer: BaseObserver):
await self._observer.remove_observer(observer)
- def set_event_loop(self, loop: asyncio.AbstractEventLoop):
- self._task_manager.set_event_loop(loop)
-
def set_reached_upstream_filter(self, types: Tuple[Type[Frame], ...]):
"""Sets which frames will be checked before calling the
on_frame_reached_upstream event handler.
@@ -358,14 +368,14 @@ class PipelineTask(BaseTask):
"""Stops the running pipeline immediately."""
await self._cancel()
- async def run(self):
+ async def run(self, params: PipelineTaskParams):
"""Starts and manages the pipeline execution until completion or cancellation."""
if self.has_finished():
return
cleanup_pipeline = True
try:
# Setup processors.
- await self._setup()
+ await self._setup(params)
# Create all main tasks and wait of the main push task. This is the
# task that pushes frames to the very beginning of our pipeline (our
@@ -485,7 +495,14 @@ class PipelineTask(BaseTask):
await self._pipeline_end_event.wait()
self._pipeline_end_event.clear()
- async def _setup(self):
+ async def _setup(self, params: PipelineTaskParams):
+ mgr_params = TaskManagerParams(
+ loop=params.loop,
+ enable_watchdog_logging=self._enable_watchdog_logging,
+ watchdog_timeout=self._watchdog_timeout_secs,
+ )
+ self._task_manager.setup(mgr_params)
+
setup = FrameProcessorSetup(
clock=self._clock,
task_manager=self._task_manager,
@@ -509,6 +526,8 @@ class PipelineTask(BaseTask):
await self._pipeline.cleanup()
await self._sink.cleanup()
+ await self._task_manager.cleanup()
+
async def _process_push_queue(self):
"""This is the task that runs the pipeline for the first time by sending
a StartFrame and by pushing any other frames queued by the user. It runs
@@ -646,12 +665,17 @@ class PipelineTask(BaseTask):
"""
running = True
last_frame_time = 0
+ frame_buffer = deque(maxlen=10) # Store last 10 frames
+
while running:
try:
frame = await asyncio.wait_for(
self._idle_queue.get(), timeout=self._idle_timeout_secs
)
+ if not isinstance(frame, InputAudioRawFrame):
+ frame_buffer.append(frame)
+
if isinstance(frame, StartFrame) or isinstance(frame, self._idle_timeout_frames):
# If we find a StartFrame or one of the frames that prevents a
# time out we update the time.
@@ -662,7 +686,7 @@ class PipelineTask(BaseTask):
# valid frames.
diff_time = time.time() - last_frame_time
if diff_time >= self._idle_timeout_secs:
- running = await self._idle_timeout_detected()
+ running = await self._idle_timeout_detected(frame_buffer)
# Reset `last_frame_time` so we don't trigger another
# immediate idle timeout if we are not cancelling. For
# example, we might want to force the bot to say goodbye
@@ -670,15 +694,20 @@ class PipelineTask(BaseTask):
last_frame_time = time.time()
self._idle_queue.task_done()
- except asyncio.TimeoutError:
- running = await self._idle_timeout_detected()
- async def _idle_timeout_detected(self) -> bool:
+ except asyncio.TimeoutError:
+ running = await self._idle_timeout_detected(frame_buffer)
+
+ async def _idle_timeout_detected(self, last_frames: Deque[Frame]) -> bool:
"""Logic for when the pipeline is idle.
Returns:
bool: Whther the pipeline task is being cancelled or not.
"""
+ logger.warning("Idle timeout detected. Last 10 frames received:")
+ for i, frame in enumerate(last_frames, 1):
+ logger.warning(f"Frame {i}: {frame}")
+
await self._call_event_handler("on_idle_timeout")
if self._cancel_on_idle_timeout:
logger.warning(f"Idle pipeline detected, cancelling pipeline task...")
diff --git a/src/pipecat/processors/aggregators/llm_response.py b/src/pipecat/processors/aggregators/llm_response.py
index e9aebb2a0..1984e3cf8 100644
--- a/src/pipecat/processors/aggregators/llm_response.py
+++ b/src/pipecat/processors/aggregators/llm_response.py
@@ -266,6 +266,7 @@ class LLMUserContextAggregator(LLMContextResponseAggregator):
self._user_speaking = False
self._bot_speaking = False
+ self._was_bot_speaking = False
self._emulating_vad = False
self._seen_interim_results = False
self._waiting_for_aggregation = False
@@ -275,6 +276,7 @@ class LLMUserContextAggregator(LLMContextResponseAggregator):
async def reset(self):
await super().reset()
+ self._was_bot_speaking = False
self._seen_interim_results = False
self._waiting_for_aggregation = False
[await s.reset() for s in self._interruption_strategies]
@@ -355,6 +357,20 @@ class LLMUserContextAggregator(LLMContextResponseAggregator):
else:
# No interruption config - normal behavior (always push aggregation)
await self._process_aggregation()
+ # Handles the case where both the user and the bot are not speaking,
+ # and the bot was previously speaking before the user interruption.
+ # Normally, when the user stops speaking, new text is expected,
+ # which triggers the bot to respond. However, if no new text
+ # is received, this safeguard ensures
+ # the bot doesn't hang indefinitely while waiting to speak again.
+ elif not self._seen_interim_results and self._was_bot_speaking and not self._bot_speaking:
+ logger.warning("User stopped speaking but no new aggregation received.")
+ # Resetting it so we don't trigger this twice
+ self._was_bot_speaking = False
+ # TODO: we are not enabling this for now, due to some STT services which can take as long as 2 seconds two return a transcription
+ # So we need more tests and probably make this feature configurable, disabled it by default.
+ # We are just pushing the same previous context to be processed again in this case
+ # await self.push_frame(OpenAILLMContextFrame(self._context))
async def _should_interrupt_based_on_strategies(self) -> bool:
"""Check if interruption should occur based on configured strategies."""
@@ -381,6 +397,7 @@ class LLMUserContextAggregator(LLMContextResponseAggregator):
async def _handle_user_started_speaking(self, frame: UserStartedSpeakingFrame):
self._user_speaking = True
self._waiting_for_aggregation = True
+ self._was_bot_speaking = self._bot_speaking
# If we get a non-emulated UserStartedSpeakingFrame but we are in the
# middle of emulating VAD, let's stop emulating VAD (i.e. don't send the
@@ -393,8 +410,15 @@ class LLMUserContextAggregator(LLMContextResponseAggregator):
# We just stopped speaking. Let's see if there's some aggregation to
# push. If the last thing we saw is an interim transcription, let's wait
# pushing the aggregation as we will probably get a final transcription.
- if not self._seen_interim_results:
- await self.push_aggregation()
+ if len(self._aggregation) > 0:
+ if not self._seen_interim_results:
+ await self.push_aggregation()
+ # Handles the case where both the user and the bot are not speaking,
+ # and the bot was previously speaking before the user interruption.
+ # So in this case we are resetting the aggregation timer
+ elif not self._seen_interim_results and self._was_bot_speaking and not self._bot_speaking:
+ # Reset aggregation timer.
+ self._aggregation_event.set()
async def _handle_bot_started_speaking(self, _: BotStartedSpeakingFrame):
self._bot_speaking = True
diff --git a/src/pipecat/processors/consumer_processor.py b/src/pipecat/processors/consumer_processor.py
index dbdfc97e5..121dd7712 100644
--- a/src/pipecat/processors/consumer_processor.py
+++ b/src/pipecat/processors/consumer_processor.py
@@ -61,5 +61,7 @@ class ConsumerProcessor(FrameProcessor):
async def _consumer_task_handler(self):
while True:
frame = await self._queue.get()
+ self.start_watchdog()
new_frame = await self._transformer(frame)
await self.push_frame(new_frame, self._direction)
+ self.reset_watchdog()
diff --git a/src/pipecat/processors/frame_processor.py b/src/pipecat/processors/frame_processor.py
index 1d2f066ed..bee5ce91c 100644
--- a/src/pipecat/processors/frame_processor.py
+++ b/src/pipecat/processors/frame_processor.py
@@ -51,6 +51,8 @@ class FrameProcessor(BaseObject):
*,
name: Optional[str] = None,
metrics: Optional[FrameProcessorMetrics] = None,
+ enable_watchdog_logging: Optional[bool] = None,
+ watchdog_timeout_secs: Optional[float] = None,
**kwargs,
):
super().__init__(name=name)
@@ -58,6 +60,12 @@ class FrameProcessor(BaseObject):
self._prev: Optional["FrameProcessor"] = None
self._next: Optional["FrameProcessor"] = None
+ # Enable watchdog logging for all tasks created by this frame processor.
+ self._enable_watchdog_logging = enable_watchdog_logging
+
+ # Allow this frame processor to control their tasks timeout.
+ self._watchdog_timeout = watchdog_timeout_secs
+
# Clock
self._clock: Optional[BaseClock] = None
@@ -171,34 +179,56 @@ class FrameProcessor(BaseObject):
await self.stop_ttfb_metrics()
await self.stop_processing_metrics()
- def create_task(self, coroutine: Coroutine, name: Optional[str] = None) -> asyncio.Task:
- if not self._task_manager:
- raise Exception(f"{self} TaskManager is still not initialized.")
+ def create_task(
+ self,
+ coroutine: Coroutine,
+ name: Optional[str] = None,
+ *,
+ enable_watchdog_logging: Optional[bool] = None,
+ watchdog_timeout_secs: Optional[float] = None,
+ ) -> asyncio.Task:
if name:
name = f"{self}::{name}"
else:
name = f"{self}::{coroutine.cr_code.co_name}"
- return self._task_manager.create_task(coroutine, name)
+ return self.get_task_manager().create_task(
+ coroutine,
+ name,
+ enable_watchdog_logging=(
+ enable_watchdog_logging
+ if enable_watchdog_logging
+ else self._enable_watchdog_logging
+ ),
+ watchdog_timeout=(
+ watchdog_timeout_secs if watchdog_timeout_secs else self._watchdog_timeout
+ ),
+ )
async def cancel_task(self, task: asyncio.Task, timeout: Optional[float] = None):
- if not self._task_manager:
- raise Exception(f"{self} TaskManager is still not initialized.")
- await self._task_manager.cancel_task(task, timeout)
+ await self.get_task_manager().cancel_task(task, timeout)
async def wait_for_task(self, task: asyncio.Task, timeout: Optional[float] = None):
- if not self._task_manager:
- raise Exception(f"{self} TaskManager is still not initialized.")
- await self._task_manager.wait_for_task(task, timeout)
+ await self.get_task_manager().wait_for_task(task, timeout)
+
+ def start_watchdog(self):
+ self.get_task_manager().start_watchdog(asyncio.current_task())
+
+ def reset_watchdog(self):
+ self.get_task_manager().reset_watchdog(asyncio.current_task())
async def setup(self, setup: FrameProcessorSetup):
self._clock = setup.clock
self._task_manager = setup.task_manager
self._observer = setup.observer
+ if self._metrics is not None:
+ await self._metrics.setup(self._task_manager)
async def cleanup(self):
await super().cleanup()
await self.__cancel_input_task()
await self.__cancel_push_task()
+ if self._metrics is not None:
+ await self._metrics.cleanup()
def link(self, processor: "FrameProcessor"):
self._next = processor
@@ -206,9 +236,7 @@ class FrameProcessor(BaseObject):
logger.debug(f"Linking {self} -> {self._next}")
def get_event_loop(self) -> asyncio.AbstractEventLoop:
- if not self._task_manager:
- raise Exception(f"{self} TaskManager is still not initialized.")
- return self._task_manager.get_event_loop()
+ return self.get_task_manager().get_event_loop()
def set_parent(self, parent: "FrameProcessor"):
self._parent = parent
@@ -296,11 +324,11 @@ class FrameProcessor(BaseObject):
await self.__cancel_push_task()
async def __pause(self, frame: FrameProcessorPauseFrame | FrameProcessorPauseUrgentFrame):
- if frame.name == self.name:
+ if frame.processor.name == self.name:
await self.pause_processing_frames()
async def __resume(self, frame: FrameProcessorResumeFrame | FrameProcessorResumeUrgentFrame):
- if frame.name == self.name:
+ if frame.processor.name == self.name:
await self.resume_processing_frames()
#
@@ -315,9 +343,8 @@ class FrameProcessor(BaseObject):
# Cancel the input task. This will stop processing queued frames.
await self.__cancel_input_task()
except Exception as e:
- logger.exception(f"Uncaught exception in {self}: {e}")
+ logger.exception(f"Uncaught exception in {self} when handling _start_interruption: {e}")
await self.push_error(ErrorFrame(str(e)))
- raise
# Create a new input queue and task.
self.__create_input_task()
@@ -360,7 +387,6 @@ class FrameProcessor(BaseObject):
except Exception as e:
logger.exception(f"Uncaught exception in {self}: {e}")
await self.push_error(ErrorFrame(str(e)))
- raise
def _check_started(self, frame: Frame):
if not self.__started:
@@ -389,15 +415,19 @@ class FrameProcessor(BaseObject):
logger.trace(f"{self}: frame processing resumed")
(frame, direction, callback) = await self.__input_queue.get()
-
- # Process the frame.
- await self.process_frame(frame, direction)
-
- # If this frame has an associated callback, call it now.
- if callback:
- await callback(self, frame, direction)
-
- self.__input_queue.task_done()
+ try:
+ self.start_watchdog()
+ # Process the frame.
+ await self.process_frame(frame, direction)
+ # If this frame has an associated callback, call it now.
+ if callback:
+ await callback(self, frame, direction)
+ except Exception as e:
+ logger.exception(f"{self}: error processing frame: {e}")
+ await self.push_error(ErrorFrame(str(e)))
+ finally:
+ self.__input_queue.task_done()
+ self.reset_watchdog()
def __create_push_task(self):
if not self.__push_frame_task:
@@ -412,5 +442,7 @@ class FrameProcessor(BaseObject):
async def __push_frame_task_handler(self):
while True:
(frame, direction) = await self.__push_queue.get()
+ self.start_watchdog()
await self.__internal_push_frame(frame, direction)
self.__push_queue.task_done()
+ self.reset_watchdog()
diff --git a/src/pipecat/processors/frameworks/rtvi.py b/src/pipecat/processors/frameworks/rtvi.py
index 771c5a3e2..29e85e5d7 100644
--- a/src/pipecat/processors/frameworks/rtvi.py
+++ b/src/pipecat/processors/frameworks/rtvi.py
@@ -783,14 +783,18 @@ class RTVIProcessor(FrameProcessor):
async def _action_task_handler(self):
while True:
frame = await self._action_queue.get()
+ self.start_watchdog()
await self._handle_action(frame.message_id, frame.rtvi_action_run)
self._action_queue.task_done()
+ self.reset_watchdog()
async def _message_task_handler(self):
while True:
message = await self._message_queue.get()
+ self.start_watchdog()
await self._handle_message(message)
self._message_queue.task_done()
+ self.reset_watchdog()
async def _handle_transport_message(self, frame: TransportMessageUrgentFrame):
try:
diff --git a/src/pipecat/processors/metrics/frame_processor_metrics.py b/src/pipecat/processors/metrics/frame_processor_metrics.py
index b24cfcd61..40a83fa38 100644
--- a/src/pipecat/processors/metrics/frame_processor_metrics.py
+++ b/src/pipecat/processors/metrics/frame_processor_metrics.py
@@ -18,15 +18,29 @@ from pipecat.metrics.metrics import (
TTFBMetricsData,
TTSUsageMetricsData,
)
+from pipecat.utils.asyncio import TaskManager
+from pipecat.utils.base_object import BaseObject
-class FrameProcessorMetrics:
+class FrameProcessorMetrics(BaseObject):
def __init__(self):
+ super().__init__()
+ self._task_manager = None
self._start_ttfb_time = 0
self._start_processing_time = 0
self._last_ttfb_time = 0
self._should_report_ttfb = True
+ async def setup(self, task_manager: TaskManager):
+ self._task_manager = task_manager
+
+ async def cleanup(self):
+ await super().cleanup()
+
+ @property
+ def task_manager(self) -> TaskManager:
+ return self._task_manager
+
@property
def ttfb(self) -> Optional[float]:
"""Get the current TTFB value in seconds.
diff --git a/src/pipecat/processors/metrics/sentry.py b/src/pipecat/processors/metrics/sentry.py
index f3ee40ae0..20d854ffb 100644
--- a/src/pipecat/processors/metrics/sentry.py
+++ b/src/pipecat/processors/metrics/sentry.py
@@ -4,8 +4,12 @@
# SPDX-License-Identifier: BSD 2-Clause License
#
+import asyncio
+
from loguru import logger
+from pipecat.utils.asyncio import TaskManager
+
try:
import sentry_sdk
except ModuleNotFoundError as e:
@@ -24,6 +28,25 @@ class SentryMetrics(FrameProcessorMetrics):
self._sentry_available = sentry_sdk.is_initialized()
if not self._sentry_available:
logger.warning("Sentry SDK not initialized. Sentry features will be disabled.")
+ self._sentry_queue = asyncio.Queue()
+ self._sentry_task = None
+
+ async def setup(self, task_manager: TaskManager):
+ await super().setup(task_manager)
+ if self._sentry_available:
+ self._sentry_queue = asyncio.Queue()
+ self._sentry_task = self.task_manager.create_task(
+ self._sentry_task_handler(), name=f"{self}::_sentry_task_handler"
+ )
+
+ async def cleanup(self):
+ await super().cleanup()
+ if self._sentry_task:
+ await self._sentry_queue.put(None)
+ await self.task_manager.wait_for_task(self._sentry_task)
+ self._sentry_task = None
+ logger.trace(f"{self} Flushing Sentry metrics")
+ sentry_sdk.flush(timeout=5.0)
async def start_ttfb_metrics(self, report_only_initial_ttfb):
await super().start_ttfb_metrics(report_only_initial_ttfb)
@@ -34,14 +57,15 @@ class SentryMetrics(FrameProcessorMetrics):
name=f"TTFB for {self._processor_name()}",
)
logger.debug(
- f"Sentry transaction started (ID: {self._ttfb_metrics_tx.span_id} Name: {self._ttfb_metrics_tx.name})"
+ f"{self} Sentry transaction started (ID: {self._ttfb_metrics_tx.span_id} Name: {self._ttfb_metrics_tx.name})"
)
async def stop_ttfb_metrics(self):
await super().stop_ttfb_metrics()
if self._sentry_available and self._ttfb_metrics_tx:
- self._ttfb_metrics_tx.finish()
+ await self._sentry_queue.put(self._ttfb_metrics_tx)
+ self._ttfb_metrics_tx = None
async def start_processing_metrics(self):
await super().start_processing_metrics()
@@ -52,11 +76,20 @@ class SentryMetrics(FrameProcessorMetrics):
name=f"Processing for {self._processor_name()}",
)
logger.debug(
- f"Sentry transaction started (ID: {self._processing_metrics_tx.span_id} Name: {self._processing_metrics_tx.name})"
+ f"{self} Sentry transaction started (ID: {self._processing_metrics_tx.span_id} Name: {self._processing_metrics_tx.name})"
)
async def stop_processing_metrics(self):
await super().stop_processing_metrics()
if self._sentry_available and self._processing_metrics_tx:
- self._processing_metrics_tx.finish()
+ await self._sentry_queue.put(self._processing_metrics_tx)
+ self._processing_metrics_tx = None
+
+ async def _sentry_task_handler(self):
+ running = True
+ while running:
+ tx = await self._sentry_queue.get()
+ if tx:
+ await self.task_manager.get_event_loop().run_in_executor(None, tx.finish)
+ running = tx is not None
diff --git a/src/pipecat/serializers/telnyx.py b/src/pipecat/serializers/telnyx.py
index 6ef9f7c0d..5f8a09252 100644
--- a/src/pipecat/serializers/telnyx.py
+++ b/src/pipecat/serializers/telnyx.py
@@ -196,8 +196,31 @@ class TelnyxFrameSerializer(FrameSerializer):
async with session.post(endpoint, headers=headers) as response:
if response.status == 200:
logger.info(f"Successfully terminated Telnyx call {call_control_id}")
+ elif response.status == 422:
+ # Handle the case where the call has already ended
+ # Error code 90018: "Call has already ended"
+ # Source: https://developers.telnyx.com/api/errors/90018
+ try:
+ error_data = await response.json()
+ if any(
+ error.get("code") == "90018"
+ for error in error_data.get("errors", [])
+ ):
+ logger.debug(
+ f"Telnyx call {call_control_id} was already terminated"
+ )
+ return
+ except:
+ pass # Fall through to log the raw error
+
+ # Log other 422 errors
+ error_text = await response.text()
+ logger.error(
+ f"Failed to terminate Telnyx call {call_control_id}: "
+ f"Status {response.status}, Response: {error_text}"
+ )
else:
- # Get the error details for better debugging
+ # Log other errors
error_text = await response.text()
logger.error(
f"Failed to terminate Telnyx call {call_control_id}: "
diff --git a/src/pipecat/services/assemblyai/stt.py b/src/pipecat/services/assemblyai/stt.py
index 14d9fb397..09aaa4d25 100644
--- a/src/pipecat/services/assemblyai/stt.py
+++ b/src/pipecat/services/assemblyai/stt.py
@@ -190,6 +190,7 @@ class AssemblyAISTTService(STTService):
while self._connected:
try:
message = await self._websocket.recv()
+ self.start_watchdog()
data = json.loads(message)
await self._handle_message(data)
except websockets.exceptions.ConnectionClosedOK:
@@ -197,6 +198,8 @@ class AssemblyAISTTService(STTService):
except Exception as e:
logger.error(f"Error processing WebSocket message: {e}")
break
+ finally:
+ self.reset_watchdog()
except Exception as e:
logger.error(f"Fatal error in receive handler: {e}")
diff --git a/src/pipecat/services/aws/stt.py b/src/pipecat/services/aws/stt.py
index d6aa1fa5d..c9c1b32d1 100644
--- a/src/pipecat/services/aws/stt.py
+++ b/src/pipecat/services/aws/stt.py
@@ -285,6 +285,9 @@ class AWSTranscribeSTTService(STTService):
try:
response = await self._ws_client.recv()
+
+ self.start_watchdog()
+
headers, payload = decode_event(response)
if headers.get(":message-type") == "event":
@@ -342,3 +345,5 @@ class AWSTranscribeSTTService(STTService):
except Exception as e:
logger.error(f"{self} Unexpected error in receive loop: {e}")
break
+ finally:
+ self.reset_watchdog()
diff --git a/src/pipecat/services/aws_nova_sonic/aws.py b/src/pipecat/services/aws_nova_sonic/aws.py
index d1332c5c4..718e3dc50 100644
--- a/src/pipecat/services/aws_nova_sonic/aws.py
+++ b/src/pipecat/services/aws_nova_sonic/aws.py
@@ -699,6 +699,8 @@ class AWSNovaSonicLLMService(LLMService):
output = await self._stream.await_output()
result = await output[1].receive()
+ self.start_watchdog()
+
if result.value and result.value.bytes_:
response_data = result.value.bytes_.decode("utf-8")
json_data = json.loads(response_data)
@@ -731,6 +733,8 @@ class AWSNovaSonicLLMService(LLMService):
logger.error(f"{self} error processing responses: {e}")
if self._wants_connection:
await self.reset_conversation()
+ finally:
+ self.reset_watchdog()
async def _handle_completion_start_event(self, event_json):
pass
diff --git a/src/pipecat/services/elevenlabs/tts.py b/src/pipecat/services/elevenlabs/tts.py
index e0301360e..ccd9b5b3f 100644
--- a/src/pipecat/services/elevenlabs/tts.py
+++ b/src/pipecat/services/elevenlabs/tts.py
@@ -284,7 +284,6 @@ class ElevenLabsTTSService(AudioContextWordTTSService):
logger.trace(f"{self}: flushing audio")
msg = {"context_id": self._context_id, "flush": True}
await self._websocket.send(json.dumps(msg))
- self._context_id = None
async def push_frame(self, frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM):
await super().push_frame(frame, direction)
@@ -380,6 +379,12 @@ class ElevenLabsTTSService(AudioContextWordTTSService):
if self._context_id and self._websocket:
logger.trace(f"Closing context {self._context_id} due to interruption")
try:
+ # ElevenLabs requires that Pipecat manages the contexts and closes them
+ # when they're not longer in use. Since a StartInterruptionFrame is pushed
+ # every time the user speaks, we'll use this as a trigger to close the context
+ # and reset the state.
+ # Note: We do not need to call remove_audio_context here, as the context is
+ # automatically reset when super ()._handle_interruption is called.
await self._websocket.send(
json.dumps({"context_id": self._context_id, "close_context": True})
)
@@ -391,10 +396,20 @@ class ElevenLabsTTSService(AudioContextWordTTSService):
async def _receive_messages(self):
async for message in self._get_websocket():
msg = json.loads(message)
- # Check if this message belongs to the current context
+
received_ctx_id = msg.get("contextId")
+
+ # Handle final messages first, regardless of context availability
+ # At the moment, this message is received AFTER the close_context message is
+ # sent, so it doesn't serve any functional purpose. For now, we'll just log it.
+ if msg.get("isFinal") is True:
+ logger.trace(f"Received final message for context {received_ctx_id}")
+ continue
+
+ # Check if this message belongs to the current context.
+ # This should never happen, so warn about it.
if not self.audio_context_available(received_ctx_id):
- logger.trace(f"Ignoring message from unavailable context: {received_ctx_id}")
+ logger.warning(f"Ignoring message from unavailable context: {received_ctx_id}")
continue
if msg.get("audio"):
@@ -408,21 +423,26 @@ class ElevenLabsTTSService(AudioContextWordTTSService):
word_times = calculate_word_times(msg["alignment"], self._cumulative_time)
await self.add_word_timestamps(word_times)
self._cumulative_time = word_times[-1][1]
- if msg.get("isFinal"):
- logger.trace(f"Received final message for context {received_ctx_id}")
- await self.remove_audio_context(received_ctx_id)
- # Reset context tracking if this was our active context
- if self._context_id == received_ctx_id:
- self._context_id = None
- self._started = False
async def _keepalive_task_handler(self):
while True:
await asyncio.sleep(10)
try:
- # Send an empty message to keep the connection alive
if self._websocket and self._websocket.open:
- await self._websocket.send(json.dumps({}))
+ if self._context_id:
+ # Send keepalive with context ID to keep the connection alive
+ keepalive_message = {
+ "text": "",
+ "context_id": self._context_id,
+ }
+ logger.trace(f"Sending keepalive for context {self._context_id}")
+ else:
+ # It's possible to have a user interruption which clears the context
+ # without generating a new TTS response. In this case, we'll just send
+ # an empty message to keep the connection alive.
+ keepalive_message = {"text": ""}
+ logger.trace("Sending keepalive without context")
+ await self._websocket.send(json.dumps(keepalive_message))
except websockets.ConnectionClosed as e:
logger.warning(f"{self} keepalive error: {e}")
break
@@ -441,14 +461,6 @@ class ElevenLabsTTSService(AudioContextWordTTSService):
await self._connect()
try:
- # Close previous context if there was one
- if self._context_id and not self._started:
- await self._websocket.send(
- json.dumps({"context_id": self._context_id, "close_context": True})
- )
- await self.remove_audio_context(self._context_id)
- self._context_id = None
-
if not self._started:
await self.start_ttfb_metrics()
yield TTSStartedFrame()
@@ -473,9 +485,6 @@ class ElevenLabsTTSService(AudioContextWordTTSService):
logger.error(f"{self} error sending message: {e}")
yield TTSStoppedFrame()
self._started = False
- if self._context_id:
- await self.remove_audio_context(self._context_id)
- self._context_id = None
return
yield None
except Exception as e:
diff --git a/src/pipecat/services/gemini_multimodal_live/gemini.py b/src/pipecat/services/gemini_multimodal_live/gemini.py
index 139a561b1..434008966 100644
--- a/src/pipecat/services/gemini_multimodal_live/gemini.py
+++ b/src/pipecat/services/gemini_multimodal_live/gemini.py
@@ -736,6 +736,8 @@ class GeminiMultimodalLiveLLMService(LLMService):
async def _receive_task_handler(self):
async for message in self._websocket:
+ self.start_watchdog()
+
evt = events.parse_server_event(message)
# logger.debug(f"Received event: {message[:500]}")
# logger.debug(f"Received event: {evt}")
@@ -764,6 +766,9 @@ class GeminiMultimodalLiveLLMService(LLMService):
logger.warning(f"Received unhandled server event type: {evt}")
pass
+
+ self.reset_watchdog()
+
#
#
#
diff --git a/src/pipecat/services/gladia/stt.py b/src/pipecat/services/gladia/stt.py
index 8e1296f0c..885fe8dc2 100644
--- a/src/pipecat/services/gladia/stt.py
+++ b/src/pipecat/services/gladia/stt.py
@@ -502,6 +502,8 @@ class GladiaSTTService(STTService):
async def _receive_task_handler(self):
try:
async for message in self._websocket:
+ self.start_watchdog()
+
content = json.loads(message)
# Handle audio chunk acknowledgments
@@ -559,11 +561,15 @@ class GladiaSTTService(STTService):
translation, "", time_now_iso8601(), translated_language
)
)
+
+ self.reset_watchdog()
except websockets.exceptions.ConnectionClosed:
# Expected when closing the connection
pass
except Exception as e:
logger.error(f"Error in Gladia WebSocket handler: {e}")
+ finally:
+ self.reset_watchdog()
async def _maybe_reconnect(self) -> bool:
"""Handle exponential backoff reconnection logic."""
diff --git a/src/pipecat/services/google/stt.py b/src/pipecat/services/google/stt.py
index bf60541f5..26186e6bf 100644
--- a/src/pipecat/services/google/stt.py
+++ b/src/pipecat/services/google/stt.py
@@ -747,9 +747,12 @@ class GoogleSTTService(STTService):
try:
while True:
try:
+ self.start_watchdog()
+
if self._request_queue.empty():
# wait for 10ms in case we don't have audio
await asyncio.sleep(0.01)
+ self.reset_watchdog()
continue
# Start bi-directional streaming
@@ -760,12 +763,13 @@ class GoogleSTTService(STTService):
# Process responses
await self._process_responses(streaming_recognize)
+ self.reset_watchdog()
+
# If we're here, check if we need to reconnect
if (int(time.time() * 1000) - self._stream_start_time) > self.STREAMING_LIMIT:
logger.debug("Reconnecting stream after timeout")
# Reset stream start time
self._stream_start_time = int(time.time() * 1000)
- continue
else:
# Normal stream end
break
@@ -775,7 +779,8 @@ class GoogleSTTService(STTService):
await asyncio.sleep(1) # Brief delay before reconnecting
self._stream_start_time = int(time.time() * 1000)
- continue
+ finally:
+ self.reset_watchdog()
except Exception as e:
logger.error(f"Error in streaming task: {e}")
@@ -800,12 +805,16 @@ class GoogleSTTService(STTService):
"""Process streaming recognition responses."""
try:
async for response in streaming_recognize:
+ self.start_watchdog()
+
# Check streaming limit
if (int(time.time() * 1000) - self._stream_start_time) > self.STREAMING_LIMIT:
logger.debug("Stream timeout reached in response processing")
+ self.reset_watchdog()
break
if not response.results:
+ self.reset_watchdog()
continue
for result in response.results:
@@ -848,8 +857,10 @@ class GoogleSTTService(STTService):
)
)
+ self.reset_watchdog()
except Exception as e:
logger.error(f"Error processing Google STT responses: {e}")
-
- # Re-raise the exception to let it propagate (e.g. in the case of a timeout, propagate to _stream_audio to reconnect)
+ self.reset_watchdog()
+ # Re-raise the exception to let it propagate (e.g. in the case of a
+ # timeout, propagate to _stream_audio to reconnect)
raise
diff --git a/src/pipecat/services/mcp_service.py b/src/pipecat/services/mcp_service.py
index ae54b84ff..a644d8f1b 100644
--- a/src/pipecat/services/mcp_service.py
+++ b/src/pipecat/services/mcp_service.py
@@ -8,8 +8,8 @@ from pipecat.adapters.schemas.tools_schema import ToolsSchema
from pipecat.utils.base_object import BaseObject
try:
- from mcp import ClientSession, StdioServerParameters, types
- from mcp.client.session import ClientSession
+ from mcp import ClientSession, StdioServerParameters
+ from mcp.client.session_group import SseServerParameters
from mcp.client.sse import sse_client
from mcp.client.stdio import stdio_client
except ModuleNotFoundError as e:
@@ -21,7 +21,7 @@ except ModuleNotFoundError as e:
class MCPClient(BaseObject):
def __init__(
self,
- server_params: Union[StdioServerParameters, str],
+ server_params: Union[StdioServerParameters, SseServerParameters],
**kwargs,
):
super().__init__(**kwargs)
@@ -30,12 +30,12 @@ class MCPClient(BaseObject):
if isinstance(server_params, StdioServerParameters):
self._client = stdio_client
self._register_tools = self._stdio_register_tools
- elif isinstance(server_params, str):
+ elif isinstance(server_params, SseServerParameters):
self._client = sse_client
self._register_tools = self._sse_register_tools
else:
raise TypeError(
- f"{self} invalid argument type: `server_params` must be either StdioServerParameters or an SSE server url string."
+ f"{self} invalid argument type: `server_params` must be either StdioServerParameters or SseServerParameters."
)
async def register_tools(self, llm) -> ToolsSchema:
@@ -90,7 +90,12 @@ class MCPClient(BaseObject):
logger.debug(f"Executing tool '{function_name}' with call ID: {tool_call_id}")
logger.trace(f"Tool arguments: {json.dumps(arguments, indent=2)}")
try:
- async with self._client(self._server_params) as (read, write):
+ async with self._client(
+ url=self._server_params.url,
+ headers=self._server_params.headers,
+ timeout=self._server_params.timeout,
+ sse_read_timeout=self._server_params.sse_read_timeout,
+ ) as (read, write):
async with self._session(read, write) as session:
await session.initialize()
await self._call_tool(session, function_name, arguments, result_callback)
@@ -100,10 +105,14 @@ class MCPClient(BaseObject):
logger.exception("Full exception details:")
await result_callback(error_msg)
- logger.debug("Starting registration of mcp.run tools")
- tool_schemas: List[FunctionSchema] = []
+ logger.debug(f"SSE server parameters: {self._server_params}")
- async with self._client(self._server_params) as (read, write):
+ async with self._client(
+ url=self._server_params.url,
+ headers=self._server_params.headers,
+ timeout=self._server_params.timeout,
+ sse_read_timeout=self._server_params.sse_read_timeout,
+ ) as (read, write):
async with self._session(read, write) as session:
await session.initialize()
tools_schema = await self._list_tools(session, mcp_tool_wrapper, llm)
diff --git a/src/pipecat/services/openai_realtime_beta/events.py b/src/pipecat/services/openai_realtime_beta/events.py
index 6fa38a8a1..289835dae 100644
--- a/src/pipecat/services/openai_realtime_beta/events.py
+++ b/src/pipecat/services/openai_realtime_beta/events.py
@@ -36,10 +36,6 @@ class InputAudioTranscription(BaseModel):
prompt: Optional[str] = None,
):
super().__init__(model=model, language=language, prompt=prompt)
- if self.model != "gpt-4o-transcribe" and (self.language or self.prompt):
- raise ValueError(
- "Fields 'language' and 'prompt' are only supported when model is 'gpt-4o-transcribe'"
- )
class TurnDetection(BaseModel):
@@ -207,12 +203,11 @@ class ResponseCancelEvent(ClientEvent):
class ServerEvent(BaseModel):
+ model_config = ConfigDict(arbitrary_types_allowed=True)
+
event_id: str
type: str
- class Config:
- arbitrary_types_allowed = True
-
class SessionCreatedEvent(ServerEvent):
type: Literal["session.created"]
diff --git a/src/pipecat/services/openai_realtime_beta/openai.py b/src/pipecat/services/openai_realtime_beta/openai.py
index 4ea5843bf..7f459000a 100644
--- a/src/pipecat/services/openai_realtime_beta/openai.py
+++ b/src/pipecat/services/openai_realtime_beta/openai.py
@@ -86,7 +86,7 @@ class OpenAIRealtimeBetaLLMService(LLMService):
self,
*,
api_key: str,
- model: str = "gpt-4o-realtime-preview-2024-12-17",
+ model: str = "gpt-4o-realtime-preview-2025-06-03",
base_url: str = "wss://api.openai.com/v1/realtime",
session_properties: Optional[events.SessionProperties] = None,
start_audio_paused: bool = False,
@@ -370,6 +370,7 @@ class OpenAIRealtimeBetaLLMService(LLMService):
async def _receive_task_handler(self):
async for message in self._websocket:
+ self.start_watchdog()
evt = events.parse_server_event(message)
if evt.type == "session.created":
await self._handle_evt_session_created(evt)
@@ -400,6 +401,7 @@ class OpenAIRealtimeBetaLLMService(LLMService):
await self._handle_evt_error(evt)
# errors are fatal, so exit the receive loop
return
+ self.reset_watchdog()
@traced_openai_realtime(operation="llm_setup")
async def _handle_evt_session_created(self, evt):
diff --git a/src/pipecat/services/riva/stt.py b/src/pipecat/services/riva/stt.py
index 1e9d8cd6c..91c207c8b 100644
--- a/src/pipecat/services/riva/stt.py
+++ b/src/pipecat/services/riva/stt.py
@@ -224,11 +224,13 @@ class RivaSTTService(STTService):
streaming_config=self._config,
)
for response in responses:
+ self.start_watchdog()
if not response.results:
continue
asyncio.run_coroutine_threadsafe(
self._response_queue.put(response), self.get_event_loop()
)
+ self.reset_watchdog()
async def _thread_task_handler(self):
try:
@@ -283,7 +285,9 @@ class RivaSTTService(STTService):
async def _response_task_handler(self):
while True:
response = await self._response_queue.get()
+ self.start_watchdog()
await self._handle_response(response)
+ self.reset_watchdog()
async def run_stt(self, audio: bytes) -> AsyncGenerator[Frame, None]:
await self.start_ttfb_metrics()
diff --git a/src/pipecat/services/sambanova/__init__.py b/src/pipecat/services/sambanova/__init__.py
new file mode 100644
index 000000000..828b755d2
--- /dev/null
+++ b/src/pipecat/services/sambanova/__init__.py
@@ -0,0 +1,8 @@
+#
+# Copyright (c) 2024–2025, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+from .llm import *
+from .stt import *
diff --git a/src/pipecat/services/sambanova/llm.py b/src/pipecat/services/sambanova/llm.py
new file mode 100644
index 000000000..01a8d294c
--- /dev/null
+++ b/src/pipecat/services/sambanova/llm.py
@@ -0,0 +1,180 @@
+#
+# Copyright (c) 2024–2025, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import json
+from typing import Any, Dict, List, Optional
+
+from loguru import logger
+from openai import AsyncStream
+from openai.types.chat import ChatCompletionChunk, ChatCompletionMessageParam
+
+from pipecat.frames.frames import (
+ LLMTextFrame,
+)
+from pipecat.metrics.metrics import LLMTokenUsage
+from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
+from pipecat.services.llm_service import FunctionCallFromLLM
+from pipecat.services.openai.llm import OpenAILLMService
+from pipecat.utils.tracing.service_decorators import traced_llm
+
+
+class SambaNovaLLMService(OpenAILLMService): # type: ignore
+ """A service for interacting with SambaNova using the OpenAI-compatible interface.
+ This service extends OpenAILLMService to connect to SambaNova's API endpoint while
+ maintaining full compatibility with OpenAI's interface and functionality.
+ Args:
+ api_key (str): The API key for accessing SambaNova API.
+ model (str, optional): The model identifier to use. Defaults to "Meta-Llama-3.3-70B-Instruct".
+ base_url (str, optional): The base URL for SambaNova API. Defaults to "https://api.sambanova.ai/v1".
+ **kwargs: Additional keyword arguments passed to OpenAILLMService.
+ """
+
+ def __init__(
+ self,
+ *,
+ api_key: str,
+ model: str = "Llama-4-Maverick-17B-128E-Instruct",
+ base_url: str = "https://api.sambanova.ai/v1",
+ **kwargs: Dict[Any, Any],
+ ) -> None:
+ super().__init__(api_key=api_key, base_url=base_url, model=model, **kwargs)
+
+ def create_client(
+ self,
+ api_key: Optional[str] = None,
+ base_url: Optional[str] = None,
+ **kwargs: Dict[Any, Any],
+ ) -> Any:
+ """Create OpenAI-compatible client for SambaNova API endpoint."""
+
+ logger.debug(f"Creating SambaNova client with API {base_url}")
+ return super().create_client(api_key, base_url, **kwargs)
+
+ async def get_chat_completions(
+ self, context: OpenAILLMContext, messages: List[ChatCompletionMessageParam]
+ ) -> Any:
+ """Get chat completions from SambaNova API endpoint."""
+
+ params = {
+ "model": self.model_name,
+ "stream": True,
+ "messages": messages,
+ "tools": context.tools,
+ "tool_choice": context.tool_choice,
+ "stream_options": {"include_usage": True},
+ "temperature": self._settings["temperature"],
+ "top_p": self._settings["top_p"],
+ "max_tokens": self._settings["max_tokens"],
+ "max_completion_tokens": self._settings["max_completion_tokens"],
+ }
+
+ params.update(self._settings["extra"])
+
+ chunks = await self._client.chat.completions.create(**params)
+ return chunks
+
+ @traced_llm # type: ignore
+ async def _process_context(self, context: OpenAILLMContext) -> AsyncStream[ChatCompletionChunk]:
+ """Redefine this method until SambaNova API introduces indexing in tool calls."""
+
+ functions_list = []
+ arguments_list = []
+ tool_id_list = []
+ func_idx = 0
+ function_name = ""
+ arguments = ""
+ tool_call_id = ""
+
+ await self.start_ttfb_metrics()
+
+ chunk_stream: AsyncStream[ChatCompletionChunk] = await self._stream_chat_completions(
+ context
+ )
+
+ async for chunk in chunk_stream:
+ if chunk.usage:
+ tokens = LLMTokenUsage(
+ prompt_tokens=chunk.usage.prompt_tokens,
+ completion_tokens=chunk.usage.completion_tokens,
+ total_tokens=chunk.usage.total_tokens,
+ )
+ await self.start_llm_usage_metrics(tokens)
+
+ if chunk.choices is None or len(chunk.choices) == 0:
+ continue
+
+ await self.stop_ttfb_metrics()
+
+ if not chunk.choices[0].delta:
+ continue
+
+ if chunk.choices[0].delta.tool_calls:
+ # We're streaming the LLM response to enable the fastest response times.
+ # For text, we just yield each chunk as we receive it and count on consumers
+ # to do whatever coalescing they need (eg. to pass full sentences to TTS)
+ #
+ # If the LLM is a function call, we'll do some coalescing here.
+ # If the response contains a function name, we'll yield a frame to tell consumers
+ # that they can start preparing to call the function with that name.
+ # We accumulate all the arguments for the rest of the streamed response, then when
+ # the response is done, we package up all the arguments and the function name and
+ # yield a frame containing the function name and the arguments.
+
+ tool_call = chunk.choices[0].delta.tool_calls[0]
+ if tool_call.index != func_idx:
+ functions_list.append(function_name)
+ arguments_list.append(arguments)
+ tool_id_list.append(tool_call_id)
+ function_name = ""
+ arguments = ""
+ tool_call_id = ""
+ func_idx += 1
+ if tool_call.function and tool_call.function.name:
+ function_name += tool_call.function.name
+ tool_call_id = tool_call.id # type: ignore
+ if tool_call.function and tool_call.function.arguments:
+ # Keep iterating through the response to collect all the argument fragments
+ arguments += tool_call.function.arguments
+ elif chunk.choices[0].delta.content:
+ await self.push_frame(LLMTextFrame(chunk.choices[0].delta.content))
+
+ # When gpt-4o-audio / gpt-4o-mini-audio is used for llm or stt+llm
+ # we need to get LLMTextFrame for the transcript
+ elif hasattr(chunk.choices[0].delta, "audio") and chunk.choices[0].delta.audio.get(
+ "transcript"
+ ):
+ await self.push_frame(LLMTextFrame(chunk.choices[0].delta.audio["transcript"]))
+
+ # if we got a function name and arguments, check to see if it's a function with
+ # a registered handler. If so, run the registered callback, save the result to
+ # the context, and re-prompt to get a chat answer. If we don't have a registered
+ # handler, raise an exception.
+ if function_name and arguments:
+ # added to the list as last function name and arguments not added to the list
+ functions_list.append(function_name)
+ arguments_list.append(arguments)
+ tool_id_list.append(tool_call_id)
+
+ function_calls = []
+
+ for function_name, arguments, tool_id in zip(
+ functions_list, arguments_list, tool_id_list
+ ):
+ # This allows compatibility until SambaNova API introduces indexing in tool calls.
+ if len(arguments) < 1:
+ continue
+
+ arguments = json.loads(arguments)
+ function_calls.append(
+ FunctionCallFromLLM(
+ context=context,
+ tool_call_id=tool_id,
+ function_name=function_name,
+ arguments=arguments,
+ )
+ )
+
+ await self.run_function_calls(function_calls)
diff --git a/src/pipecat/services/sambanova/stt.py b/src/pipecat/services/sambanova/stt.py
new file mode 100644
index 000000000..ed518d6b8
--- /dev/null
+++ b/src/pipecat/services/sambanova/stt.py
@@ -0,0 +1,65 @@
+#
+# Copyright (c) 2024–2025, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+from typing import Any, Optional
+
+from pipecat.services.whisper.base_stt import BaseWhisperSTTService, Transcription
+from pipecat.transcriptions.language import Language
+
+
+class SambaNovaSTTService(BaseWhisperSTTService): # type: ignore
+ """SambaNova Whisper speech-to-text service.
+ Uses SambaNova's Whisper API to convert audio to text.
+ Requires a SambaNova API key set via the api_key parameter or SAMBANOVA_API_KEY environment variable.
+ Args:
+ model: Whisper model to use. Defaults to "Whisper-Large-v3".
+ api_key: SambaNova API key. Defaults to None.
+ base_url: API base URL. Defaults to "https://api.sambanova.ai/v1".
+ language: Language of the audio input. Defaults to English.
+ prompt: Optional text to guide the model's style or continue a previous segment.
+ temperature: Optional sampling temperature between 0 and 1. Defaults to 0.0.
+ **kwargs: Additional arguments passed to `pipecat.services.whisper.base_stt.BaseWhisperSTTService`.
+ """
+
+ def __init__(
+ self,
+ *,
+ model: str = "Whisper-Large-v3",
+ api_key: Optional[str] = None,
+ base_url: str = "https://api.sambanova.ai/v1",
+ language: Optional[Language] = Language.EN,
+ prompt: Optional[str] = None,
+ temperature: Optional[float] = None,
+ **kwargs: Any,
+ ) -> None:
+ super().__init__(
+ model=model,
+ api_key=api_key,
+ base_url=base_url,
+ language=language,
+ prompt=prompt,
+ temperature=temperature,
+ **kwargs,
+ )
+
+ async def _transcribe(self, audio: bytes) -> Transcription:
+ assert self._language is not None # Assigned in the BaseWhisperSTTService class
+
+ # Build kwargs dict with only set parameters
+ kwargs = {
+ "file": ("audio.wav", audio, "audio/wav"),
+ "model": self.model_name,
+ "response_format": "json",
+ "language": self._language,
+ }
+
+ if self._prompt is not None:
+ kwargs["prompt"] = self._prompt
+
+ if self._temperature is not None:
+ kwargs["temperature"] = self._temperature
+
+ return await self._client.audio.transcriptions.create(**kwargs)
diff --git a/src/pipecat/services/simli/video.py b/src/pipecat/services/simli/video.py
index 121801e4b..76381b9c6 100644
--- a/src/pipecat/services/simli/video.py
+++ b/src/pipecat/services/simli/video.py
@@ -62,6 +62,7 @@ class SimliVideoService(FrameProcessor):
async def _consume_and_process_audio(self):
await self._pipecat_resampler_event.wait()
async for audio_frame in self._simli_client.getAudioStreamIterator():
+ self.start_watchdog()
resampled_frames = self._pipecat_resampler.resample(audio_frame)
for resampled_frame in resampled_frames:
audio_array = resampled_frame.to_ndarray()
@@ -74,10 +75,12 @@ class SimliVideoService(FrameProcessor):
num_channels=1,
),
)
+ self.reset_watchdog()
async def _consume_and_process_video(self):
await self._pipecat_resampler_event.wait()
async for video_frame in self._simli_client.getVideoStreamIterator(targetFormat="rgb24"):
+ self.start_watchdog()
# Process the video frame
convertedFrame: OutputImageRawFrame = OutputImageRawFrame(
image=video_frame.to_rgb().to_image().tobytes(),
@@ -86,6 +89,7 @@ class SimliVideoService(FrameProcessor):
)
convertedFrame.pts = video_frame.pts
await self.push_frame(convertedFrame)
+ self.reset_watchdog()
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
diff --git a/src/pipecat/services/tavus/video.py b/src/pipecat/services/tavus/video.py
index e6c78813d..4ec744c51 100644
--- a/src/pipecat/services/tavus/video.py
+++ b/src/pipecat/services/tavus/video.py
@@ -217,5 +217,7 @@ class TavusVideoService(AIService):
async def _send_task_handler(self):
while True:
frame = await self._queue.get()
- if isinstance(frame, OutputAudioRawFrame):
+ self.start_watchdog()
+ if isinstance(frame, OutputAudioRawFrame) and self._client:
await self._client.write_audio_frame(frame)
+ self.reset_watchdog()
diff --git a/src/pipecat/transports/base_input.py b/src/pipecat/transports/base_input.py
index 68f58694a..d707c5969 100644
--- a/src/pipecat/transports/base_input.py
+++ b/src/pipecat/transports/base_input.py
@@ -43,6 +43,8 @@ from pipecat.metrics.metrics import MetricsData
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.transports.base_transport import TransportParams
+AUDIO_INPUT_TIMEOUT_SECS = 0.5
+
class BaseInputTransport(FrameProcessor):
def __init__(self, params: TransportParams, **kwargs):
@@ -56,6 +58,9 @@ class BaseInputTransport(FrameProcessor):
# Track bot speaking state for interruption logic
self._bot_speaking = False
+ # Track user speaking state for interruption logic
+ self._user_speaking = False
+
# We read audio from a single queue one at a time and we then run VAD in
# a thread. Therefore, only one thread should be necessary.
self._executor = ThreadPoolExecutor(max_workers=1)
@@ -130,6 +135,7 @@ class BaseInputTransport(FrameProcessor):
async def start(self, frame: StartFrame):
self._paused = False
+ self._user_speaking = False
self._sample_rate = self._params.audio_in_sample_rate or frame.audio_in_sample_rate
@@ -240,6 +246,7 @@ class BaseInputTransport(FrameProcessor):
async def _handle_user_interruption(self, frame: Frame):
if isinstance(frame, UserStartedSpeakingFrame):
logger.debug("User started speaking")
+ self._user_speaking = True
await self.push_frame(frame)
# Only push StartInterruptionFrame if:
@@ -263,6 +270,7 @@ class BaseInputTransport(FrameProcessor):
)
elif isinstance(frame, UserStoppedSpeakingFrame):
logger.debug("User stopped speaking")
+ self._user_speaking = False
await self.push_frame(frame)
if self.interruptions_allowed:
await self._stop_interruption()
@@ -355,26 +363,42 @@ class BaseInputTransport(FrameProcessor):
async def _audio_task_handler(self):
vad_state: VADState = VADState.QUIET
while True:
- frame: InputAudioRawFrame = await self._audio_in_queue.get()
+ try:
+ frame: InputAudioRawFrame = await asyncio.wait_for(
+ self._audio_in_queue.get(), timeout=AUDIO_INPUT_TIMEOUT_SECS
+ )
- # If an audio filter is available, run it before VAD.
- if self._params.audio_in_filter:
- frame.audio = await self._params.audio_in_filter.filter(frame.audio)
+ self.start_watchdog()
- # Check VAD and push event if necessary. We just care about
- # changes from QUIET to SPEAKING and vice versa.
- previous_vad_state = vad_state
- if self._params.vad_analyzer:
- vad_state = await self._handle_vad(frame, vad_state)
+ # If an audio filter is available, run it before VAD.
+ if self._params.audio_in_filter:
+ frame.audio = await self._params.audio_in_filter.filter(frame.audio)
- if self._params.turn_analyzer:
- await self._run_turn_analyzer(frame, vad_state, previous_vad_state)
+ # Check VAD and push event if necessary. We just care about
+ # changes from QUIET to SPEAKING and vice versa.
+ previous_vad_state = vad_state
+ if self._params.vad_analyzer:
+ vad_state = await self._handle_vad(frame, vad_state)
- # Push audio downstream if passthrough is set.
- if self._params.audio_in_passthrough:
- await self.push_frame(frame)
+ if self._params.turn_analyzer:
+ await self._run_turn_analyzer(frame, vad_state, previous_vad_state)
- self._audio_in_queue.task_done()
+ # Push audio downstream if passthrough is set.
+ if self._params.audio_in_passthrough:
+ await self.push_frame(frame)
+
+ self._audio_in_queue.task_done()
+ except asyncio.TimeoutError:
+ if self._user_speaking:
+ logger.warning(
+ "Forcing user stopped speaking due to timeout receiving audio frame!"
+ )
+ vad_state = VADState.QUIET
+ if self._params.turn_analyzer:
+ self._params.turn_analyzer.clear()
+ await self._handle_user_interruption(UserStoppedSpeakingFrame())
+ finally:
+ self.reset_watchdog()
async def _handle_prediction_result(self, result: MetricsData):
"""Handle a prediction result event from the turn analyzer.
diff --git a/src/pipecat/transports/network/fastapi_websocket.py b/src/pipecat/transports/network/fastapi_websocket.py
index 97ff8e03a..7f6256b83 100644
--- a/src/pipecat/transports/network/fastapi_websocket.py
+++ b/src/pipecat/transports/network/fastapi_websocket.py
@@ -70,11 +70,22 @@ class FastAPIWebsocketClient:
return self._websocket.iter_bytes() if self._is_binary else self._websocket.iter_text()
async def send(self, data: str | bytes):
- if self._can_send():
- if self._is_binary:
- await self._websocket.send_bytes(data)
- else:
- await self._websocket.send_text(data)
+ try:
+ if self._can_send():
+ if self._is_binary:
+ await self._websocket.send_bytes(data)
+ else:
+ await self._websocket.send_text(data)
+ except Exception as e:
+ logger.error(
+ f"{self} exception sending data: {e.__class__.__name__} ({e}), application_state: {self._websocket.application_state}"
+ )
+ # For some reason the websocket is disconnected, and we are not able to send data
+ # So let's properly handle it and disconnect the transport
+ if self._websocket.application_state == WebSocketState.DISCONNECTED:
+ logger.warning("Closing already disconnected websocket!")
+ self._closing = True
+ await self.trigger_client_disconnected()
async def disconnect(self):
self._leave_counter -= 1
@@ -171,6 +182,8 @@ class FastAPIWebsocketInputTransport(BaseInputTransport):
if not self._params.serializer:
continue
+ self.start_watchdog()
+
frame = await self._params.serializer.deserialize(message)
if not frame:
@@ -180,9 +193,13 @@ class FastAPIWebsocketInputTransport(BaseInputTransport):
await self.push_audio_frame(frame)
else:
await self.push_frame(frame)
+
+ self.reset_watchdog()
except Exception as e:
logger.error(f"{self} exception receiving data: {e.__class__.__name__} ({e})")
+ self.reset_watchdog()
+
await self._client.trigger_client_disconnected()
async def _monitor_websocket(self):
diff --git a/src/pipecat/transports/network/small_webrtc.py b/src/pipecat/transports/network/small_webrtc.py
index ddc32d06a..5b36fec37 100644
--- a/src/pipecat/transports/network/small_webrtc.py
+++ b/src/pipecat/transports/network/small_webrtc.py
@@ -423,8 +423,10 @@ class SmallWebRTCInputTransport(BaseInputTransport):
async def _receive_audio(self):
try:
async for audio_frame in self._client.read_audio_frame():
+ self.start_watchdog()
if audio_frame:
await self.push_audio_frame(audio_frame)
+ self.reset_watchdog()
except Exception as e:
logger.error(f"{self} exception receiving data: {e.__class__.__name__} ({e})")
@@ -432,6 +434,7 @@ class SmallWebRTCInputTransport(BaseInputTransport):
async def _receive_video(self):
try:
async for video_frame in self._client.read_video_frame():
+ self.start_watchdog()
if video_frame:
await self.push_video_frame(video_frame)
@@ -450,6 +453,7 @@ class SmallWebRTCInputTransport(BaseInputTransport):
await self.push_video_frame(image_frame)
# Remove from pending requests
del self._image_requests[req_id]
+ self.reset_watchdog()
except Exception as e:
logger.error(f"{self} exception receiving data: {e.__class__.__name__} ({e})")
diff --git a/src/pipecat/transports/services/helpers/daily_rest.py b/src/pipecat/transports/services/helpers/daily_rest.py
index 8bc6cf3ce..796022920 100644
--- a/src/pipecat/transports/services/helpers/daily_rest.py
+++ b/src/pipecat/transports/services/helpers/daily_rest.py
@@ -300,6 +300,7 @@ class DailyRESTHelper:
Args:
room_url: Daily room URL
expiry_time: Token validity duration in seconds (default: 1 hour)
+ eject_at_token_exp: Whether to eject user when token expires
owner: Whether token has owner privileges
params: Optional additional token properties. Note that room_name,
exp, and is_owner will be set based on the other function
diff --git a/src/pipecat/transports/services/livekit.py b/src/pipecat/transports/services/livekit.py
index 6381c1dd4..ecb2718c8 100644
--- a/src/pipecat/transports/services/livekit.py
+++ b/src/pipecat/transports/services/livekit.py
@@ -415,6 +415,7 @@ class LiveKitInputTransport(BaseInputTransport):
logger.info("Audio input task started")
while True:
audio_data = await self._client.get_next_audio_frame()
+ self.start_watchdog()
if audio_data:
audio_frame_event, participant_id = audio_data
pipecat_audio_frame = await self._convert_livekit_audio_to_pipecat(
@@ -427,6 +428,7 @@ class LiveKitInputTransport(BaseInputTransport):
num_channels=pipecat_audio_frame.num_channels,
)
await self.push_audio_frame(input_audio_frame)
+ self.reset_watchdog()
async def _convert_livekit_audio_to_pipecat(
self, audio_frame_event: rtc.AudioFrameEvent
diff --git a/src/pipecat/utils/asyncio.py b/src/pipecat/utils/asyncio.py
index cea447329..36479edd4 100644
--- a/src/pipecat/utils/asyncio.py
+++ b/src/pipecat/utils/asyncio.py
@@ -5,15 +5,30 @@
#
import asyncio
+import time
from abc import ABC, abstractmethod
-from typing import Coroutine, Dict, Optional, Sequence, Set
+from dataclasses import dataclass
+from typing import Coroutine, Dict, List, Optional, Sequence
from loguru import logger
+WATCHDOG_TIMEOUT = 5.0
+
+
+@dataclass
+class TaskManagerParams:
+ loop: asyncio.AbstractEventLoop
+ enable_watchdog_logging: bool = False
+ watchdog_timeout: float = WATCHDOG_TIMEOUT
+
class BaseTaskManager(ABC):
@abstractmethod
- def set_event_loop(self, loop: asyncio.AbstractEventLoop):
+ def setup(self, params: TaskManagerParams):
+ pass
+
+ @abstractmethod
+ async def cleanup(self):
pass
@abstractmethod
@@ -21,7 +36,14 @@ class BaseTaskManager(ABC):
pass
@abstractmethod
- def create_task(self, coroutine: Coroutine, name: str) -> asyncio.Task:
+ def create_task(
+ self,
+ coroutine: Coroutine,
+ name: str,
+ *,
+ enable_watchdog_logging: Optional[bool] = None,
+ watchdog_timeout: Optional[float] = None,
+ ) -> asyncio.Task:
"""
Creates and schedules a new asyncio Task that runs the given coroutine.
@@ -31,6 +53,8 @@ class BaseTaskManager(ABC):
loop (asyncio.AbstractEventLoop): The event loop to use for creating the task.
coroutine (Coroutine): The coroutine to be executed within the task.
name (str): The name to assign to the task for identification.
+ enable_watchdog_logging(bool): whether this task should log watchdog processing times.
+ watchdog_timeout(float): watchdog timer timeout for this task.
Returns:
asyncio.Task: The created task object.
@@ -73,21 +97,64 @@ class BaseTaskManager(ABC):
"""Returns the list of currently created/registered tasks."""
pass
+ @abstractmethod
+ def start_watchdog(self, task: asyncio.Task):
+ """Starts the given task watchdog timer. If not reset, a warning will be
+ logged indicating the task is stalling.
+
+ """
+ pass
+
+ @abstractmethod
+ def reset_watchdog(self, task: asyncio.Task):
+ """Resets the given task watchdog timer. If not reset, a warning will be
+ logged indicating the task is stalling.
+
+ """
+ pass
+
+
+@dataclass
+class TaskData:
+ task: asyncio.Task
+ watchdog_start: asyncio.Event
+ watchdog_timer: asyncio.Event
+ enable_watchdog_logging: bool
+ watchdog_timeout: float
+
class TaskManager(BaseTaskManager):
def __init__(self) -> None:
- self._tasks: Dict[str, asyncio.Task] = {}
- self._loop: Optional[asyncio.AbstractEventLoop] = None
+ self._tasks: Dict[str, TaskData] = {}
+ self._params: Optional[TaskManagerParams] = None
+ self._watchdog_tasks: List[asyncio.Task] = []
- def set_event_loop(self, loop: asyncio.AbstractEventLoop):
- self._loop = loop
+ def setup(self, params: TaskManagerParams):
+ if not self._params:
+ self._params = params
+
+ async def cleanup(self):
+ for task in self._watchdog_tasks:
+ try:
+ task.cancel()
+ await task
+ except asyncio.CancelledError:
+ # This is expected, no need to re-raise.
+ pass
def get_event_loop(self) -> asyncio.AbstractEventLoop:
- if not self._loop:
- raise Exception("TaskManager missing event loop, use TaskManager.set_event_loop().")
- return self._loop
+ if not self._params:
+ raise Exception("TaskManager is not setup: unable to get event loop")
+ return self._params.loop
- def create_task(self, coroutine: Coroutine, name: str) -> asyncio.Task:
+ def create_task(
+ self,
+ coroutine: Coroutine,
+ name: str,
+ *,
+ enable_watchdog_logging: Optional[bool] = None,
+ watchdog_timeout: Optional[float] = None,
+ ) -> asyncio.Task:
"""
Creates and schedules a new asyncio Task that runs the given coroutine.
@@ -97,6 +164,8 @@ class TaskManager(BaseTaskManager):
loop (asyncio.AbstractEventLoop): The event loop to use for creating the task.
coroutine (Coroutine): The coroutine to be executed within the task.
name (str): The name to assign to the task for identification.
+ enable_watchdog_logging(bool): whether this task should log watchdog processing time.
+ watchdog_timeout(float): watchdog timer timeout for this task.
Returns:
asyncio.Task: The created task object.
@@ -112,12 +181,26 @@ class TaskManager(BaseTaskManager):
except Exception as e:
logger.exception(f"{name}: unexpected exception: {e}")
- if not self._loop:
- raise Exception("TaskManager missing event loop, use TaskManager.set_event_loop().")
+ if not self._params:
+ raise Exception("TaskManager is not setup: unable to get event loop")
- task = self._loop.create_task(run_coroutine())
+ task = self._params.loop.create_task(run_coroutine())
task.set_name(name)
- self._add_task(task)
+ self._add_task(
+ TaskData(
+ task=task,
+ watchdog_start=asyncio.Event(),
+ watchdog_timer=asyncio.Event(),
+ enable_watchdog_logging=(
+ enable_watchdog_logging
+ if enable_watchdog_logging
+ else self._params.enable_watchdog_logging
+ ),
+ watchdog_timeout=(
+ watchdog_timeout if watchdog_timeout else self._params.watchdog_timeout
+ ),
+ )
+ )
logger.trace(f"{name}: task created")
return task
@@ -165,6 +248,8 @@ class TaskManager(BaseTaskManager):
name = task.get_name()
task.cancel()
try:
+ # Make sure to reset watchdog if a task is cancelled.
+ self.reset_watchdog(task)
if timeout:
await asyncio.wait_for(task, timeout=timeout)
else:
@@ -176,16 +261,51 @@ class TaskManager(BaseTaskManager):
pass
except Exception as e:
logger.exception(f"{name}: unexpected exception while cancelling task: {e}")
+ except BaseException as e:
+ logger.critical(f"{name}: fatal base exception while cancelling task: {e}")
+ raise
finally:
self._remove_task(task)
def current_tasks(self) -> Sequence[asyncio.Task]:
"""Returns the list of currently created/registered tasks."""
- return list(self._tasks.values())
+ return [data.task for data in self._tasks.values()]
- def _add_task(self, task: asyncio.Task):
+ def start_watchdog(self, task: asyncio.Task):
+ """Starts the given task watchdog timer. If not reset, a warning will be
+ logged indicating the task is stalling. If the timer was already started
+ a warning will be logged.
+
+ """
name = task.get_name()
- self._tasks[name] = task
+ if name in self._tasks:
+ if self._tasks[name].watchdog_start.is_set():
+ logger.warning(f"Watchdog timer for task {name} already started")
+ else:
+ self._tasks[name].watchdog_timer.clear()
+ self._tasks[name].watchdog_start.set()
+ else:
+ logger.warning(f"Unable to start watchdog timer: task {name} does not exist")
+
+ def reset_watchdog(self, task: asyncio.Task):
+ """Resets the given task watchdog timer. If not reset, a warning will be
+ logged indicating the task is stalling.
+
+ """
+ name = task.get_name()
+ if name in self._tasks:
+ self._tasks[name].watchdog_start.clear()
+ self._tasks[name].watchdog_timer.set()
+ else:
+ logger.warning(f"Unable to reset watchdog timer: task {name} does not exist")
+
+ def _add_task(self, task_data: TaskData):
+ name = task_data.task.get_name()
+ self._tasks[name] = task_data
+ watchdog_task = self.get_event_loop().create_task(
+ self._watchdog_task_handler(self._tasks[name])
+ )
+ self._watchdog_tasks.append(watchdog_task)
def _remove_task(self, task: asyncio.Task):
name = task.get_name()
@@ -193,3 +313,33 @@ class TaskManager(BaseTaskManager):
del self._tasks[name]
except KeyError as e:
logger.trace(f"{name}: unable to remove task (already removed?): {e}")
+
+ async def _watchdog_task_handler(self, task_data: TaskData):
+ name = task_data.task.get_name()
+ start = task_data.watchdog_start
+ timer = task_data.watchdog_timer
+ enable_watchdog_logging = task_data.enable_watchdog_logging
+ watchdog_timeout = task_data.watchdog_timeout
+
+ async def wait_for_reset():
+ waiting = True
+ while waiting:
+ try:
+ start_time = time.time()
+ await asyncio.wait_for(timer.wait(), timeout=watchdog_timeout)
+ total_time = time.time() - start_time
+ if enable_watchdog_logging:
+ logger.debug(f"{name} task processing time: {total_time:.20f}")
+ waiting = False
+ except asyncio.TimeoutError:
+ logger.warning(
+ f"{name}: task is taking too long {WATCHDOG_TIMEOUT} second(s) (forgot to reset watchdog?)"
+ )
+ finally:
+ timer.clear()
+
+ while True:
+ # Wait for the user to start the watchdog timer.
+ await start.wait()
+ # Now, waiting for the task to finish.
+ await wait_for_reset()
diff --git a/tests/test_pipeline.py b/tests/test_pipeline.py
index 1146681f9..4b2c34828 100644
--- a/tests/test_pipeline.py
+++ b/tests/test_pipeline.py
@@ -17,6 +17,7 @@ from pipecat.frames.frames import (
TextFrame,
)
from pipecat.observers.base_observer import BaseObserver, FramePushed
+from pipecat.pipeline.base_task import PipelineTaskParams
from pipecat.pipeline.parallel_pipeline import ParallelPipeline
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -96,11 +97,10 @@ class TestPipelineTask(unittest.IsolatedAsyncioTestCase):
async def test_task_single(self):
pipeline = Pipeline([IdentityFilter()])
task = PipelineTask(pipeline)
- task.set_event_loop(asyncio.get_event_loop())
await task.queue_frame(TextFrame(text="Hello!"))
await task.queue_frames([TextFrame(text="Bye!"), EndFrame()])
- await task.run()
+ await task.run(PipelineTaskParams(loop=asyncio.get_event_loop()))
assert task.has_finished()
async def test_task_observers(self):
@@ -116,10 +116,9 @@ class TestPipelineTask(unittest.IsolatedAsyncioTestCase):
identity = IdentityFilter()
pipeline = Pipeline([identity])
task = PipelineTask(pipeline, observers=[CustomObserver()])
- task.set_event_loop(asyncio.get_event_loop())
await task.queue_frames([TextFrame(text="Hello Downstream!"), EndFrame()])
- await task.run()
+ await task.run(PipelineTaskParams(loop=asyncio.get_event_loop()))
assert frame_received
async def test_task_add_observer(self):
@@ -156,8 +155,6 @@ class TestPipelineTask(unittest.IsolatedAsyncioTestCase):
observer1 = CustomAddObserver1()
task.add_observer(observer1)
- task.set_event_loop(asyncio.get_event_loop())
-
async def delayed_add_observer():
observer2 = CustomAddObserver2()
# Wait after the pipeline is started and add another observer.
@@ -176,7 +173,9 @@ class TestPipelineTask(unittest.IsolatedAsyncioTestCase):
# Finally end the pipeline.
await task.queue_frame(EndFrame())
- await asyncio.gather(task.run(), delayed_add_observer())
+ await asyncio.gather(
+ task.run(PipelineTaskParams(loop=asyncio.get_event_loop())), delayed_add_observer()
+ )
assert frame_received
assert frame_count_1 == 1
@@ -189,7 +188,6 @@ class TestPipelineTask(unittest.IsolatedAsyncioTestCase):
identity = IdentityFilter()
pipeline = Pipeline([identity])
task = PipelineTask(pipeline)
- task.set_event_loop(asyncio.get_event_loop())
@task.event_handler("on_pipeline_started")
async def on_pipeline_started(task, frame: StartFrame):
@@ -202,7 +200,7 @@ class TestPipelineTask(unittest.IsolatedAsyncioTestCase):
end_received = True
await task.queue_frame(EndFrame())
- await task.run()
+ await task.run(PipelineTaskParams(loop=asyncio.get_event_loop()))
assert start_received
assert end_received
@@ -213,7 +211,6 @@ class TestPipelineTask(unittest.IsolatedAsyncioTestCase):
identity = IdentityFilter()
pipeline = Pipeline([identity])
task = PipelineTask(pipeline)
- task.set_event_loop(asyncio.get_event_loop())
@task.event_handler("on_pipeline_stopped")
async def on_pipeline_ended(task, frame: StopFrame):
@@ -221,7 +218,7 @@ class TestPipelineTask(unittest.IsolatedAsyncioTestCase):
stop_received = True
await task.queue_frame(StopFrame())
- await task.run()
+ await task.run(PipelineTaskParams(loop=asyncio.get_event_loop()))
assert stop_received
@@ -232,7 +229,6 @@ class TestPipelineTask(unittest.IsolatedAsyncioTestCase):
identity = IdentityFilter()
pipeline = Pipeline([identity])
task = PipelineTask(pipeline, cancel_on_idle_timeout=False)
- task.set_event_loop(asyncio.get_event_loop())
task.set_reached_upstream_filter((TextFrame,))
task.set_reached_downstream_filter((TextFrame,))
@@ -254,7 +250,10 @@ class TestPipelineTask(unittest.IsolatedAsyncioTestCase):
await task.queue_frame(TextFrame(text="Hello Downstream!"))
try:
- await asyncio.wait_for(asyncio.shield(task.run()), timeout=1.0)
+ await asyncio.wait_for(
+ asyncio.shield(task.run(PipelineTaskParams(loop=asyncio.get_event_loop()))),
+ timeout=1.0,
+ )
except asyncio.TimeoutError:
pass
@@ -282,13 +281,15 @@ class TestPipelineTask(unittest.IsolatedAsyncioTestCase):
observers=[heartbeats_observer],
cancel_on_idle_timeout=False,
)
- task.set_event_loop(asyncio.get_event_loop())
expected_heartbeats = 1.0 / 0.2
await task.queue_frame(TextFrame(text="Hello!"))
try:
- await asyncio.wait_for(asyncio.shield(task.run()), timeout=1.0)
+ await asyncio.wait_for(
+ asyncio.shield(task.run(PipelineTaskParams(loop=asyncio.get_event_loop()))),
+ timeout=1.0,
+ )
except asyncio.TimeoutError:
pass
assert heartbeats_counter == expected_heartbeats
@@ -297,17 +298,18 @@ class TestPipelineTask(unittest.IsolatedAsyncioTestCase):
identity = IdentityFilter()
pipeline = Pipeline([identity])
task = PipelineTask(pipeline, idle_timeout_secs=0.2)
- task.set_event_loop(asyncio.get_event_loop())
- await task.run()
+ await task.run(PipelineTaskParams(loop=asyncio.get_event_loop()))
assert True
async def test_no_idle_task(self):
identity = IdentityFilter()
pipeline = Pipeline([identity])
task = PipelineTask(pipeline, idle_timeout_secs=0.2, cancel_on_idle_timeout=False)
- task.set_event_loop(asyncio.get_event_loop())
try:
- await asyncio.wait_for(asyncio.shield(task.run()), timeout=0.3)
+ await asyncio.wait_for(
+ asyncio.shield(task.run(PipelineTaskParams(loop=asyncio.get_event_loop()))),
+ timeout=0.3,
+ )
except asyncio.TimeoutError:
assert True
else:
@@ -324,15 +326,13 @@ class TestPipelineTask(unittest.IsolatedAsyncioTestCase):
),
idle_timeout_secs=0.3,
)
- task.set_event_loop(asyncio.get_event_loop())
- await task.run()
+ await task.run(PipelineTaskParams(loop=asyncio.get_event_loop()))
assert True
async def test_idle_task_event_handler_no_frames(self):
identity = IdentityFilter()
pipeline = Pipeline([identity])
task = PipelineTask(pipeline, idle_timeout_secs=0.2, cancel_on_idle_timeout=False)
- task.set_event_loop(asyncio.get_event_loop())
idle_timeout = False
@@ -342,14 +342,13 @@ class TestPipelineTask(unittest.IsolatedAsyncioTestCase):
idle_timeout = True
await task.cancel()
- await task.run()
+ await task.run(PipelineTaskParams(loop=asyncio.get_event_loop()))
assert idle_timeout
async def test_idle_task_event_handler_quiet_user(self):
identity = IdentityFilter()
pipeline = Pipeline([identity])
task = PipelineTask(pipeline, idle_timeout_secs=0.2, cancel_on_idle_timeout=False)
- task.set_event_loop(asyncio.get_event_loop())
idle_timeout = 0
@@ -373,7 +372,9 @@ class TestPipelineTask(unittest.IsolatedAsyncioTestCase):
)
await asyncio.sleep(0.01)
- await asyncio.gather(send_audio(), task.run())
+ await asyncio.gather(
+ send_audio(), task.run(PipelineTaskParams(loop=asyncio.get_event_loop()))
+ )
assert idle_timeout == 1
async def test_idle_task_frames(self):
@@ -387,7 +388,6 @@ class TestPipelineTask(unittest.IsolatedAsyncioTestCase):
idle_timeout_secs=idle_timeout_secs,
idle_timeout_frames=(TextFrame,),
)
- task.set_event_loop(asyncio.get_event_loop())
async def delayed_frames():
await asyncio.sleep(sleep_time_secs)
@@ -399,7 +399,10 @@ class TestPipelineTask(unittest.IsolatedAsyncioTestCase):
start_time = time.time()
- tasks = {asyncio.create_task(task.run()), asyncio.create_task(delayed_frames())}
+ tasks = [
+ asyncio.create_task(task.run(PipelineTaskParams(loop=asyncio.get_event_loop()))),
+ asyncio.create_task(delayed_frames()),
+ ]
await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
diff --git a/tests/test_turn_tracking_observer.py b/tests/test_turn_tracking_observer.py
index 14cfd472f..dd1f39e71 100644
--- a/tests/test_turn_tracking_observer.py
+++ b/tests/test_turn_tracking_observer.py
@@ -9,6 +9,7 @@ import unittest
from pipecat.frames.frames import (
BotStartedSpeakingFrame,
BotStoppedSpeakingFrame,
+ CancelFrame,
UserStartedSpeakingFrame,
UserStoppedSpeakingFrame,
)
@@ -150,7 +151,10 @@ class TestTurnTrackingObserver(unittest.IsolatedAsyncioTestCase):
self.assertEqual(turn_observer._turn_count, 2)
async def test_user_interrupts_bot(self):
- """Test when user interrupts bot speaking, should end current turn and start new one."""
+ """Test when user interrupts bot speaking, should end current turn and start new one.
+
+ Note: This test also verifies that the EndFrame ends the turn correctly.
+ """
# Create observer with a short timeout
turn_observer = TurnTrackingObserver(turn_end_timeout_secs=0.2)
@@ -197,6 +201,7 @@ class TestTurnTrackingObserver(unittest.IsolatedAsyncioTestCase):
"Turn 1 started",
"Turn 1 ended (interrupted: True)", # First turn was interrupted
"Turn 2 started", # New turn started after interruption
+ "Turn 2 ended (interrupted: True)", # Second turn ends due to EndFrame
]
self.assertEqual(turn_events, expected_events)
self.assertEqual(turn_observer._turn_count, 2)
@@ -256,6 +261,109 @@ class TestTurnTrackingObserver(unittest.IsolatedAsyncioTestCase):
self.assertEqual(turn_events, expected_events)
self.assertEqual(turn_observer._turn_count, 1)
+ async def test_cancel_frame_flushes_active_turn(self):
+ """Test that CancelFrame properly flushes an active turn."""
+ # Create observer with a long timeout to ensure CancelFrame is what ends the turn
+ turn_observer = TurnTrackingObserver(turn_end_timeout_secs=5.0)
+
+ # Create identity filter (passes all frames through)
+ processor = IdentityFilter()
+
+ # Record start/end events with turn numbers
+ turn_events = []
+
+ @turn_observer.event_handler("on_turn_started")
+ async def on_turn_started(observer, turn_number):
+ turn_events.append(f"Turn {turn_number} started")
+
+ @turn_observer.event_handler("on_turn_ended")
+ async def on_turn_ended(observer, turn_number, duration, was_interrupted):
+ turn_events.append(f"Turn {turn_number} ended (interrupted: {was_interrupted})")
+
+ frames_to_send = [
+ # Start a turn but don't complete it naturally
+ UserStartedSpeakingFrame(),
+ UserStoppedSpeakingFrame(),
+ BotStartedSpeakingFrame(),
+ # Send CancelFrame while bot is still speaking
+ CancelFrame(),
+ ]
+
+ expected_down_frames = [
+ UserStartedSpeakingFrame,
+ UserStoppedSpeakingFrame,
+ BotStartedSpeakingFrame,
+ CancelFrame,
+ ]
+
+ await run_test(
+ processor,
+ frames_to_send=frames_to_send,
+ expected_down_frames=expected_down_frames,
+ observers=[turn_observer],
+ send_end_frame=False, # Don't send EndFrame since we're testing CancelFrame
+ )
+
+ # Verify that the turn was ended due to CancelFrame (marked as interrupted)
+ expected_events = [
+ "Turn 1 started",
+ "Turn 1 ended (interrupted: True)", # Should be interrupted due to CancelFrame
+ ]
+ self.assertEqual(turn_events, expected_events)
+ self.assertEqual(turn_observer._turn_count, 1)
+
+ async def test_end_frame_with_no_active_turn(self):
+ """Test that EndFrame doesn't cause issues when no turn is active."""
+ # Create observer
+ turn_observer = TurnTrackingObserver(turn_end_timeout_secs=0.2)
+
+ # Create identity filter (passes all frames through)
+ processor = IdentityFilter()
+
+ # Record start/end events with turn numbers
+ turn_events = []
+
+ @turn_observer.event_handler("on_turn_started")
+ async def on_turn_started(observer, turn_number):
+ turn_events.append(f"Turn {turn_number} started")
+
+ @turn_observer.event_handler("on_turn_ended")
+ async def on_turn_ended(observer, turn_number, duration, was_interrupted):
+ turn_events.append(f"Turn {turn_number} ended (interrupted: {was_interrupted})")
+
+ frames_to_send = [
+ # Complete a turn normally
+ UserStartedSpeakingFrame(),
+ UserStoppedSpeakingFrame(),
+ BotStartedSpeakingFrame(),
+ BotStoppedSpeakingFrame(),
+ SleepFrame(sleep=0.4), # Let turn end naturally due to timeout
+ # EndFrame will be sent by run_test when no turn is active
+ ]
+
+ expected_down_frames = [
+ UserStartedSpeakingFrame,
+ UserStoppedSpeakingFrame,
+ BotStartedSpeakingFrame,
+ BotStoppedSpeakingFrame,
+ ]
+
+ await run_test(
+ processor,
+ frames_to_send=frames_to_send,
+ expected_down_frames=expected_down_frames,
+ observers=[turn_observer],
+ send_end_frame=True,
+ )
+
+ # Should only see one turn that ends naturally, EndFrame shouldn't create additional events
+ expected_events = [
+ "Turn 1 started",
+ "Turn 1 ended (interrupted: False)", # Ends due to timeout, not EndFrame
+ ]
+ self.assertEqual(turn_events, expected_events)
+ self.assertEqual(turn_observer._turn_count, 1)
+
if __name__ == "__main__":
unittest.main()