diff --git a/CHANGELOG.md b/CHANGELOG.md index 025731588..59efa9e3a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,136 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- Added Polish support to `AWSTranscribeSTTService`. + +- Added new frames `FrameProcessorPauseFrame` and `FrameProcessorResumeFrame` + which allow pausing and resuming frame processing for a given frame + processor. These are control frames, so they are ordered. Pausing frame + processor will keep old frames in the internal queues until resume takes + place. Frames being pushed while a frame processor is paused will be pushed to + the queues. When frame processing is resumed all queued frames will be + processed in order. Also added `FrameProcessorPauseUrgentFrame` and + `FrameProcessorResumeUrgentFrame` which are system frames and therefore they + have high priority. + +- Added a property called `has_function_calls_in_progress` in + `LLMAssistantContextAggregator` that exposes whether a function call is in + progress. + +### Changed + +- Upgraded `daily-python` to 0.19.3. + +### Fixed + +- Fixed an issue with `GroqTTSService` where it was not properly parsing the + WAV file header. + +- Fixed an issue with `GoogleSTTService` where it was constantly reconnecting + before starting to receive audio from the user. + +- Fixed an issue where `GoogleLLMService`'s TTFB value was incorrect. + +### Other + +- Rename `14e-function-calling-gemini.py` to `14e-function-calling-google.py`. + +## [0.0.71] - 2025-06-10 + +### Added + +- Adds a parameter called `additional_span_attributes` to PipelineTask that + lets you add any additional attributes you'd like to the conversation span. + +### Fixed + +- Fixed an issue with `CartesiaSTTService` initialization. + +## [0.0.70] - 2025-06-10 + +### Added + +- Added `ExotelFrameSerializer` to handle telephony calls via Exotel. + +- Added the option `informal` to `TranslationConfig` on Gladia config. + Allowing to force informal language forms when available. + +- Added `CartesiaSTTService` which is a websocket based implementation to + transcribe audio. Added a foundational example in + `13f-cartesia-transcription.py` + +- Added an `websocket` example, showing how to use the new Pipecat client + `WebsocketTransport` to connect with Pipecat `FastAPIWebsocketTransport` or + `WebsocketServerTransport`. + +- Added language support to `RimeHttpTTSService`. Extended languages to include + German and French for both `RimeTTSService` and `RimeHttpTTSService`. + +### Changed + +- Upgraded `daily-python` to 0.19.2. + +- Make `PipelineTask.add_observer()` synchronous. This allows callers to call it + before doing the work of running the `PipelineTask` (i.e. without invoking + `PipelineTask.set_event_loop()` first). + +- Pipecat 0.0.69 forced `uvloop` event loop on Linux on macOS. Unfortunately, + this is causing issue in some systems. So, `uvloop` is not enabled by default + anymore. If you want to use `uvloop` you can just set the `asyncio` event + policy before starting your agent with: + +```python +asyncio.set_event_loop_policy(uvloop.EventLoopPolicy()) +``` + +### Fixed + +- Fixed an issue with various TTS services that would cause audio glitches at + the start of every bot turn. + +- Fixed an `ElevenLabsTTSService` issue where a context warning was printed + when pushing a `TTSSpeakFrame`. + +- Fixed an `AssemblyAISTTService` issue that could cause unexpected behavior + when yielding empty `Frame()`s. + +- Fixed an issue where `OutputAudioRawFrame.transport_destination` was being + reset to `None` instead of retaining its intended value before sending the + audio frame to `write_audio_frame`. + +- Fixed a typo in Livekit transport that prevented initialization. + +## [0.0.69] - 2025-06-02 "AI Engineer World's Fair release" ✨ + +### Added + +- Added a new frame `FunctionCallsStartedFrame`. This frame is pushed both + upstream and downstream from the LLM service to indicate that one or more + function calls are going to be executed. + +- Added LLM services `on_function_calls_started` event. This event will be + triggered when the LLM service receives function calls from the model and is + going to start executing them. + +- Function calls can now be executed sequentially (in the order received in the + completion) by passing `run_in_parallel=False` when creating your LLM + service. By default, if the LLM completion returns 2 or more function calls + they run concurrently. In both cases, concurrently and sequentially, a new LLM + completion will run when the last function call finishes. + +- Added OpenTelemetry tracing for `GeminiMultimodalLiveLLMService` and + `OpenAIRealtimeBetaLLMService`. + +- Added initial support for interruption strategies, which determine if the user + should interrupt the bot while the bot is speaking. Interruption strategies + can be based on factors such as audio volume or the number of words spoken by + the user. These can be specified via the new `interruption_strategies` field + in `PipelineParams`. A new `MinWordsInterruptionStrategy` strategy has been + introduced which triggers an interruption if the user has spoken a minimum + number of words. If no interruption strategies are specified, the normal + interruption behavior applies. If multiple strategies are provided, the first + one that evaluates to true will trigger the interruption. + - `BaseInputTransport` now handles `StopFrame`. When a `StopFrame` is received the transport will pause sending frames downstream until a new `StartFrame` is received. This allows the transport to be reused (keeping the same connection) @@ -41,6 +171,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `DailyTransport.stop_transcription()` to be able to start and stop Daily transcription dynamically (maybe with different settings). +### Changed + +- Reverted the default model for `GeminiMultimodalLiveLLMService` back to + `models/gemini-2.0-flash-live-001`. + `gemini-2.5-flash-preview-native-audio-dialog` has inconsistent performance. + You can opt in to using this model by setting the `model` arg. + +- Function calls are now cancelled by default if there's an interruption. To + disable this behavior you can set `cancel_on_interruption=False` when + registering the function call. Since function calls are executed as tasks you + can tell if a function call has been cancelled by catching the + `asyncio.CancelledError` exception (and don't forget to raise it again!). + +- Updated OpenTelemetry tracing attribute `metrics.ttfb_ms` to `metrics.ttfb`. + The attribute reports TTFB in seconds. + ### Deprecated - `DailyTransport.send_dtmf()` is deprecated, push an `OutputDTMFFrame` or an @@ -48,6 +194,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- Fixed an issue with `ElevenLabsTTSService` where long responses would + continue generating output even after an interruption. + +- Fixed an issue with the `OpenAILLMContext` where non-Roman characters were + being incorrectly encoded as Unicode escape sequences. This was a logging + issue and did not impact the actual conversation. + - In `AWSBedrockLLMService`, worked around a possible bug in AWS Bedrock where a `toolConfig` is required if there has been previous tool use in the messages array. This workaround includes a no_op factory function call is diff --git a/README.md b/README.md index 3966e8d65..7ec4c6000 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ You can connect to Pipecat from any platform using our official SDKs: | Category | Services | | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Speech-to-Text | [AssemblyAI](https://docs.pipecat.ai/server/services/stt/assemblyai), [AWS](https://docs.pipecat.ai/server/services/stt/aws), [Azure](https://docs.pipecat.ai/server/services/stt/azure), [Deepgram](https://docs.pipecat.ai/server/services/stt/deepgram), [Fal Wizper](https://docs.pipecat.ai/server/services/stt/fal), [Gladia](https://docs.pipecat.ai/server/services/stt/gladia), [Google](https://docs.pipecat.ai/server/services/stt/google), [Groq (Whisper)](https://docs.pipecat.ai/server/services/stt/groq), [OpenAI (Whisper)](https://docs.pipecat.ai/server/services/stt/openai), [Parakeet (NVIDIA)](https://docs.pipecat.ai/server/services/stt/parakeet), [Ultravox](https://docs.pipecat.ai/server/services/stt/ultravox), [Whisper](https://docs.pipecat.ai/server/services/stt/whisper) | +| Speech-to-Text | [AssemblyAI](https://docs.pipecat.ai/server/services/stt/assemblyai), [AWS](https://docs.pipecat.ai/server/services/stt/aws), [Azure](https://docs.pipecat.ai/server/services/stt/azure), [Cartesia](https://docs.pipecat.ai/server/services/stt/cartesia), [Deepgram](https://docs.pipecat.ai/server/services/stt/deepgram), [Fal Wizper](https://docs.pipecat.ai/server/services/stt/fal), [Gladia](https://docs.pipecat.ai/server/services/stt/gladia), [Google](https://docs.pipecat.ai/server/services/stt/google), [Groq (Whisper)](https://docs.pipecat.ai/server/services/stt/groq), [OpenAI (Whisper)](https://docs.pipecat.ai/server/services/stt/openai), [Parakeet (NVIDIA)](https://docs.pipecat.ai/server/services/stt/parakeet), [Ultravox](https://docs.pipecat.ai/server/services/stt/ultravox), [Whisper](https://docs.pipecat.ai/server/services/stt/whisper) | | LLMs | [Anthropic](https://docs.pipecat.ai/server/services/llm/anthropic), [AWS](https://docs.pipecat.ai/server/services/llm/aws), [Azure](https://docs.pipecat.ai/server/services/llm/azure), [Cerebras](https://docs.pipecat.ai/server/services/llm/cerebras), [DeepSeek](https://docs.pipecat.ai/server/services/llm/deepseek), [Fireworks AI](https://docs.pipecat.ai/server/services/llm/fireworks), [Gemini](https://docs.pipecat.ai/server/services/llm/gemini), [Grok](https://docs.pipecat.ai/server/services/llm/grok), [Groq](https://docs.pipecat.ai/server/services/llm/groq), [NVIDIA NIM](https://docs.pipecat.ai/server/services/llm/nim), [Ollama](https://docs.pipecat.ai/server/services/llm/ollama), [OpenAI](https://docs.pipecat.ai/server/services/llm/openai), [OpenRouter](https://docs.pipecat.ai/server/services/llm/openrouter), [Perplexity](https://docs.pipecat.ai/server/services/llm/perplexity), [Qwen](https://docs.pipecat.ai/server/services/llm/qwen), [Together AI](https://docs.pipecat.ai/server/services/llm/together) | | Text-to-Speech | [AWS](https://docs.pipecat.ai/server/services/tts/aws), [Azure](https://docs.pipecat.ai/server/services/tts/azure), [Cartesia](https://docs.pipecat.ai/server/services/tts/cartesia), [Deepgram](https://docs.pipecat.ai/server/services/tts/deepgram), [ElevenLabs](https://docs.pipecat.ai/server/services/tts/elevenlabs), [FastPitch (NVIDIA)](https://docs.pipecat.ai/server/services/tts/fastpitch), [Fish](https://docs.pipecat.ai/server/services/tts/fish), [Google](https://docs.pipecat.ai/server/services/tts/google), [LMNT](https://docs.pipecat.ai/server/services/tts/lmnt), [MiniMax](https://docs.pipecat.ai/server/services/tts/minimax), [Neuphonic](https://docs.pipecat.ai/server/services/tts/neuphonic), [OpenAI](https://docs.pipecat.ai/server/services/tts/openai), [Piper](https://docs.pipecat.ai/server/services/tts/piper), [PlayHT](https://docs.pipecat.ai/server/services/tts/playht), [Rime](https://docs.pipecat.ai/server/services/tts/rime), [Sarvam](https://docs.pipecat.ai/server/services/tts/sarvam), [XTTS](https://docs.pipecat.ai/server/services/tts/xtts) | | Speech-to-Speech | [AWS Nova Sonic](https://docs.pipecat.ai/server/services/s2s/aws), [Gemini Multimodal Live](https://docs.pipecat.ai/server/services/s2s/gemini), [OpenAI Realtime](https://docs.pipecat.ai/server/services/s2s/openai) | diff --git a/dev-requirements.txt b/dev-requirements.txt index af1c35721..2bab71684 100644 --- a/dev-requirements.txt +++ b/dev-requirements.txt @@ -3,11 +3,11 @@ coverage~=7.6.12 grpcio-tools~=1.67.1 pip-tools~=7.4.1 pre-commit~=4.0.1 -pyright~=1.1.397 +pyright~=1.1.400 pytest~=8.3.4 pytest-asyncio~=0.25.3 pytest-aiohttp==1.1.0 -ruff~=0.11.1 +ruff~=0.11.13 setuptools~=70.0.0 setuptools_scm~=8.1.0 python-dotenv~=1.0.1 diff --git a/examples/foundational/01b-livekit-audio.py b/examples/foundational/01b-livekit-audio.py index 732783b05..1b5c1b45a 100644 --- a/examples/foundational/01b-livekit-audio.py +++ b/examples/foundational/01b-livekit-audio.py @@ -77,37 +77,36 @@ async def configure_livekit(): async def main(): - async with aiohttp.ClientSession() as session: - (url, token, room_name) = await configure_livekit() + (url, token, room_name) = await configure_livekit() - transport = LiveKitTransport( - url=url, - token=token, - room_name=room_name, - params=LiveKitParams(audio_out_enabled=True), - ) + transport = LiveKitTransport( + url=url, + token=token, + room_name=room_name, + params=LiveKitParams(audio_out_enabled=True), + ) - tts = CartesiaTTSService( - api_key=os.getenv("CARTESIA_API_KEY"), - voice_id="71a7ad14-091c-4e8e-a314-022ece01c121", # British Reading Lady - ) + tts = CartesiaTTSService( + api_key=os.getenv("CARTESIA_API_KEY"), + voice_id="71a7ad14-091c-4e8e-a314-022ece01c121", # British Reading Lady + ) - runner = PipelineRunner() + runner = PipelineRunner() - task = PipelineTask(Pipeline([tts, transport.output()])) + task = PipelineTask(Pipeline([tts, transport.output()])) - # Register an event handler so we can play the audio when the - # participant joins. - @transport.event_handler("on_first_participant_joined") - async def on_first_participant_joined(transport, participant_id): - await asyncio.sleep(1) - await task.queue_frame( - TextFrame( - "Hello there! How are you doing today? Would you like to talk about the weather?" - ) + # Register an event handler so we can play the audio when the + # participant joins. + @transport.event_handler("on_first_participant_joined") + async def on_first_participant_joined(transport, participant_id): + await asyncio.sleep(1) + await task.queue_frame( + TextFrame( + "Hello there! How are you doing today? Would you like to talk about the weather?" ) + ) - await runner.run(task) + await runner.run(task) if __name__ == "__main__": diff --git a/examples/foundational/04-transports-small-webrtc.py b/examples/foundational/04-transports-small-webrtc.py index c70b566d4..627c13ad1 100644 --- a/examples/foundational/04-transports-small-webrtc.py +++ b/examples/foundational/04-transports-small-webrtc.py @@ -156,7 +156,7 @@ async def offer(request: dict, background_tasks: BackgroundTasks): @asynccontextmanager async def lifespan(app: FastAPI): yield # Run app - coros = [pc.close() for pc in pcs_map.values()] + coros = [pc.disconnect() for pc in pcs_map.values()] await asyncio.gather(*coros) pcs_map.clear() diff --git a/examples/foundational/04a-transports-daily.py b/examples/foundational/04a-transports-daily.py index 98060dbab..a968c3abb 100644 --- a/examples/foundational/04a-transports-daily.py +++ b/examples/foundational/04a-transports-daily.py @@ -9,11 +9,11 @@ import os import sys import aiohttp -from daily_runner import configure from dotenv import load_dotenv from loguru import logger from pipecat.audio.vad.silero import SileroVADAnalyzer +from pipecat.examples.daily_runner import configure from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask diff --git a/examples/foundational/13f-cartesia-transcription.py b/examples/foundational/13f-cartesia-transcription.py new file mode 100644 index 000000000..147d5fb3e --- /dev/null +++ b/examples/foundational/13f-cartesia-transcription.py @@ -0,0 +1,71 @@ +# +# Copyright (c) 2024–2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +import argparse +import os + +from dotenv import load_dotenv +from loguru import logger + +from pipecat.frames.frames import Frame, TranscriptionFrame +from pipecat.pipeline.pipeline import Pipeline +from pipecat.pipeline.runner import PipelineRunner +from pipecat.pipeline.task import PipelineTask +from pipecat.processors.frame_processor import FrameDirection, FrameProcessor +from pipecat.services.cartesia.stt import CartesiaSTTService +from pipecat.transports.base_transport import BaseTransport, TransportParams +from pipecat.transports.network.fastapi_websocket import FastAPIWebsocketParams +from pipecat.transports.services.daily import DailyParams + +load_dotenv(override=True) + + +class TranscriptionLogger(FrameProcessor): + async def process_frame(self, frame: Frame, direction: FrameDirection): + await super().process_frame(frame, direction) + + if isinstance(frame, TranscriptionFrame): + print(f"Transcription: {frame.text}") + + +# We store functions so objects (e.g. SileroVADAnalyzer) don't get +# instantiated. The function will be called when the desired transport gets +# selected. +transport_params = { + "daily": lambda: DailyParams(audio_in_enabled=True), + "twilio": lambda: FastAPIWebsocketParams(audio_in_enabled=True), + "webrtc": lambda: TransportParams(audio_in_enabled=True), +} + + +async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_sigint: bool): + logger.info(f"Starting bot") + + stt = CartesiaSTTService( + api_key=os.getenv("CARTESIA_API_KEY"), + base_url=os.getenv("CARTESIA_BASE_URL"), + ) + + tl = TranscriptionLogger() + + pipeline = Pipeline([transport.input(), stt, tl]) + + task = PipelineTask(pipeline) + + @transport.event_handler("on_client_disconnected") + async def on_client_disconnected(transport, client): + logger.info(f"Client disconnected") + await task.cancel() + + runner = PipelineRunner(handle_sigint=handle_sigint) + + await runner.run(task) + + +if __name__ == "__main__": + from pipecat.examples.run import main + + main(run_example, transport_params=transport_params) diff --git a/examples/foundational/14-function-calling.py b/examples/foundational/14-function-calling.py index 7c80416b1..b4e09c99a 100644 --- a/examples/foundational/14-function-calling.py +++ b/examples/foundational/14-function-calling.py @@ -30,10 +30,13 @@ load_dotenv(override=True) async def fetch_weather_from_api(params: FunctionCallParams): - await params.llm.push_frame(TTSSpeakFrame("Let me check on that.")) await params.result_callback({"conditions": "nice", "temperature": "75"}) +async def fetch_restaurant_recommendation(params: FunctionCallParams): + await params.result_callback({"name": "The Golden Dragon"}) + + # We store functions so objects (e.g. SileroVADAnalyzer) don't get # instantiated. The function will be called when the desired transport gets # selected. @@ -71,6 +74,11 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si # You can also register a function_name of None to get all functions # sent to the same callback with an additional function_name parameter. llm.register_function("get_current_weather", fetch_weather_from_api) + llm.register_function("get_restaurant_recommendation", fetch_restaurant_recommendation) + + @llm.event_handler("on_function_calls_started") + async def on_function_calls_started(service, function_calls): + await tts.queue_frame(TTSSpeakFrame("Let me check on that.")) weather_function = FunctionSchema( name="get_current_weather", @@ -88,7 +96,18 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si }, required=["location", "format"], ) - tools = ToolsSchema(standard_tools=[weather_function]) + restaurant_function = FunctionSchema( + name="get_restaurant_recommendation", + description="Get a restaurant recommendation", + properties={ + "location": { + "type": "string", + "description": "The city and state, e.g. San Francisco, CA", + }, + }, + required=["location"], + ) + tools = ToolsSchema(standard_tools=[weather_function, restaurant_function]) messages = [ { diff --git a/examples/foundational/14a-function-calling-anthropic.py b/examples/foundational/14a-function-calling-anthropic.py index f46a42db1..bf34e211c 100644 --- a/examples/foundational/14a-function-calling-anthropic.py +++ b/examples/foundational/14a-function-calling-anthropic.py @@ -33,6 +33,10 @@ async def get_weather(params: FunctionCallParams): await params.result_callback(f"The weather in {location} is currently 72 degrees and sunny.") +async def fetch_restaurant_recommendation(params: FunctionCallParams): + await params.result_callback({"name": "The Golden Dragon"}) + + # We store functions so objects (e.g. SileroVADAnalyzer) don't get # instantiated. The function will be called when the desired transport gets # selected. @@ -66,9 +70,11 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si ) llm = AnthropicLLMService( - api_key=os.getenv("ANTHROPIC_API_KEY"), model="claude-3-7-sonnet-latest" + api_key=os.getenv("ANTHROPIC_API_KEY"), + model="claude-3-7-sonnet-latest", ) llm.register_function("get_weather", get_weather) + llm.register_function("get_restaurant_recommendation", fetch_restaurant_recommendation) weather_function = FunctionSchema( name="get_weather", @@ -81,7 +87,18 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si }, required=["location"], ) - tools = ToolsSchema(standard_tools=[weather_function]) + restaurant_function = FunctionSchema( + name="get_restaurant_recommendation", + description="Get a restaurant recommendation", + properties={ + "location": { + "type": "string", + "description": "The city and state, e.g. San Francisco, CA", + }, + }, + required=["location"], + ) + tools = ToolsSchema(standard_tools=[weather_function, restaurant_function]) # todo: test with very short initial user message diff --git a/examples/foundational/14c-function-calling-together.py b/examples/foundational/14c-function-calling-together.py index 54545b0b2..1188f32e2 100644 --- a/examples/foundational/14c-function-calling-together.py +++ b/examples/foundational/14c-function-calling-together.py @@ -30,7 +30,6 @@ load_dotenv(override=True) async def fetch_weather_from_api(params: FunctionCallParams): - await params.llm.push_frame(TTSSpeakFrame("Let me check on that.")) await params.result_callback({"conditions": "nice", "temperature": "75"}) @@ -74,6 +73,10 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si # sent to the same callback with an additional function_name parameter. llm.register_function("get_current_weather", fetch_weather_from_api) + @llm.event_handler("on_function_calls_started") + async def on_function_calls_started(service, function_calls): + await tts.queue_frame(TTSSpeakFrame("Let me check on that.")) + weather_function = FunctionSchema( name="get_current_weather", description="Get the current weather", diff --git a/examples/foundational/14e-function-calling-gemini.py b/examples/foundational/14e-function-calling-google.py similarity index 87% rename from examples/foundational/14e-function-calling-gemini.py rename to examples/foundational/14e-function-calling-google.py index 4e6a582f3..ccbe952fb 100644 --- a/examples/foundational/14e-function-calling-gemini.py +++ b/examples/foundational/14e-function-calling-google.py @@ -35,11 +35,14 @@ client_id = "" async def get_weather(params: FunctionCallParams): - await params.llm.push_frame(TTSSpeakFrame("Let me check on that.")) location = params.arguments["location"] await params.result_callback(f"The weather in {location} is currently 72 degrees and sunny.") +async def fetch_restaurant_recommendation(params: FunctionCallParams): + await params.result_callback({"name": "The Golden Dragon"}) + + async def get_image(params: FunctionCallParams): question = params.arguments["question"] logger.debug(f"Requesting image with user_id={client_id}, question={question}") @@ -93,6 +96,11 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si llm = GoogleLLMService(api_key=os.getenv("GOOGLE_API_KEY"), model="gemini-2.0-flash-001") llm.register_function("get_weather", get_weather) llm.register_function("get_image", get_image) + llm.register_function("get_restaurant_recommendation", fetch_restaurant_recommendation) + + @llm.event_handler("on_function_calls_started") + async def on_function_calls_started(service, function_calls): + await tts.queue_frame(TTSSpeakFrame("Let me check on that.")) weather_function = FunctionSchema( name="get_weather", @@ -110,6 +118,17 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si }, required=["location", "format"], ) + restaurant_function = FunctionSchema( + name="get_restaurant_recommendation", + description="Get a restaurant recommendation", + properties={ + "location": { + "type": "string", + "description": "The city and state, e.g. San Francisco, CA", + }, + }, + required=["location"], + ) get_image_function = FunctionSchema( name="get_image", description="Get an image from the video stream.", @@ -121,14 +140,14 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si }, required=["question"], ) - tools = ToolsSchema(standard_tools=[weather_function, get_image_function]) + tools = ToolsSchema(standard_tools=[weather_function, get_image_function, restaurant_function]) system_prompt = """\ You are a helpful assistant who converses with a user and answers questions. Respond concisely to general questions. Your response will be turned into speech so use only simple words and punctuation. -You have access to two tools: get_weather and get_image. +You have access to three tools: get_weather, get_restaurant_recommendation, and get_image. You can respond to questions about the weather using the get_weather tool. diff --git a/examples/foundational/14f-function-calling-groq.py b/examples/foundational/14f-function-calling-groq.py index 698845028..3ff56a915 100644 --- a/examples/foundational/14f-function-calling-groq.py +++ b/examples/foundational/14f-function-calling-groq.py @@ -31,7 +31,6 @@ load_dotenv(override=True) async def fetch_weather_from_api(params: FunctionCallParams): - await params.llm.push_frame(TTSSpeakFrame("Let me check on that.")) await params.result_callback({"conditions": "nice", "temperature": "75"}) @@ -74,6 +73,10 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si # sent to the same callback with an additional function_name parameter. llm.register_function("get_current_weather", fetch_weather_from_api) + @llm.event_handler("on_function_calls_started") + async def on_function_calls_started(service, function_calls): + await tts.queue_frame(TTSSpeakFrame("Let me check on that.")) + weather_function = FunctionSchema( name="get_current_weather", description="Get the current weather", diff --git a/examples/foundational/14h-function-calling-azure.py b/examples/foundational/14h-function-calling-azure.py index a54fb1429..25cf7defa 100644 --- a/examples/foundational/14h-function-calling-azure.py +++ b/examples/foundational/14h-function-calling-azure.py @@ -30,7 +30,6 @@ load_dotenv(override=True) async def fetch_weather_from_api(params: FunctionCallParams): - await params.llm.push_frame(TTSSpeakFrame("Let me check on that.")) await params.result_callback({"conditions": "nice", "temperature": "75"}) @@ -75,6 +74,10 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si # sent to the same callback with an additional function_name parameter. llm.register_function("get_current_weather", fetch_weather_from_api) + @llm.event_handler("on_function_calls_started") + async def on_function_calls_started(service, function_calls): + await tts.queue_frame(TTSSpeakFrame("Let me check on that.")) + weather_function = FunctionSchema( name="get_current_weather", description="Get the current weather", diff --git a/examples/foundational/14i-function-calling-fireworks.py b/examples/foundational/14i-function-calling-fireworks.py index 5467e685a..028d9fa64 100644 --- a/examples/foundational/14i-function-calling-fireworks.py +++ b/examples/foundational/14i-function-calling-fireworks.py @@ -30,7 +30,6 @@ load_dotenv(override=True) async def fetch_weather_from_api(params: FunctionCallParams): - await params.llm.push_frame(TTSSpeakFrame("Let me check on that.")) await params.result_callback({"conditions": "nice", "temperature": "75"}) @@ -74,6 +73,10 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si # sent to the same callback with an additional function_name parameter. llm.register_function("get_current_weather", fetch_weather_from_api) + @llm.event_handler("on_function_calls_started") + async def on_function_calls_started(service, function_calls): + await tts.queue_frame(TTSSpeakFrame("Let me check on that.")) + weather_function = FunctionSchema( name="get_current_weather", description="Get the current weather", diff --git a/examples/foundational/14j-function-calling-nim.py b/examples/foundational/14j-function-calling-nim.py index f9b437f83..d254e0d4f 100644 --- a/examples/foundational/14j-function-calling-nim.py +++ b/examples/foundational/14j-function-calling-nim.py @@ -30,7 +30,6 @@ load_dotenv(override=True) async def fetch_weather_from_api(params: FunctionCallParams): - await params.llm.push_frame(TTSSpeakFrame("Let me check on that.")) await params.result_callback({"conditions": "nice", "temperature": "75"}) @@ -72,6 +71,10 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si # sent to the same callback with an additional function_name parameter. llm.register_function("get_current_weather", fetch_weather_from_api) + @llm.event_handler("on_function_calls_started") + async def on_function_calls_started(service, function_calls): + await tts.queue_frame(TTSSpeakFrame("Let me check on that.")) + weather_function = FunctionSchema( name="get_current_weather", description="Get the current weather", diff --git a/examples/foundational/14k-function-calling-cerebras.py b/examples/foundational/14k-function-calling-cerebras.py index 7aa5a2af4..70ffceffd 100644 --- a/examples/foundational/14k-function-calling-cerebras.py +++ b/examples/foundational/14k-function-calling-cerebras.py @@ -30,7 +30,6 @@ load_dotenv(override=True) async def fetch_weather_from_api(params: FunctionCallParams): - await params.llm.push_frame(TTSSpeakFrame("Let me check on that.")) await params.result_callback({"conditions": "nice", "temperature": "75"}) @@ -71,6 +70,10 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si # sent to the same callback with an additional function_name parameter. llm.register_function("get_current_weather", fetch_weather_from_api) + @llm.event_handler("on_function_calls_started") + async def on_function_calls_started(service, function_calls): + await tts.queue_frame(TTSSpeakFrame("Let me check on that.")) + weather_function = FunctionSchema( name="get_current_weather", description="Get the current weather", diff --git a/examples/foundational/14l-function-calling-deepseek.py b/examples/foundational/14l-function-calling-deepseek.py index 0daa814e6..7e992942d 100644 --- a/examples/foundational/14l-function-calling-deepseek.py +++ b/examples/foundational/14l-function-calling-deepseek.py @@ -30,7 +30,6 @@ load_dotenv(override=True) async def fetch_weather_from_api(params: FunctionCallParams): - await params.llm.push_frame(TTSSpeakFrame("Let me check on that.")) await params.result_callback({"conditions": "nice", "temperature": "75"}) @@ -71,6 +70,10 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si # sent to the same callback with an additional function_name parameter. llm.register_function("get_current_weather", fetch_weather_from_api) + @llm.event_handler("on_function_calls_started") + async def on_function_calls_started(service, function_calls): + await tts.queue_frame(TTSSpeakFrame("Let me check on that.")) + weather_function = FunctionSchema( name="get_current_weather", description="Get the current weather", diff --git a/examples/foundational/14m-function-calling-openrouter.py b/examples/foundational/14m-function-calling-openrouter.py index 13487b3b6..023f725f6 100644 --- a/examples/foundational/14m-function-calling-openrouter.py +++ b/examples/foundational/14m-function-calling-openrouter.py @@ -30,7 +30,6 @@ load_dotenv(override=True) async def fetch_weather_from_api(params: FunctionCallParams): - await params.llm.push_frame(TTSSpeakFrame("Let me check on that.")) await params.result_callback({"conditions": "nice", "temperature": "75"}) @@ -75,6 +74,10 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si # sent to the same callback with an additional function_name parameter. llm.register_function("get_current_weather", fetch_weather_from_api) + @llm.event_handler("on_function_calls_started") + async def on_function_calls_started(service, function_calls): + await tts.queue_frame(TTSSpeakFrame("Let me check on that.")) + weather_function = FunctionSchema( name="get_current_weather", description="Get the current weather", diff --git a/examples/foundational/14o-function-calling-gemini-openai-format.py b/examples/foundational/14o-function-calling-gemini-openai-format.py index 8e1e0fdf1..8f0036e07 100644 --- a/examples/foundational/14o-function-calling-gemini-openai-format.py +++ b/examples/foundational/14o-function-calling-gemini-openai-format.py @@ -30,7 +30,6 @@ load_dotenv(override=True) async def fetch_weather_from_api(params: FunctionCallParams): - await params.llm.push_frame(TTSSpeakFrame("Let me check on that.")) await params.result_callback({"conditions": "nice", "temperature": "75"}) @@ -71,6 +70,10 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si # sent to the same callback with an additional function_name parameter. llm.register_function("get_current_weather", fetch_weather_from_api) + @llm.event_handler("on_function_calls_started") + async def on_function_calls_started(service, function_calls): + await tts.queue_frame(TTSSpeakFrame("Let me check on that.")) + weather_function = FunctionSchema( name="get_current_weather", description="Get the current weather", diff --git a/examples/foundational/14p-function-calling-gemini-vertex-ai.py b/examples/foundational/14p-function-calling-gemini-vertex-ai.py index 42567787a..4348b663a 100644 --- a/examples/foundational/14p-function-calling-gemini-vertex-ai.py +++ b/examples/foundational/14p-function-calling-gemini-vertex-ai.py @@ -30,7 +30,6 @@ load_dotenv(override=True) async def fetch_weather_from_api(params: FunctionCallParams): - await params.llm.push_frame(TTSSpeakFrame("Let me check on that.")) await params.result_callback({"conditions": "nice", "temperature": "75"}) @@ -76,6 +75,10 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si # sent to the same callback with an additional function_name parameter. llm.register_function("get_current_weather", fetch_weather_from_api) + @llm.event_handler("on_function_calls_started") + async def on_function_calls_started(service, function_calls): + await tts.queue_frame(TTSSpeakFrame("Let me check on that.")) + weather_function = FunctionSchema( name="get_current_weather", description="Get the current weather", diff --git a/examples/foundational/14q-function-calling-qwen.py b/examples/foundational/14q-function-calling-qwen.py index d3740d25f..6e07a97e1 100644 --- a/examples/foundational/14q-function-calling-qwen.py +++ b/examples/foundational/14q-function-calling-qwen.py @@ -30,7 +30,6 @@ load_dotenv(override=True) async def fetch_weather_from_api(params: FunctionCallParams): - await params.llm.push_frame(TTSSpeakFrame("Let me check on that.")) await params.result_callback({"conditions": "nice", "temperature": "75"}) @@ -72,6 +71,10 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si # sent to the same callback with an additional function_name parameter. llm.register_function("get_current_weather", fetch_weather_from_api) + @llm.event_handler("on_function_calls_started") + async def on_function_calls_started(service, function_calls): + await tts.queue_frame(TTSSpeakFrame("Let me check on that.")) + weather_function = FunctionSchema( name="get_current_weather", description="Get the current weather", diff --git a/examples/foundational/14r-function-calling-aws.py b/examples/foundational/14r-function-calling-aws.py index f7796157a..4443bec27 100644 --- a/examples/foundational/14r-function-calling-aws.py +++ b/examples/foundational/14r-function-calling-aws.py @@ -32,6 +32,10 @@ async def fetch_weather_from_api(params: FunctionCallParams): await params.result_callback({"conditions": "nice", "temperature": "75"}) +async def fetch_restaurant_recommendation(params: FunctionCallParams): + await params.result_callback({"name": "The Golden Dragon"}) + + # We store functions so objects (e.g. SileroVADAnalyzer) don't get # instantiated. The function will be called when the desired transport gets # selected. @@ -74,6 +78,7 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si # You can also register a function_name of None to get all functions # sent to the same callback with an additional function_name parameter. llm.register_function("get_current_weather", fetch_weather_from_api) + llm.register_function("get_restaurant_recommendation", fetch_restaurant_recommendation) weather_function = FunctionSchema( name="get_current_weather", @@ -91,7 +96,18 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si }, required=["location", "format"], ) - tools = ToolsSchema(standard_tools=[weather_function]) + restaurant_function = FunctionSchema( + name="get_restaurant_recommendation", + description="Get a restaurant recommendation", + properties={ + "location": { + "type": "string", + "description": "The city and state, e.g. San Francisco, CA", + }, + }, + required=["location"], + ) + tools = ToolsSchema(standard_tools=[weather_function, restaurant_function]) messages = [ { diff --git a/examples/foundational/19-openai-realtime-beta.py b/examples/foundational/19-openai-realtime-beta.py index f1dd7269d..9eb816432 100644 --- a/examples/foundational/19-openai-realtime-beta.py +++ b/examples/foundational/19-openai-realtime-beta.py @@ -14,10 +14,12 @@ from loguru import logger from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.audio.vad.silero import SileroVADAnalyzer +from pipecat.frames.frames import TranscriptionMessage from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext +from pipecat.processors.transcript_processor import TranscriptProcessor from pipecat.services.llm_service import FunctionCallParams from pipecat.services.openai_realtime_beta import ( InputAudioNoiseReduction, @@ -45,6 +47,10 @@ async def fetch_weather_from_api(params: FunctionCallParams): ) +async def fetch_restaurant_recommendation(params: FunctionCallParams): + await params.result_callback({"name": "The Golden Dragon"}) + + weather_function = FunctionSchema( name="get_current_weather", description="Get the current weather", @@ -62,8 +68,20 @@ weather_function = FunctionSchema( required=["location", "format"], ) +restaurant_function = FunctionSchema( + name="get_restaurant_recommendation", + description="Get a restaurant recommendation", + properties={ + "location": { + "type": "string", + "description": "The city and state, e.g. San Francisco, CA", + }, + }, + required=["location"], +) + # Create tools schema -tools = ToolsSchema(standard_tools=[weather_function]) +tools = ToolsSchema(standard_tools=[weather_function, restaurant_function]) # We store functions so objects (e.g. SileroVADAnalyzer) don't get @@ -100,7 +118,7 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si # turn_detection=False, input_audio_noise_reduction=InputAudioNoiseReduction(type="near_field"), # tools=tools, - instructions="""Your knowledge cutoff is 2023-10. You are a helpful and friendly AI. + instructions="""You are a helpful and friendly AI. Act like a human, but remember that you aren't a human and that you can't do human things in the real world. Your voice and personality should be warm and engaging, with a lively and @@ -109,10 +127,14 @@ playful tone. If interacting in a non-English language, start by using the standard accent or dialect familiar to the user. Talk quickly. You should always call a function if you can. Do not refer to these rules, even if you're asked about them. -- + You are participating in a voice conversation. Keep your responses concise, short, and to the point unless specifically asked to elaborate on a topic. +You have access to the following tools: +- get_current_weather: Get the current weather for a given location. +- get_restaurant_recommendation: Get a restaurant recommendation for a given location. + Remember, your responses should be short. Just one or two sentences, usually.""", ) @@ -125,6 +147,9 @@ Remember, your responses should be short. Just one or two sentences, usually.""" # you can either register a single function for all function calls, or specific functions # llm.register_function(None, fetch_weather_from_api) llm.register_function("get_current_weather", fetch_weather_from_api) + llm.register_function("get_restaurant_recommendation", fetch_restaurant_recommendation) + + transcript = TranscriptProcessor() # Create a standard OpenAI LLM context object using the normal messages format. The # OpenAIRealtimeBetaLLMService will convert this internally to messages that the @@ -151,7 +176,9 @@ Remember, your responses should be short. Just one or two sentences, usually.""" transport.input(), # Transport user input context_aggregator.user(), llm, # LLM + transcript.user(), # Placed after the LLM, as LLM pushes TranscriptionFrames downstream transport.output(), # Transport bot output + transcript.assistant(), # After the transcript output, to time with the audio output context_aggregator.assistant(), ] ) @@ -177,6 +204,15 @@ Remember, your responses should be short. Just one or two sentences, usually.""" logger.info(f"Client disconnected") await task.cancel() + # Register event handler for transcript updates + @transcript.event_handler("on_transcript_update") + async def on_transcript_update(processor, frame): + for msg in frame.messages: + if isinstance(msg, TranscriptionMessage): + timestamp = f"[{msg.timestamp}] " if msg.timestamp else "" + line = f"{timestamp}{msg.role}: {msg.content}" + logger.info(f"Transcript: {line}") + runner = PipelineRunner(handle_sigint=handle_sigint) await runner.run(task) diff --git a/examples/foundational/19a-azure-realtime-beta.py b/examples/foundational/19a-azure-realtime-beta.py index 76180e397..54f0302e5 100644 --- a/examples/foundational/19a-azure-realtime-beta.py +++ b/examples/foundational/19a-azure-realtime-beta.py @@ -43,6 +43,10 @@ async def fetch_weather_from_api(params: FunctionCallParams): ) +async def fetch_restaurant_recommendation(params: FunctionCallParams): + await params.result_callback({"name": "The Golden Dragon"}) + + # Define weather function using standardized schema weather_function = FunctionSchema( name="get_current_weather", @@ -61,8 +65,20 @@ weather_function = FunctionSchema( required=["location", "format"], ) +restaurant_function = FunctionSchema( + name="get_restaurant_recommendation", + description="Get a restaurant recommendation", + properties={ + "location": { + "type": "string", + "description": "The city and state, e.g. San Francisco, CA", + }, + }, + required=["location"], +) + # Create tools schema -tools = ToolsSchema(standard_tools=[weather_function]) +tools = ToolsSchema(standard_tools=[weather_function, restaurant_function]) # We store functions so objects (e.g. SileroVADAnalyzer) don't get @@ -98,7 +114,7 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si # Or set to False to disable openai turn detection and use transport VAD # turn_detection=False, # tools=tools, - instructions="""Your knowledge cutoff is 2023-10. You are a helpful and friendly AI. + instructions="""You are a helpful and friendly AI. Act like a human, but remember that you aren't a human and that you can't do human things in the real world. Your voice and personality should be warm and engaging, with a lively and @@ -111,6 +127,10 @@ even if you're asked about them. You are participating in a voice conversation. Keep your responses concise, short, and to the point unless specifically asked to elaborate on a topic. +You have access to the following tools: +- get_current_weather: Get the current weather for a given location. +- get_restaurant_recommendation: Get a restaurant recommendation for a given location. + Remember, your responses should be short. Just one or two sentences, usually.""", ) @@ -124,6 +144,7 @@ Remember, your responses should be short. Just one or two sentences, usually.""" # you can either register a single function for all function calls, or specific functions # llm.register_function(None, fetch_weather_from_api) llm.register_function("get_current_weather", fetch_weather_from_api) + llm.register_function("get_restaurant_recommendation", fetch_restaurant_recommendation) # Create a standard OpenAI LLM context object using the normal messages format. The # OpenAIRealtimeBetaLLMService will convert this internally to messages that the diff --git a/examples/foundational/22b-natural-conversation-proposal.py b/examples/foundational/22b-natural-conversation-proposal.py index 58259b266..b76bfd274 100644 --- a/examples/foundational/22b-natural-conversation-proposal.py +++ b/examples/foundational/22b-natural-conversation-proposal.py @@ -198,7 +198,6 @@ class OutputGate(FrameProcessor): async def fetch_weather_from_api(params: FunctionCallParams): - await params.llm.push_frame(TTSSpeakFrame("Let me check on that.")) await params.result_callback({"conditions": "nice", "temperature": "75"}) @@ -245,6 +244,10 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si # sent to the same callback with an additional function_name parameter. llm.register_function("get_current_weather", fetch_weather_from_api) + @llm.event_handler("on_function_calls_started") + async def on_function_calls_started(service, function_calls): + await tts.queue_frame(TTSSpeakFrame("Let me check on that.")) + tools = [ ChatCompletionToolParam( type="function", @@ -309,9 +312,6 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si # to start the conversation. bot_output_gate = OutputGate(notifier=notifier, start_open=True) - async def block_user_stopped_speaking(frame): - return not isinstance(frame, UserStoppedSpeakingFrame) - async def pass_only_llm_trigger_frames(frame): return ( isinstance(frame, OpenAILLMContextFrame) @@ -328,11 +328,6 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si stt, context_aggregator.user(), ParallelPipeline( - [ - # Pass everything except UserStoppedSpeaking to the elements after - # this ParallelPipeline - FunctionFilter(filter=block_user_stopped_speaking), - ], [ # Ignore everything except an OpenAILLMContextFrame. Pass a specially constructed # LLMMessagesFrame to the statement classifier LLM. The only frame this diff --git a/examples/foundational/22c-natural-conversation-mixed-llms.py b/examples/foundational/22c-natural-conversation-mixed-llms.py index 7d6a1ce18..0bcbf75bd 100644 --- a/examples/foundational/22c-natural-conversation-mixed-llms.py +++ b/examples/foundational/22c-natural-conversation-mixed-llms.py @@ -402,7 +402,6 @@ class OutputGate(FrameProcessor): async def fetch_weather_from_api(params: FunctionCallParams): - await params.llm.push_frame(TTSSpeakFrame("Let me check on that.")) await params.result_callback({"conditions": "nice", "temperature": "75"}) @@ -449,6 +448,10 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si # sent to the same callback with an additional function_name parameter. llm.register_function("get_current_weather", fetch_weather_from_api) + @llm.event_handler("on_function_calls_started") + async def on_function_calls_started(service, function_calls): + await tts.queue_frame(TTSSpeakFrame("Let me check on that.")) + tools = [ ChatCompletionToolParam( type="function", diff --git a/examples/foundational/26b-gemini-multimodal-live-function-calling.py b/examples/foundational/26b-gemini-multimodal-live-function-calling.py index 84c7d5045..15db2faa0 100644 --- a/examples/foundational/26b-gemini-multimodal-live-function-calling.py +++ b/examples/foundational/26b-gemini-multimodal-live-function-calling.py @@ -40,11 +40,17 @@ async def fetch_weather_from_api(params: FunctionCallParams): ) +async def fetch_restaurant_recommendation(params: FunctionCallParams): + await params.result_callback({"name": "The Golden Dragon"}) + + system_instruction = """ You are a helpful assistant who can answer questions and use tools. -You have a tool called "get_current_weather" that can be used to get the current weather. If the user asks -for the weather, call this function. +You have three tools available to you: +1. get_current_weather: Use this tool to get the current weather in a specific location. +2. get_restaurant_recommendation: Use this tool to get a restaurant recommendation in a specific location. +3. google_search: Use this tool to search the web for information. """ @@ -101,9 +107,21 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si }, required=["location", "format"], ) + restaurant_function = FunctionSchema( + name="get_restaurant_recommendation", + description="Get a restaurant recommendation", + properties={ + "location": { + "type": "string", + "description": "The city and state, e.g. San Francisco, CA", + }, + }, + required=["location"], + ) search_tool = {"google_search": {}} tools = ToolsSchema( - standard_tools=[weather_function], custom_tools={AdapterType.GEMINI: [search_tool]} + standard_tools=[weather_function, restaurant_function], + custom_tools={AdapterType.GEMINI: [search_tool]}, ) llm = GeminiMultimodalLiveLLMService( @@ -113,6 +131,7 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si ) llm.register_function("get_current_weather", fetch_weather_from_api) + llm.register_function("get_restaurant_recommendation", fetch_restaurant_recommendation) context = OpenAILLMContext( [{"role": "user", "content": "Say hello."}], diff --git a/examples/foundational/41a-text-only-webrtc.py b/examples/foundational/41a-text-only-webrtc.py index 0125a41c4..bfd7a6051 100644 --- a/examples/foundational/41a-text-only-webrtc.py +++ b/examples/foundational/41a-text-only-webrtc.py @@ -82,78 +82,77 @@ transport_params = { async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_sigint: bool): logger.info(f"Starting bot") - # Create an HTTP session for API calls - async with aiohttp.ClientSession() as session: - llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY")) + llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY")) - messages = [ - { - "role": "system", - "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Respond to what the user said in a creative and helpful way.", - }, + messages = [ + { + "role": "system", + "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Respond to what the user said in a creative and helpful way.", + }, + ] + + context = OpenAILLMContext(messages) + context_aggregator = llm.create_context_aggregator(context) + + action_llm_append_to_messages = create_action_llm_append_to_messages(context_aggregator) + rtvi = RTVIProcessor(config=RTVIConfig(config=[])) + rtvi.register_action(action_llm_append_to_messages) + + pipeline = Pipeline( + [ + transport.input(), + rtvi, + context_aggregator.user(), + llm, + transport.output(), + context_aggregator.assistant(), ] + ) - context = OpenAILLMContext(messages) - context_aggregator = llm.create_context_aggregator(context) + task = PipelineTask( + pipeline, + params=PipelineParams( + allow_interruptions=True, + enable_metrics=True, + ), + observers=[RTVIObserver(rtvi)], + ) - action_llm_append_to_messages = create_action_llm_append_to_messages(context_aggregator) - rtvi = RTVIProcessor(config=RTVIConfig(config=[])) - rtvi.register_action(action_llm_append_to_messages) + @rtvi.event_handler("on_client_ready") + async def on_client_ready(rtvi): + logger.info("Pipecat client ready.") + await rtvi.set_bot_ready() - pipeline = Pipeline( - [ - transport.input(), - rtvi, - context_aggregator.user(), - llm, - transport.output(), - context_aggregator.assistant(), - ] - ) + # This block is frontend UI specific + # These messages are intended for small webrtc UI to only handle text + # https://github.com/pipecat-ai/small-webrtc-prebuilt + messages = { + "show_text_container": True, + "show_video_container": False, + "show_debug_container": False, + } - task = PipelineTask( - pipeline, - params=PipelineParams( - allow_interruptions=True, - enable_metrics=True, - ), - observers=[RTVIObserver(rtvi)], - ) + rtvi_frame = RTVIServerMessageFrame(data=messages) + await task.queue_frames([rtvi_frame]) - @rtvi.event_handler("on_client_ready") - async def on_client_ready(rtvi): - logger.info("Pipecat client ready.") - await rtvi.set_bot_ready() + @transport.event_handler("on_client_connected") + async def on_client_connected(transport, client): + logger.info(f"Client connected: {client}") + # Kick off the conversation. + await task.queue_frames([context_aggregator.user().get_context_frame()]) - # This block is frontend UI specific - # These messages are intended for small webrtc UI to only handle text - # https://github.com/pipecat-ai/small-webrtc-prebuilt - messages = { - "show_text_container": True, - "show_video_container": False, - "show_debug_container": False, - } - rtvi_frame = RTVIServerMessageFrame(data=messages) - await task.queue_frames([rtvi_frame]) + @transport.event_handler("on_client_disconnected") + async def on_client_disconnected(transport, client): + logger.info(f"Client disconnected") - @transport.event_handler("on_client_connected") - async def on_client_connected(transport, client): - logger.info(f"Client connected: {client}") - # Kick off the conversation. - await task.queue_frames([context_aggregator.user().get_context_frame()]) + @transport.event_handler("on_client_closed") + async def on_client_closed(transport, client): + logger.info(f"Client closed connection") + await task.cancel() - @transport.event_handler("on_client_disconnected") - async def on_client_disconnected(transport, client): - logger.info(f"Client disconnected") + runner = PipelineRunner(handle_sigint=False) - @transport.event_handler("on_client_closed") - async def on_client_closed(transport, client): - logger.info(f"Client closed connection") - await task.cancel() - - runner = PipelineRunner(handle_sigint=False) - - await runner.run(task) + await runner.run(task) if __name__ == "__main__": diff --git a/examples/foundational/41b-text-and-audio-webrtc.py b/examples/foundational/41b-text-and-audio-webrtc.py index 57adbfdfc..5ac250286 100644 --- a/examples/foundational/41b-text-and-audio-webrtc.py +++ b/examples/foundational/41b-text-and-audio-webrtc.py @@ -92,85 +92,83 @@ transport_params = { async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_sigint: bool): logger.info(f"Starting bot") - # Create an HTTP session for API calls - async with aiohttp.ClientSession() as session: - stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY")) + stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY")) - llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY")) + llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY")) - tts = CartesiaTTSService( - api_key=os.getenv("CARTESIA_API_KEY"), voice_id="71a7ad14-091c-4e8e-a314-022ece01c121" - ) + tts = CartesiaTTSService( + api_key=os.getenv("CARTESIA_API_KEY"), voice_id="71a7ad14-091c-4e8e-a314-022ece01c121" + ) - messages = [ - { - "role": "system", - "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Respond to what the user says in a creative and helpful way. Explain to the User they can speak or type text to communicate with you.", - }, + messages = [ + { + "role": "system", + "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Respond to what the user says in a creative and helpful way. Explain to the User they can speak or type text to communicate with you.", + }, + ] + + context = OpenAILLMContext(messages) + context_aggregator = llm.create_context_aggregator(context) + + action_llm_append_to_messages = create_action_llm_append_to_messages(context_aggregator) + rtvi = RTVIProcessor(config=RTVIConfig(config=[])) + rtvi.register_action(action_llm_append_to_messages) + + pipeline = Pipeline( + [ + transport.input(), + rtvi, + stt, + context_aggregator.user(), + llm, + tts, + transport.output(), + context_aggregator.assistant(), ] + ) - context = OpenAILLMContext(messages) - context_aggregator = llm.create_context_aggregator(context) + task = PipelineTask( + pipeline, + params=PipelineParams( + allow_interruptions=True, + enable_metrics=True, + ), + observers=[RTVIObserver(rtvi)], + ) - action_llm_append_to_messages = create_action_llm_append_to_messages(context_aggregator) - rtvi = RTVIProcessor(config=RTVIConfig(config=[])) - rtvi.register_action(action_llm_append_to_messages) + @rtvi.event_handler("on_client_ready") + async def on_client_ready(rtvi): + logger.info("Pipecat client ready.") + await rtvi.set_bot_ready() - pipeline = Pipeline( - [ - transport.input(), - rtvi, - stt, - context_aggregator.user(), - llm, - tts, - transport.output(), - context_aggregator.assistant(), - ] - ) + # This block is frontend UI specific + # These messages are intended for small webrtc UI to only handle text + # https://github.com/pipecat-ai/small-webrtc-prebuilt + messages = { + "show_text_container": True, + "show_debug_container": False, + } + rtvi_frame = RTVIServerMessageFrame(data=messages) + await task.queue_frames([rtvi_frame]) - task = PipelineTask( - pipeline, - params=PipelineParams( - allow_interruptions=True, - enable_metrics=True, - ), - observers=[RTVIObserver(rtvi)], - ) + @transport.event_handler("on_client_connected") + async def on_client_connected(transport, client): + logger.info(f"Client connected: {client}") + # Kick off the conversation. + await task.queue_frames([context_aggregator.user().get_context_frame()]) - @rtvi.event_handler("on_client_ready") - async def on_client_ready(rtvi): - logger.info("Pipecat client ready.") - await rtvi.set_bot_ready() + @transport.event_handler("on_client_disconnected") + async def on_client_disconnected(transport, client): + logger.info(f"Client disconnected") - # This block is frontend UI specific - # These messages are intended for small webrtc UI to only handle text - # https://github.com/pipecat-ai/small-webrtc-prebuilt - messages = { - "show_text_container": True, - "show_debug_container": False, - } - rtvi_frame = RTVIServerMessageFrame(data=messages) - await task.queue_frames([rtvi_frame]) + @transport.event_handler("on_client_closed") + async def on_client_closed(transport, client): + logger.info(f"Client closed connection") + await task.cancel() - @transport.event_handler("on_client_connected") - async def on_client_connected(transport, client): - logger.info(f"Client connected: {client}") - # Kick off the conversation. - await task.queue_frames([context_aggregator.user().get_context_frame()]) + runner = PipelineRunner(handle_sigint=False) - @transport.event_handler("on_client_disconnected") - async def on_client_disconnected(transport, client): - logger.info(f"Client disconnected") - - @transport.event_handler("on_client_closed") - async def on_client_closed(transport, client): - logger.info(f"Client closed connection") - await task.cancel() - - runner = PipelineRunner(handle_sigint=False) - - await runner.run(task) + await runner.run(task) if __name__ == "__main__": diff --git a/examples/foundational/42-interruption-config.py b/examples/foundational/42-interruption-config.py new file mode 100644 index 000000000..3cb64c204 --- /dev/null +++ b/examples/foundational/42-interruption-config.py @@ -0,0 +1,125 @@ +# +# Copyright (c) 2024–2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +import argparse +import os + +from dotenv import load_dotenv +from loguru import logger + +from pipecat.audio.vad.silero import SileroVADAnalyzer +from pipecat.frames.frames import MinWordsInterruptionStrategy +from pipecat.pipeline.pipeline import Pipeline +from pipecat.pipeline.runner import PipelineRunner +from pipecat.pipeline.task import PipelineParams, PipelineTask +from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext +from pipecat.processors.transcript_processor import TranscriptProcessor +from pipecat.services.cartesia.tts import CartesiaTTSService +from pipecat.services.deepgram.stt import DeepgramSTTService +from pipecat.services.openai.llm import OpenAILLMService +from pipecat.transports.base_transport import BaseTransport, TransportParams +from pipecat.transports.network.fastapi_websocket import FastAPIWebsocketParams +from pipecat.transports.services.daily import DailyParams + +load_dotenv(override=True) + +# We store functions so objects (e.g. SileroVADAnalyzer) don't get +# instantiated. The function will be called when the desired transport gets +# selected. +transport_params = { + "daily": lambda: DailyParams( + audio_in_enabled=True, + audio_out_enabled=True, + vad_analyzer=SileroVADAnalyzer(), + ), + "twilio": lambda: FastAPIWebsocketParams( + audio_in_enabled=True, + audio_out_enabled=True, + vad_analyzer=SileroVADAnalyzer(), + ), + "webrtc": lambda: TransportParams( + audio_in_enabled=True, + audio_out_enabled=True, + vad_analyzer=SileroVADAnalyzer(), + ), +} + + +async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_sigint: bool): + logger.info(f"Starting bot") + + stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY")) + + tts = CartesiaTTSService( + api_key=os.getenv("CARTESIA_API_KEY"), + voice_id="71a7ad14-091c-4e8e-a314-022ece01c121", # British Reading Lady + ) + + llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY")) + + transcript = TranscriptProcessor() + + messages = [ + { + "role": "system", + "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.", + }, + ] + + context = OpenAILLMContext(messages) + context_aggregator = llm.create_context_aggregator(context) + + pipeline = Pipeline( + [ + transport.input(), # Transport user input + stt, + transcript.user(), # User transcripts + context_aggregator.user(), # User responses + llm, # LLM + tts, # TTS + transport.output(), # Transport bot output + context_aggregator.assistant(), # Assistant spoken responses + ] + ) + + task = PipelineTask( + pipeline, + params=PipelineParams( + allow_interruptions=True, + enable_metrics=True, + enable_usage_metrics=True, + report_only_initial_ttfb=True, + interruption_strategies=[MinWordsInterruptionStrategy(min_words=3)], + ), + ) + + @transport.event_handler("on_client_connected") + async def on_client_connected(transport, client): + logger.info(f"Client connected") + # Kick off the conversation. + messages.append({"role": "system", "content": "Please introduce yourself to the user."}) + await task.queue_frames([context_aggregator.user().get_context_frame()]) + + @transport.event_handler("on_client_disconnected") + async def on_client_disconnected(transport, client): + logger.info(f"Client disconnected") + await task.cancel() + + # Register event handler for transcript updates + @transcript.event_handler("on_transcript_update") + async def on_transcript_update(processor, frame): + for message in frame.messages: + logger.info(f"Transcription [{message.role}]: {message.content}") + + runner = PipelineRunner(handle_sigint=handle_sigint) + + await runner.run(task) + + +if __name__ == "__main__": + from pipecat.examples.run import main + + main(run_example, transport_params=transport_params) diff --git a/examples/foundational/requirements.txt b/examples/foundational/requirements.txt index 18e2f367c..8e4255cf8 100644 --- a/examples/foundational/requirements.txt +++ b/examples/foundational/requirements.txt @@ -1,5 +1,5 @@ fastapi uvicorn python-dotenv -pipecat-ai[webrtc,deepgram,cartesia] +pipecat-ai[webrtc,daily,deepgram,cartesia] pipecat-ai-small-webrtc-prebuilt \ No newline at end of file diff --git a/examples/open-telemetry/jaeger/bot.py b/examples/open-telemetry/jaeger/bot.py index 74e587410..0f3084fd3 100644 --- a/examples/open-telemetry/jaeger/bot.py +++ b/examples/open-telemetry/jaeger/bot.py @@ -49,7 +49,6 @@ if IS_TRACING_ENABLED: async def fetch_weather_from_api(params: FunctionCallParams): - await params.llm.push_frame(TTSSpeakFrame("Let me check on that.")) await params.result_callback({"conditions": "nice", "temperature": "75"}) @@ -93,6 +92,10 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si # sent to the same callback with an additional function_name parameter. llm.register_function("get_current_weather", fetch_weather_from_api) + @llm.event_handler("on_function_calls_started") + async def on_function_calls_started(service, function_calls): + await tts.queue_frame(TTSSpeakFrame("Let me check on that.")) + weather_function = FunctionSchema( name="get_current_weather", description="Get the current weather", diff --git a/examples/open-telemetry/langfuse/bot.py b/examples/open-telemetry/langfuse/bot.py index 22ff399a0..9c4f34695 100644 --- a/examples/open-telemetry/langfuse/bot.py +++ b/examples/open-telemetry/langfuse/bot.py @@ -46,7 +46,6 @@ if IS_TRACING_ENABLED: async def fetch_weather_from_api(params: FunctionCallParams): - await params.llm.push_frame(TTSSpeakFrame("Let me check on that.")) await params.result_callback({"conditions": "nice", "temperature": "75"}) @@ -90,6 +89,10 @@ async def run_example(transport: BaseTransport, _: argparse.Namespace, handle_si # sent to the same callback with an additional function_name parameter. llm.register_function("get_current_weather", fetch_weather_from_api) + @llm.event_handler("on_function_calls_started") + async def on_function_calls_started(service, function_calls): + await tts.queue_frame(TTSSpeakFrame("Let me check on that.")) + weather_function = FunctionSchema( name="get_current_weather", description="Get the current weather", diff --git a/examples/p2p-webrtc/daily-interop-bridge/server.py b/examples/p2p-webrtc/daily-interop-bridge/server.py index 0025b3490..c8dae7888 100644 --- a/examples/p2p-webrtc/daily-interop-bridge/server.py +++ b/examples/p2p-webrtc/daily-interop-bridge/server.py @@ -74,7 +74,7 @@ async def offer(request: dict, background_tasks: BackgroundTasks): @asynccontextmanager async def lifespan(app: FastAPI): yield # Run app - coros = [pc.close() for pc in pcs_map.values()] + coros = [pc.disconnect() for pc in pcs_map.values()] await asyncio.gather(*coros) pcs_map.clear() diff --git a/examples/p2p-webrtc/video-transform/client/android/gradle/libs.versions.toml b/examples/p2p-webrtc/video-transform/client/android/gradle/libs.versions.toml index 3f3d6a981..be90d31de 100644 --- a/examples/p2p-webrtc/video-transform/client/android/gradle/libs.versions.toml +++ b/examples/p2p-webrtc/video-transform/client/android/gradle/libs.versions.toml @@ -2,7 +2,7 @@ accompanistPermissions = "0.34.0" agp = "8.7.3" constraintlayoutCompose = "1.1.0" -pipecatClient = "0.3.6" +pipecatClient = "0.3.7" kotlin = "2.0.20" coreKtx = "1.15.0" lifecycleRuntimeKtx = "2.8.7" diff --git a/examples/p2p-webrtc/video-transform/server/bot.py b/examples/p2p-webrtc/video-transform/server/bot.py index 44684d4ea..8b11ffe21 100644 --- a/examples/p2p-webrtc/video-transform/server/bot.py +++ b/examples/p2p-webrtc/video-transform/server/bot.py @@ -122,8 +122,8 @@ async def run_bot(webrtc_connection): pipeline, params=PipelineParams( allow_interruptions=True, - observers=[RTVIObserver(rtvi)], ), + observers=[RTVIObserver(rtvi)], ) @rtvi.event_handler("on_client_ready") diff --git a/examples/p2p-webrtc/video-transform/server/server.py b/examples/p2p-webrtc/video-transform/server/server.py index 0025b3490..c8dae7888 100644 --- a/examples/p2p-webrtc/video-transform/server/server.py +++ b/examples/p2p-webrtc/video-transform/server/server.py @@ -74,7 +74,7 @@ async def offer(request: dict, background_tasks: BackgroundTasks): @asynccontextmanager async def lifespan(app: FastAPI): yield # Run app - coros = [pc.close() for pc in pcs_map.values()] + coros = [pc.disconnect() for pc in pcs_map.values()] await asyncio.gather(*coros) pcs_map.clear() diff --git a/examples/p2p-webrtc/voice-agent/server.py b/examples/p2p-webrtc/voice-agent/server.py index b2ee0b4cd..f442333bf 100644 --- a/examples/p2p-webrtc/voice-agent/server.py +++ b/examples/p2p-webrtc/voice-agent/server.py @@ -69,7 +69,7 @@ async def serve_index(): @asynccontextmanager async def lifespan(app: FastAPI): yield # Run app - coros = [pc.close() for pc in pcs_map.values()] + coros = [pc.disconnect() for pc in pcs_map.values()] await asyncio.gather(*coros) pcs_map.clear() diff --git a/examples/phone-chatbot/README.md b/examples/phone-chatbot/README.md index 83e2c57fd..3811c5bb5 100644 --- a/examples/phone-chatbot/README.md +++ b/examples/phone-chatbot/README.md @@ -8,505 +8,31 @@ This repository contains examples for building intelligent phone chatbots using AI for various use cases including: -- **Simple dial-in**: Basic incoming call handling -- **Simple dial-out**: Basic outgoing call handling -- **Voicemail detection**: Bot calls a number, detects if it reaches voicemail or a human, and responds appropriately -- **Call transfer**: Bot handles initial customer interaction and transfers to a human operator when needed +- **daily-pstn-dial-in**: Basic incoming call handling +- **daily-pstn-dial-out**: Basic outgoing call handling +- **daily-twilio-sip-dial-in**: Basic incoming call handling using Daily SIP + Twilio +- **daily-twilio-sip-dial-out**: Basic outgoing call handling using Daily SIP + Twilio +- **daily-pstn-simple-voicemail-detection**: Voicemail detection Bot. Bot calls a number, detects if it reaches voicemail or a human, and responds appropriately +- **daily-pstn-advanced-voicemail-detection**: A more advanced example of the voicemail detection bot. Utilises multiple pipelines. The first pipeline uses a much simpler, faster and cheaper LLM to detect the voicemail machine. Then switches to a more powerful LLM if it needs to have a conversation with a user. You should use this one if you want to use different LLMs for different tasks, it also shows how to do audio input for one LLM (multimodal LLM) and then STT for the other one. Switching out methods of sending data to the LLM +- **daily-pstn-simple-call-transfer**: Bot handles initial customer interaction and transfers to a human operator when needed ## Architecture Overview These examples use the following components: - 🔁 **Transport**: Daily WebRTC -- 💬 **Speech-to-Text**: Deepgram via Daily transport +- 💬 **Speech-to-Text**: Deepgram via Daily transport, or via separate Deepgram service - 🤖 **LLMs**: Each example uses a specific LLM (OpenAI GPT-4o or Google Gemini) +- 📞 **SIP/PSTN**: Examples either use Daily PSTN or SIP with a SIP provider such as Twilio - 🔉 **Text-to-Speech**: Cartesia -## Getting Started - -### Prerequisites - -1. Create and activate a virtual environment: - - ```shell - python3 -m venv venv - source venv/bin/activate # On Windows: venv\Scripts\activate - ``` - -2. Install requirements: - - ```shell - pip install -r requirements.txt - ``` - -3. Set up your environment variables: - - ```shell - cp env.example .env - ``` - - Edit the `.env` file to include your API keys. - -4. Install [ngrok](https://ngrok.com/) to make your local server accessible to external services. - ### Phone Number Provider: Daily vs Twilio If you're starting from scratch, we recommend using Daily to provision phone numbers alongside Daily as a transport for simplicity (this provides automatic call forwarding). If you already have Twilio numbers and workflows, you can connect them to your Pipecat bots with some additional configuration (`on_dialin_ready` and using the Twilio client to trigger forwarding). -Most examples in this repository show how to use Daily for dial-in/dial-out operations. - -## Running the Examples - -### 1. Start the Bot Runner Service - -The bot runner handles incoming requests and manages bot processes: - -```shell -python bot_runner.py --host localhost -``` - -### 2. Create a Public Endpoint with ngrok - -Start ngrok to create a public URL for your local server: - -```shell -ngrok http --domain yourdomain.ngrok.app 7860 -``` - -## Example 1: Simple Dial-in - -This example demonstrates basic handling of incoming calls without additional features like call transfer. - -### Testing in Daily Prebuilt (No Actual Phone Calls) - -```shell -curl -X POST "http://localhost:7860/start" \ - -H "Content-Type: application/json" \ - -d '{ - "config": { - "simple_dialin": { - "testInPrebuilt": true - } - } - }' -``` - -This returns a Daily room URL where you can test the bot's basic conversation capabilities. - -## Example 2: Simple Dial-out - -This example demonstrates basic handling of outgoing calls without additional features like voicemail detection. - -### Testing in Daily Prebuilt (No Actual Phone Calls) - -```shell -curl -X POST "http://localhost:7860/start" \ - -H "Content-Type: application/json" \ - -d '{ - "config": { - "simple_dialout": { - "testInPrebuilt": true - } - } - }' -``` - -This returns a Daily room URL where you can test the bot's basic conversation capabilities. - -### Making Actual Phone Calls - -```shell -curl -X POST "http://localhost:7860/start" \ - -H "Content-Type: application/json" \ - -d '{ - "config": { - "dialout_settings": [{ - "phoneNumber": "+12345678910" - }], - "simple_dialout": { - "testInPrebuilt": false - } - } - }' -``` - -## Example 3: Voicemail Detection - -This example demonstrates a bot that can dial out to a phone number, detect whether it reached a human or voicemail system, and respond appropriately. - -### How It Works - -1. Bot dials a phone number -2. Bot listens to determine if it's connected to a person or voicemail -3. If it detects voicemail, it leaves a predefined message and hangs up -4. If it detects a human, it engages in conversation - -### Testing in Daily Prebuilt (No Actual Phone Calls) - -To test without making actual phone calls: - -```shell -curl -X POST "http://localhost:7860/start" \ - -H "Content-Type: application/json" \ - -d '{ - "config": { - "voicemail_detection": { - "testInPrebuilt": true - } - } - }' -``` - -This will return a Daily room URL you can use to test the bot in the browser. - -### Making Actual Phone Calls - -To have the bot dial out to a real phone number: - -```shell -curl -X POST "http://localhost:7860/start" \ - -H "Content-Type: application/json" \ - -d '{ - "config": { - "dialout_settings": [{ - "phoneNumber": "+12345678910" - }], - "voicemail_detection": { - "testInPrebuilt": false - } - } - }' -``` - -> **Note:** To enable dial-out capabilities, you must first: -> -> 1. Contact [help@daily.co](mailto:help@daily.co) to enable dial-out for your domain -> 2. Purchase a phone number to dial out from -> 3. Ensure rooms have dial-out enabled (the bot runner handles this) -> 4. Use an owner token for the bot (also handled by the bot runner) - -## Example 4: Call Transfer - -This example demonstrates a bot that handles initial customer interaction and can transfer the call to a human operator when requested. - -### How It Works - -1. Customer calls in and speaks with the bot -2. When the customer asks for a supervisor/manager, the bot initiates a transfer -3. The bot dials out to an appropriate operator -4. When the operator joins, the bot summarizes the conversation -5. The bot remains silent while operator and customer talk -6. When the operator leaves, the bot resumes handling the call - -### Testing in Daily Prebuilt (No Actual Phone Calls) - -```shell -curl -X POST "http://localhost:7860/start" \ - -H "Content-Type: application/json" \ - -d '{ - "config": { - "call_transfer": { - "mode": "dialout", - "speakSummary": true, - "storeSummary": false, - "operatorNumber": "+12345678910", - "testInPrebuilt": true - } - } - }' -``` - -This returns a Daily room URL. In the room, the expected flow is: - -1. Join the room and speak with the bot -2. Ask to speak with a manager/supervisor -3. The bot will add the "operator" to the call -4. The bot will summarize the conversation and then go silent -5. To simulate the operator, you can mute yourself in Daily Prebuilt and speak as if you're the operator -6. When finished, have the "operator" leave the call -7. The bot will resume speaking and can recall details from the conversation -8. End the call by closing Daily Prebuilt or telling the bot you're done - -### Using with Real Phone Calls - -For incoming calls from customers, Daily will send a webhook to your `/start` endpoint. This webhook contains: - -```json -{ - "From": "+CALLERS_PHONE", - "To": "$PURCHASED_PHONE", - "callId": "callid-read-only-string", - "callDomain": "callDomain-read-only-string" -} -``` - -The system will: - -1. Identify the customer based on their phone number -2. Determine the appropriate operator to contact -3. Customize the bot's behavior based on transfer settings - -#### Operator Assignment - -The `call_connection_manager.py` file contains mappings for: - -1. `CUSTOMER_MAP`: Links phone numbers to customer names -2. `OPERATOR_CONTACT_MAP`: Contains operator contact information -3. `CUSTOMER_TO_OPERATOR_MAP`: Defines which operators should handle which customers - -You can customize these mappings or integrate with your existing customer database. - -## Configuration Options - -### Request Body Structure - -When making requests to the `/start` endpoint, the config object can include: - -```json -{ - "config": { - "prompts": [ - { - "name": "call_transfer_initial_prompt", - "text": "Your custom prompt here" - }, - { - "name": "call_transfer_prompt", - "text": "Your custom prompt here" - }, - { - "name": "call_transfer_finished_prompt", - "text": "Your custom prompt here" - }, - { - "name": "voicemail_detection_prompt", - "text": "Your custom prompt here" - }, - { - "name": "voicemail_prompt", - "text": "Your custom prompt here" - }, - { - "name": "human_conversation_prompt", - "text": "Your custom prompt here" - } - ], - "dialin_settings": { - "From": "+CALLERS_PHONE", - "To": "$PURCHASED_PHONE", - "callId": "callid-read-only-string", - "callDomain": "callDomain-read-only-string" - }, - "dialout_settings": [ - { - "phoneNumber": "+12345678910", - "callerId": "caller-id-uuid", - "sipUri": "sip:maria@example.com" - } - ], - "call_transfer": { - "mode": "dialout", - "speakSummary": true, - "storeSummary": false, - "operatorNumber": "+12345678910", - "testInPrebuilt": false - }, - "voicemail_detection": { - "testInPrebuilt": true - }, - "simple_dialin": { - "testInPrebuilt": true - }, - "simple_dialout": { - "testInPrebuilt": true - } - } -} -``` - -### Configuration Parameters - -- `prompts`: An array of objects containing prompts that you want the examples to use. -- `dialin_settings`: Information about incoming calls (typically from webhook) -- `dialout_settings`: For outbound calls: - - `phoneNumber`: Number to dial - - `callerId`: UUID of the number to display (optional) - - `sipUri`: SIP URI to connect to (alternative to phoneNumber) -- `call_transfer`: For call transfer example: - - `mode`: Currently only `"dialout"` is supported - - `speakSummary`: Whether the bot should summarize the conversation for the operator - - `storeSummary`: For future implementation - - `operatorNumber`: Operator phone number - - `testInPrebuilt`: Test without actual phone calls -- `voicemail_detection`: For voicemail detection example: - - `testInPrebuilt`: Test without actual phone calls -- `simple_dialin`: For simple dialin example: - - `testInPrebuilt`: Test without actual phone calls -- `simple_dialout`: For simple dialout example: - - `testInPrebuilt`: Test without actual phone calls - -## Feature Compatibility - -The following table shows which feature combinations are supported when making requests to the `/start` endpoint. The table is organized by use case to help you create the correct configuration. - -| Use Case | `call_transfer` | `voicemail_detection` | `simple_dialin` | `simple_dialout` | `dialin_settings` | `dialout_settings` | `operatorNumber` | `testInPrebuilt` | Status | -| --------------------------------------------------------------- | --------------- | --------------------- | --------------- | ---------------- | ----------------- | ------------------ | ---------------- | ---------------- | ---------------- | -| **Basic incoming call handling (simple_dialin)** | ✗ | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✅ Supported | -| **Test mode: Simple dialin in Daily Prebuilt** | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✅ Supported | -| **Basic outgoing call handling (simple_dialout)** | ✗ | ✗ | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ | ✅ Supported | -| **Test mode: Simple dialout in Daily Prebuilt** | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✓ | ✅ Supported | -| **Standard call transfer (incoming call)** | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✓/✗ | ✗ | ✅ Supported | -| **Standard voicemail detection (outgoing call)** | ✗ | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✅ Supported | -| **Test mode: Call transfer in Daily Prebuilt** | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✅ Supported | -| **Test mode: Voicemail detection in Daily Prebuilt** | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✅ Supported | -| Call transfer requires operatorNumber | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✓/✗ | ❌ Not Supported | -| Voicemail detection requires dialout_settings or testInPrebuilt | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✓/✗ | ❌ Not Supported | -| Cannot combine different bot types | ✓ | ✓ | ✗ | ✗ | ✓ | ✓ | ✓ | ✓/✗ | ❌ Not Supported | -| Call_transfer needs dialin_settings in non-test mode | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ❌ Not Supported | -| Voicemail_detection needs dialout_settings in non-test mode | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ❌ Not Supported | -| Insufficient configuration | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓/✗ | ❌ Not Supported | - -### Legend: - -- ✓: Required -- ✗: Not allowed -- ✓/✗: Optional -- ✅: Supported -- ❌: Not Supported - -### Notes: - -- `dialin_settings` is typically populated automatically from webhook data for incoming calls -- `dialout_settings` must be specified manually for outgoing calls -- `operatorNumber` is specified within the `call_transfer` object (`"call_transfer": {"operatorNumber": "+1234567890", ...}`) -- `testInPrebuilt` is specified within the bot type object (e.g., `"call_transfer": {"testInPrebuilt": true, ...}`) -- For call transfers, `operatorNumber` must be provided to specify which operator to dial. If it is not provided, we will base it off of the operator map in call_connection_manager.py -- In test mode (`testInPrebuilt: true`), some requirements are relaxed to allow testing in Daily Prebuilt -- Multiple customers to dial out to can be specified by providing an array of objects in `dialout_settings` -- Bot types are mutually exclusive - you cannot combine multiple bot types in a single configuration - -### Configuration Examples - -#### Standard call transfer (incoming call): - -```json -{ - "config": { - "dialin_settings": { - "from": "+12345678901", - "to": "+19876543210", - "call_id": "call-id-string", - "call_domain": "domain-string" - }, - "call_transfer": { - "mode": "dialout", - "speakSummary": true, - "operatorNumber": "+12345678910" - } - } -} -``` - -#### Test mode: Call transfer in Daily Prebuilt: - -```json -{ - "config": { - "call_transfer": { - "mode": "dialout", - "speakSummary": true, - "operatorNumber": "+12345678910", - "testInPrebuilt": true - } - } -} -``` - -#### Test mode: Voicemail detection in Daily Prebuilt: - -```json -{ - "config": { - "voicemail_detection": { - "testInPrebuilt": true - } - } -} -``` - -#### Standard voicemail detection: - -```json -{ - "config": { - "dialout_settings": [ - { - "phoneNumber": "+12345678910" - } - ], - "voicemail_detection": { - "testInPrebuilt": false - } - } -} -``` - -#### Simple dialin (incoming call): - -```json -{ - "config": { - "dialin_settings": { - "from": "+12345678901", - "to": "+19876543210", - "call_id": "call-id-string", - "call_domain": "domain-string" - }, - "simple_dialin": {} - } -} -``` - -#### Test mode: Simple dialin in Daily Prebuilt: - -```json -{ - "config": { - "simple_dialin": { - "testInPrebuilt": true - } - } -} -``` - -#### Simple dialout (outgoing call): - -```json -{ - "config": { - "dialout_settings": [ - { - "phoneNumber": "+12345678910" - } - ], - "simple_dialout": {} - } -} -``` - -#### Test mode: Simple dialout in Daily Prebuilt: - -```json -{ - "config": { - "simple_dialout": { - "testInPrebuilt": true - } - } -} -``` +The Twilio dial-out example shows you how to configure the SIP URI domain and TwiML bins. ## Deployment @@ -518,61 +44,24 @@ We also have a great, easy to use quickstart guide here: https://docs.pipecat.da Each example in this repository is implemented with a specific LLM provider: -- **Simple dial-in**: Uses OpenAI -- **Simple dial-out**: Uses OpenAI -- **Voicemail detection**: Uses Google Gemini -- **Call transfer**: Uses OpenAI +- **daily-pstn-dial-in**: Uses OpenAI +- **daily-pstn-dial-out**: Uses OpenAI +- **daily-twilio-sip-dial-in**: Uses OpenAI +- **daily-twilio-sip-dial-out**: Uses OpenAI +- **daily-pstn-simple-voicemail-detection**: Uses Google Gemini Flash 2.0 +- **daily-pstn-simple-voicemail-detection**: Uses Google Gemini Flash Lite 2.0 and Flash 2.0 +- **daily-pstn-simple-call-transfer**: Uses OpenAI If you want to implement one of these examples with a different LLM provider than what's provided: -- To implement **call_transfer** with **Gemini**, reference the `voicemail_detection.py` file for how to structure LLM context, function calling, and other Gemini-specific implementations. -- To implement **voicemail_detection** with **OpenAI**, reference the `call_transfer.py` file for OpenAI-specific implementation details. +- To implement **call_transfer** with **Gemini**, reference the `bot.py` file inside the voicemail detection example for how to structure LLM context, function calling, and other Gemini-specific implementations. +- To implement **voicemail_detection** with **OpenAI**, reference the `bot.py` file inside the call_transfer example for OpenAI-specific implementation details. The key differences between implementations involve how context is managed, function calling syntax, and message formatting. Looking at both implementations side-by-side provides a good template for adapting any example to your preferred LLM provider. ## Customizing Bot Prompts -All examples include default prompts that work well for standard use cases. However, you can customize how the bot behaves by providing your own prompts in the request body. - -### Available Prompt Types - -- `call_transfer_initial_prompt`: The initial prompt the bot uses when greeting a customer -- `call_transfer_prompt`: Instructions for the bot when summarizing the conversation for an operator -- `call_transfer_finished_prompt`: Instructions for when the operator leaves the call -- `voicemail_detection_prompt`: Instructions for detecting whether a call connected to voicemail -- `voicemail_prompt`: The message to leave when voicemail is detected -- `human_conversation_prompt`: Instructions for conversation when a human is detected - -### Customization Example - -```shell -curl -X POST "http://localhost:7860/start" \ - -H "Content-Type: application/json" \ - -d '{ - "config": { - "prompts": [ - { - "name": "voicemail_prompt", - "text": "Hello, this is ACME Corporation calling. Please call us back at 555-123-4567 regarding your recent order. Thank you!" - } - ], - "dialout_settings": [{ - "phoneNumber": "+12345678910" - }], - "voicemail_detection": { - "testInPrebuilt": false - } - } - }' -``` - -This example would use all default prompts except for the voicemail message, which would be replaced with your custom message. - -### Template Variables - -Some prompts support template variables that are automatically replaced: - -- `{customer_name}`: Will be replaced with the customer's name if available +All examples include default prompts that work well for standard use cases. ## Advanced Usage diff --git a/examples/phone-chatbot/bot_constants.py b/examples/phone-chatbot/bot_constants.py deleted file mode 100644 index 6b28de1a3..000000000 --- a/examples/phone-chatbot/bot_constants.py +++ /dev/null @@ -1,23 +0,0 @@ -# bot_constants.py -"""Constants used across the bot runner application.""" - -# Maximum session time -MAX_SESSION_TIME = 5 * 60 # 5 minutes - -# Required environment variables -REQUIRED_ENV_VARS = [ - "OPENAI_API_KEY", - "GOOGLE_API_KEY", - "DAILY_API_KEY", - "CARTESIA_API_KEY", - "DEEPGRAM_API_KEY", -] - -# Default example to use when handling dialin webhooks - determines which bot type to run -DEFAULT_DIALIN_EXAMPLE = "call_transfer" # Options: call_transfer, simple_dialin - -# Call transfer configuration constants -DEFAULT_CALLTRANSFER_MODE = "dialout" -DEFAULT_SPEAK_SUMMARY = True # Speak a summary of the call to the operator -DEFAULT_STORE_SUMMARY = False # Store summary of the call (for future implementation) -DEFAULT_TEST_IN_PREBUILT = False # Test in prebuilt mode (bypasses need to dial in/out) diff --git a/examples/phone-chatbot/bot_definitions.py b/examples/phone-chatbot/bot_definitions.py deleted file mode 100644 index 33d249e03..000000000 --- a/examples/phone-chatbot/bot_definitions.py +++ /dev/null @@ -1,55 +0,0 @@ -# bot_definitions.py -"""Definitions of different bot types for the bot registry.""" - -from bot_registry import BotRegistry, BotType -from bot_runner_helpers import ( - create_call_transfer_settings, - create_simple_dialin_settings, - create_simple_dialout_settings, -) - -# Create and configure the bot registry -bot_registry = BotRegistry() - -# Register bot types -bot_registry.register( - BotType( - name="call_transfer", - settings_creator=create_call_transfer_settings, - required_settings=["dialin_settings"], - incompatible_with=["simple_dialin", "simple_dialout", "voicemail_detection"], - auto_add_settings={"dialin_settings": {}}, - ) -) - -bot_registry.register( - BotType( - name="simple_dialin", - settings_creator=create_simple_dialin_settings, - required_settings=["dialin_settings"], - incompatible_with=["call_transfer", "simple_dialout", "voicemail_detection"], - auto_add_settings={"dialin_settings": {}}, - ) -) - -bot_registry.register( - BotType( - name="simple_dialout", - settings_creator=create_simple_dialout_settings, - required_settings=["dialout_settings"], - incompatible_with=["call_transfer", "simple_dialin", "voicemail_detection"], - auto_add_settings={"dialout_settings": [{}]}, - ) -) - -bot_registry.register( - BotType( - name="voicemail_detection", - settings_creator=lambda body: body.get( - "voicemail_detection", {} - ), # No creator function in original code - required_settings=["dialout_settings"], - incompatible_with=["call_transfer", "simple_dialin", "simple_dialout"], - auto_add_settings={"dialout_settings": [{}]}, - ) -) diff --git a/examples/phone-chatbot/bot_registry.py b/examples/phone-chatbot/bot_registry.py deleted file mode 100644 index a60fe614c..000000000 --- a/examples/phone-chatbot/bot_registry.py +++ /dev/null @@ -1,137 +0,0 @@ -# bot_registry.py -"""Bot registry pattern for managing different bot types.""" - -from typing import Any, Callable, Dict, List, Optional - -from bot_constants import DEFAULT_DIALIN_EXAMPLE -from bot_runner_helpers import ensure_dialout_settings_array -from fastapi import HTTPException - - -class BotType: - """Bot type configuration and handling.""" - - def __init__( - self, - name: str, - settings_creator: Callable[[Dict[str, Any]], Dict[str, Any]], - required_settings: list = None, - incompatible_with: list = None, - auto_add_settings: dict = None, - ): - """Initialize a bot type. - - Args: - name: Name of the bot type - settings_creator: Function to create/update settings for this bot type - required_settings: List of settings this bot type requires - incompatible_with: List of bot types this one cannot be used with - auto_add_settings: Settings to add if this bot is being run in test mode - """ - self.name = name - self.settings_creator = settings_creator - self.required_settings = required_settings or [] - self.incompatible_with = incompatible_with or [] - self.auto_add_settings = auto_add_settings or {} - - def has_test_mode(self, body: Dict[str, Any]) -> bool: - """Check if this bot type is configured for test mode.""" - return self.name in body and body[self.name].get("testInPrebuilt", False) - - def create_settings(self, body: Dict[str, Any]) -> Dict[str, Any]: - """Create or update settings for this bot type.""" - body[self.name] = self.settings_creator(body) - return body - - def prepare_for_test(self, body: Dict[str, Any]) -> Dict[str, Any]: - """Add required settings for test mode if they don't exist.""" - for setting, default_value in self.auto_add_settings.items(): - if setting not in body: - body[setting] = default_value - return body - - -class BotRegistry: - """Registry for managing different bot types.""" - - def __init__(self): - self.bots = {} - self.bot_validation_rules = [] - - def register(self, bot_type: BotType): - """Register a bot type.""" - self.bots[bot_type.name] = bot_type - return self - - def get_bot(self, name: str) -> BotType: - """Get a bot type by name.""" - return self.bots.get(name) - - def detect_bot_type(self, body: Dict[str, Any]) -> Optional[str]: - """Detect which bot type to use based on configuration.""" - # First check for test mode bots - for name, bot in self.bots.items(): - if bot.has_test_mode(body): - return name - - # Then check for specific combinations of settings - for name, bot in self.bots.items(): - if name in body and all(req in body for req in bot.required_settings): - return name - - # Default for dialin settings - if "dialin_settings" in body: - return DEFAULT_DIALIN_EXAMPLE - - return None - - def validate_bot_combination(self, body: Dict[str, Any]) -> List[str]: - """Validate that bot types in the configuration are compatible.""" - errors = [] - bot_types_in_config = [name for name in self.bots.keys() if name in body] - - # Check each bot type against its incompatible list - for bot_name in bot_types_in_config: - bot = self.bots[bot_name] - for incompatible in bot.incompatible_with: - if incompatible in body: - errors.append( - f"Cannot have both '{bot_name}' and '{incompatible}' in the same configuration" - ) - - return errors - - def setup_configuration(self, body: Dict[str, Any]) -> Dict[str, Any]: - """Set up bot configuration based on detected bot type.""" - # Ensure dialout_settings is an array if present - body = ensure_dialout_settings_array(body) - - # Detect which bot type to use - bot_type_name = self.detect_bot_type(body) - if not bot_type_name: - raise HTTPException( - status_code=400, detail="Configuration doesn't match any supported scenario" - ) - - # If we have a dialin scenario but no explicit bot type, add the default - if "dialin_settings" in body and bot_type_name == DEFAULT_DIALIN_EXAMPLE: - if bot_type_name not in body: - body[bot_type_name] = {} - - # Get the bot type object - bot_type = self.get_bot(bot_type_name) - - # Create/update settings for the bot type - body = bot_type.create_settings(body) - - # If in test mode, add any required settings - if bot_type.has_test_mode(body): - body = bot_type.prepare_for_test(body) - - # Validate bot combinations - errors = self.validate_bot_combination(body) - if errors: - error_message = "Invalid configuration: " + "; ".join(errors) - raise HTTPException(status_code=400, detail=error_message) - - return body diff --git a/examples/phone-chatbot/bot_runner.py b/examples/phone-chatbot/bot_runner.py deleted file mode 100644 index 0c3d2e65e..000000000 --- a/examples/phone-chatbot/bot_runner.py +++ /dev/null @@ -1,247 +0,0 @@ -import argparse -import json -import os -import shlex -import subprocess -from contextlib import asynccontextmanager -from typing import Any, Dict - -import aiohttp -from bot_constants import ( - MAX_SESSION_TIME, - REQUIRED_ENV_VARS, -) -from bot_definitions import bot_registry -from bot_runner_helpers import ( - determine_room_capabilities, - ensure_prompt_config, - process_dialin_request, -) -from dotenv import load_dotenv -from fastapi import FastAPI, HTTPException, Request -from fastapi.middleware.cors import CORSMiddleware -from fastapi.responses import JSONResponse - -from pipecat.transports.services.helpers.daily_rest import ( - DailyRESTHelper, - DailyRoomParams, - DailyRoomProperties, - DailyRoomSipParams, -) - -load_dotenv(override=True) - -daily_helpers = {} - - -# ----------------- Daily Room Management ----------------- # - - -async def create_daily_room(room_url: str = None, config_body: Dict[str, Any] = None): - """Create or retrieve a Daily room with appropriate properties based on the configuration. - - Args: - room_url: Optional existing room URL - config_body: Optional configuration that determines room capabilities - - Returns: - Dict containing room URL, token, and SIP endpoint - """ - if not room_url: - # Get room capabilities based on the configuration - capabilities = determine_room_capabilities(config_body) - - # Configure SIP parameters if dialin is needed - sip_params = None - if capabilities["enable_dialin"]: - sip_params = DailyRoomSipParams( - display_name="dialin-user", video=False, sip_mode="dial-in", num_endpoints=2 - ) - - # Create the properties object with the appropriate settings - properties = DailyRoomProperties(sip=sip_params) - - # Set dialout capability if needed - if capabilities["enable_dialout"]: - properties.enable_dialout = True - - # Log the capabilities being used - capability_str = ", ".join([f"{k}={v}" for k, v in capabilities.items()]) - print(f"Creating room with capabilities: {capability_str}") - - params = DailyRoomParams(properties=properties) - - print("Creating new room...") - room = await daily_helpers["rest"].create_room(params=params) - else: - # Check if passed room URL exists - try: - room = await daily_helpers["rest"].get_room_from_url(room_url) - except Exception: - raise HTTPException(status_code=500, detail=f"Room not found: {room_url}") - - print(f"Daily room: {room.url} {room.config.sip_endpoint}") - - # Get token for the agent - token = await daily_helpers["rest"].get_token(room.url, MAX_SESSION_TIME) - - if not room or not token: - raise HTTPException(status_code=500, detail="Failed to get room or token") - - return {"room": room.url, "token": token, "sip_endpoint": room.config.sip_endpoint} - - -# ----------------- Bot Process Management ----------------- # - - -async def start_bot(room_details: Dict[str, str], body: Dict[str, Any], example: str) -> bool: - """Start a bot process with the given configuration. - - Args: - room_details: Room URL and token - body: Bot configuration - example: Example script to run - - Returns: - Boolean indicating success - """ - room_url = room_details["room"] - token = room_details["token"] - - # Properly format body as JSON string for command line - body_json = json.dumps(body).replace('"', '\\"') - print(f"++++ Body JSON: {body_json}") - - # Modified to use non-LLM-specific bot module names - bot_proc = f'python3 -m {example} -u {room_url} -t {token} -b "{body_json}"' - print(f"Starting bot. Example: {example}, Room: {room_url}") - - try: - command_parts = shlex.split(bot_proc) - subprocess.Popen(command_parts, bufsize=1, cwd=os.path.dirname(os.path.abspath(__file__))) - return True - except Exception as e: - raise HTTPException(status_code=500, detail=f"Failed to start subprocess: {e}") - - -# ----------------- API Setup ----------------- # - - -@asynccontextmanager -async def lifespan(app: FastAPI): - aiohttp_session = aiohttp.ClientSession() - daily_helpers["rest"] = DailyRESTHelper( - daily_api_key=os.getenv("DAILY_API_KEY", ""), - daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"), - aiohttp_session=aiohttp_session, - ) - yield - await aiohttp_session.close() - - -app = FastAPI(lifespan=lifespan) - -app.add_middleware( - CORSMiddleware, - allow_origins=["*"], - allow_credentials=True, - allow_methods=["*"], - allow_headers=["*"], -) - - -# ----------------- API Endpoints ----------------- # - - -@app.post("/start") -async def handle_start_request(request: Request) -> JSONResponse: - """Unified endpoint to handle bot configuration for different scenarios.""" - # Get default room URL from environment - room_url = os.getenv("DAILY_SAMPLE_ROOM_URL", None) - - try: - data = await request.json() - - # Handle webhook test - if "test" in data: - return JSONResponse({"test": True}) - - # Handle direct dialin webhook from Daily - if all(key in data for key in ["From", "To", "callId", "callDomain"]): - body = await process_dialin_request(data) - # Handle body-based request - elif "config" in data: - # Use the registry to set up the bot configuration - body = bot_registry.setup_configuration(data["config"]) - else: - raise HTTPException(status_code=400, detail="Invalid request format") - - # Ensure prompt configuration - body = ensure_prompt_config(body) - - # Detect which bot type to use - bot_type_name = bot_registry.detect_bot_type(body) - if not bot_type_name: - raise HTTPException( - status_code=400, detail="Configuration doesn't match any supported scenario" - ) - - # Create the Daily room - room_details = await create_daily_room(room_url, body) - - # Start the bot - await start_bot(room_details, body, bot_type_name) - - # Get the bot type - bot_type = bot_registry.get_bot(bot_type_name) - - # Build the response - response = {"status": "Bot started", "bot_type": bot_type_name} - - # Add room URL for test mode - if bot_type.has_test_mode(body): - response["room_url"] = room_details["room"] - # Remove llm_model from response as it's no longer relevant - if "llm" in body: - response["llm_provider"] = body["llm"] # Optionally keep track of provider - - # Add dialout info for dialout scenarios - if "dialout_settings" in body and len(body["dialout_settings"]) > 0: - first_setting = body["dialout_settings"][0] - if "phoneNumber" in first_setting: - response["dialing_to"] = f"phone:{first_setting['phoneNumber']}" - elif "sipUri" in first_setting: - response["dialing_to"] = f"sip:{first_setting['sipUri']}" - - return JSONResponse(response) - - except json.JSONDecodeError: - raise HTTPException(status_code=400, detail="Invalid JSON in request body") - except Exception as e: - raise HTTPException(status_code=400, detail=f"Request processing error: {str(e)}") - - -# ----------------- Main ----------------- # - -if __name__ == "__main__": - # Check environment variables - for env_var in REQUIRED_ENV_VARS: - if env_var not in os.environ: - raise Exception(f"Missing environment variable: {env_var}.") - - parser = argparse.ArgumentParser(description="Pipecat Bot Runner") - parser.add_argument( - "--host", type=str, default=os.getenv("HOST", "0.0.0.0"), help="Host address" - ) - parser.add_argument("--port", type=int, default=os.getenv("PORT", 7860), help="Port number") - parser.add_argument("--reload", action="store_true", default=True, help="Reload code on change") - - config = parser.parse_args() - - try: - import uvicorn - - uvicorn.run("bot_runner:app", host=config.host, port=config.port, reload=config.reload) - - except KeyboardInterrupt: - print("Pipecat runner shutting down...") diff --git a/examples/phone-chatbot/bot_runner_helpers.py b/examples/phone-chatbot/bot_runner_helpers.py deleted file mode 100644 index bcb04d394..000000000 --- a/examples/phone-chatbot/bot_runner_helpers.py +++ /dev/null @@ -1,211 +0,0 @@ -# bot_runner_helpers.py -from typing import Any, Dict, Optional - -from bot_constants import ( - DEFAULT_CALLTRANSFER_MODE, - DEFAULT_DIALIN_EXAMPLE, - DEFAULT_SPEAK_SUMMARY, - DEFAULT_STORE_SUMMARY, - DEFAULT_TEST_IN_PREBUILT, -) -from call_connection_manager import CallConfigManager - -# ----------------- Configuration Helpers ----------------- # - - -def determine_room_capabilities(config_body: Optional[Dict[str, Any]] = None) -> Dict[str, bool]: - """Determine room capabilities based on the configuration. - - This function examines the configuration to determine which capabilities - the Daily room should have enabled. - - Args: - config_body: Configuration dictionary that determines room capabilities - - Returns: - Dictionary of capability flags - """ - capabilities = { - "enable_dialin": False, - "enable_dialout": False, - # Add more capabilities here in the future as needed - } - - if not config_body: - return capabilities - - # Check for dialin capability - capabilities["enable_dialin"] = "dialin_settings" in config_body - - # Check for dialout capability - needed for outbound calls or transfers - has_dialout_settings = "dialout_settings" in config_body - - # Check if there's a transfer to an operator configured - has_call_transfer = "call_transfer" in config_body - - # Enable dialout if any condition requires it - capabilities["enable_dialout"] = has_dialout_settings or has_call_transfer - - return capabilities - - -def ensure_dialout_settings_array(body: Dict[str, Any]) -> Dict[str, Any]: - """Ensures dialout_settings is an array of objects. - - Args: - body: The configuration dictionary - - Returns: - Updated configuration with dialout_settings as an array - """ - if "dialout_settings" in body: - # Convert to array if it's not already one - if not isinstance(body["dialout_settings"], list): - body["dialout_settings"] = [body["dialout_settings"]] - - return body - - -def ensure_prompt_config(body: Dict[str, Any]) -> Dict[str, Any]: - """Ensures the body has appropriate prompts settings, but doesn't add defaults. - - Only makes sure the prompt section exists, allowing the bot script to handle defaults. - - Args: - body: The configuration dictionary - - Returns: - Updated configuration with prompt settings section - """ - if "prompts" not in body: - body["prompts"] = [] - return body - - -def create_call_transfer_settings(body: Dict[str, Any]) -> Dict[str, Any]: - """Create call transfer settings based on configuration and customer mapping. - - Args: - body: The configuration dictionary - - Returns: - Call transfer settings dictionary - """ - # Default transfer settings - transfer_settings = { - "mode": DEFAULT_CALLTRANSFER_MODE, - "speakSummary": DEFAULT_SPEAK_SUMMARY, - "storeSummary": DEFAULT_STORE_SUMMARY, - "testInPrebuilt": DEFAULT_TEST_IN_PREBUILT, - } - - # If call_transfer already exists, merge the defaults with the existing settings - # This ensures all required fields exist while preserving user-specified values - if "call_transfer" in body: - existing_settings = body["call_transfer"] - # Update defaults with existing settings (existing values will override defaults) - for key, value in existing_settings.items(): - transfer_settings[key] = value - else: - # No existing call_transfer - check if we have dialin settings for customer lookup - if "dialin_settings" in body: - # Create a temporary routing manager just for customer lookup - call_config_manager = CallConfigManager(body) - - # Get caller info - caller_info = call_config_manager.get_caller_info() - from_number = caller_info.get("caller_number") - - if from_number: - # Get customer name from phone number - customer_name = call_config_manager.get_customer_name(from_number) - - # If we know the customer name, add it to the config for the bot to use - if customer_name: - transfer_settings["customerName"] = customer_name - - return transfer_settings - - -def create_simple_dialin_settings(body: Dict[str, Any]) -> Dict[str, Any]: - """Create simple dialin settings based on configuration. - - Args: - body: The configuration dictionary - - Returns: - Simple dialin settings dictionary - """ - # Default simple dialin settings - simple_dialin_settings = { - "testInPrebuilt": DEFAULT_TEST_IN_PREBUILT, - } - - # If simple_dialin already exists, merge the defaults with the existing settings - if "simple_dialin" in body: - existing_settings = body["simple_dialin"] - # Update defaults with existing settings (existing values will override defaults) - for key, value in existing_settings.items(): - simple_dialin_settings[key] = value - - return simple_dialin_settings - - -def create_simple_dialout_settings(body: Dict[str, Any]) -> Dict[str, Any]: - """Create simple dialout settings based on configuration. - - Args: - body: The configuration dictionary - - Returns: - Simple dialout settings dictionary - """ - # Default simple dialout settings - simple_dialout_settings = { - "testInPrebuilt": DEFAULT_TEST_IN_PREBUILT, - } - - # If simple_dialout already exists, merge the defaults with the existing settings - if "simple_dialout" in body: - existing_settings = body["simple_dialout"] - # Update defaults with existing settings (existing values will override defaults) - for key, value in existing_settings.items(): - simple_dialout_settings[key] = value - - return simple_dialout_settings - - -async def process_dialin_request(data: Dict[str, Any]) -> Dict[str, Any]: - """Process incoming dial-in request data to create a properly formatted body. - - Converts camelCase fields received from webhook to snake_case format - for internal consistency across the codebase. - - Args: - data: Raw dialin data from webhook - - Returns: - Properly formatted configuration with snake_case keys - """ - # Create base body with dialin settings - body = { - "dialin_settings": { - "to": data.get("To", ""), - "from": data.get("From", ""), - "call_id": data.get("callId", data.get("CallSid", "")), # Convert to snake_case - "call_domain": data.get("callDomain", ""), # Convert to snake_case - } - } - - # Use the global default to determine which example to run for dialin webhooks - example = DEFAULT_DIALIN_EXAMPLE - - # Configure the bot based on the example - if example == "call_transfer": - # Create call transfer settings - body["call_transfer"] = create_call_transfer_settings(body) - elif example == "simple_dialin": - # Create simple dialin settings - body["simple_dialin"] = create_simple_dialin_settings(body) - - return body diff --git a/examples/phone-chatbot/call_connection_manager.py b/examples/phone-chatbot/call_connection_manager.py deleted file mode 100644 index ef1b9b97a..000000000 --- a/examples/phone-chatbot/call_connection_manager.py +++ /dev/null @@ -1,608 +0,0 @@ -# -# Copyright (c) 2024–2025, Daily -# -# SPDX-License-Identifier: BSD 2-Clause License -# -"""call_connection_manager.py. - -Manages customer/operator relationships and call routing for voice bots. -Provides mapping between customers and operators, and functions for retrieving -contact information. Also includes call state management. -""" - -import json -import os -from typing import Any, Dict, List, Optional - -from loguru import logger - - -class CallFlowState: - """State for tracking call flow operations and state transitions.""" - - def __init__(self): - # Operator-related state - self.dialed_operator = False - self.operator_connected = False - self.current_operator_index = 0 - self.operator_dialout_settings = [] - self.summary_finished = False - - # Voicemail detection state - self.voicemail_detected = False - self.human_detected = False - self.voicemail_message_left = False - - # Call termination state - self.call_terminated = False - self.participant_left_early = False - - # Operator-related methods - def set_operator_dialed(self): - """Mark that an operator has been dialed.""" - self.dialed_operator = True - - def set_operator_connected(self): - """Mark that an operator has connected to the call.""" - self.operator_connected = True - # Summary is not finished when operator first connects - self.summary_finished = False - - def set_operator_disconnected(self): - """Handle operator disconnection.""" - self.operator_connected = False - self.summary_finished = False - - def set_summary_finished(self): - """Mark the summary as finished.""" - self.summary_finished = True - - def set_operator_dialout_settings(self, settings): - """Set the list of operator dialout settings to try.""" - self.operator_dialout_settings = settings - self.current_operator_index = 0 - - def get_current_dialout_setting(self): - """Get the current operator dialout setting to try.""" - if not self.operator_dialout_settings or self.current_operator_index >= len( - self.operator_dialout_settings - ): - return None - return self.operator_dialout_settings[self.current_operator_index] - - def move_to_next_operator(self): - """Move to the next operator in the list.""" - self.current_operator_index += 1 - return self.get_current_dialout_setting() - - # Voicemail detection methods - def set_voicemail_detected(self): - """Mark that a voicemail system has been detected.""" - self.voicemail_detected = True - self.human_detected = False - - def set_human_detected(self): - """Mark that a human has been detected (not voicemail).""" - self.human_detected = True - self.voicemail_detected = False - - def set_voicemail_message_left(self): - """Mark that a voicemail message has been left.""" - self.voicemail_message_left = True - - # Call termination methods - def set_call_terminated(self): - """Mark that the call has been terminated by the bot.""" - self.call_terminated = True - - def set_participant_left_early(self): - """Mark that a participant left the call early.""" - self.participant_left_early = True - - -class SessionManager: - """Centralized management of session IDs and state for all call participants.""" - - def __init__(self): - # Track session IDs of different participant types - self.session_ids = { - "operator": None, - "customer": None, - "bot": None, - # Add other participant types as needed - } - - # References for easy access in processors that need mutable containers - self.session_id_refs = { - "operator": [None], - "customer": [None], - "bot": [None], - # Add other participant types as needed - } - - # State object for call flow - self.call_flow_state = CallFlowState() - - def set_session_id(self, participant_type, session_id): - """Set the session ID for a specific participant type. - - Args: - participant_type: Type of participant (e.g., "operator", "customer", "bot") - session_id: The session ID to set - """ - if participant_type in self.session_ids: - self.session_ids[participant_type] = session_id - - # Also update the corresponding reference if it exists - if participant_type in self.session_id_refs: - self.session_id_refs[participant_type][0] = session_id - - def get_session_id(self, participant_type): - """Get the session ID for a specific participant type. - - Args: - participant_type: Type of participant (e.g., "operator", "customer", "bot") - - Returns: - The session ID or None if not set - """ - return self.session_ids.get(participant_type) - - def get_session_id_ref(self, participant_type): - """Get the mutable reference for a specific participant type. - - Args: - participant_type: Type of participant (e.g., "operator", "customer", "bot") - - Returns: - A mutable list container holding the session ID or None if not available - """ - return self.session_id_refs.get(participant_type) - - def is_participant_type(self, session_id, participant_type): - """Check if a session ID belongs to a specific participant type. - - Args: - session_id: The session ID to check - participant_type: Type of participant (e.g., "operator", "customer", "bot") - - Returns: - True if the session ID matches the participant type, False otherwise - """ - return self.session_ids.get(participant_type) == session_id - - def reset_participant(self, participant_type): - """Reset the state for a specific participant type. - - Args: - participant_type: Type of participant (e.g., "operator", "customer", "bot") - """ - if participant_type in self.session_ids: - self.session_ids[participant_type] = None - - if participant_type in self.session_id_refs: - self.session_id_refs[participant_type][0] = None - - # Additional reset actions for specific participant types - if participant_type == "operator": - self.call_flow_state.set_operator_disconnected() - - -class CallConfigManager: - """Manages customer/operator relationships and call routing.""" - - def __init__(self, body_data: Dict[str, Any] = None): - """Initialize with optional body data. - - Args: - body_data: Optional dictionary containing request body data - """ - self.body = body_data or {} - - # Get environment variables with fallbacks - self.dial_in_from_number = os.getenv("DIAL_IN_FROM_NUMBER", "+10000000001") - self.dial_out_to_number = os.getenv("DIAL_OUT_TO_NUMBER", "+10000000002") - self.operator_number = os.getenv("OPERATOR_NUMBER", "+10000000003") - - # Initialize maps with dynamic values - self._initialize_maps() - self._build_reverse_lookup_maps() - - def _initialize_maps(self): - """Initialize the customer and operator maps with environment variables.""" - # Maps customer names to their contact information - self.CUSTOMER_MAP = { - "Dominic": { - "phoneNumber": self.dial_in_from_number, # I have two phone numbers, one for dialing in and one for dialing out. I give myself a separate name for each. - }, - "Stewart": { - "phoneNumber": self.dial_out_to_number, - }, - "James": { - "phoneNumber": "+10000000000", - "callerId": "james-caller-id-uuid", - "sipUri": "sip:james@example.com", - }, - "Sarah": { - "sipUri": "sip:sarah@example.com", - }, - "Michael": { - "phoneNumber": "+16505557890", - "callerId": "michael-caller-id-uuid", - }, - } - - # Maps customer names to their assigned operator names - self.CUSTOMER_TO_OPERATOR_MAP = { - "Dominic": ["Yunyoung", "Maria"], # Try Yunyoung first, then Maria - "Stewart": "Yunyoung", - "James": "Yunyoung", - "Sarah": "Jennifer", - "Michael": "Paul", - # Default mapping to ensure all customers have an operator - "Default": "Yunyoung", - } - - # Maps operator names to their contact details - self.OPERATOR_CONTACT_MAP = { - "Paul": { - "phoneNumber": "+12345678904", - "callerId": "paul-caller-id-uuid", - }, - "Yunyoung": { - "phoneNumber": self.operator_number, # Dials out to my other phone number. - }, - "Maria": { - "sipUri": "sip:maria@example.com", - }, - "Jennifer": {"phoneNumber": "+14155559876", "callerId": "jennifer-caller-id-uuid"}, - "Default": { - "phoneNumber": self.operator_number, # Use the operator number as default - }, - } - - def _build_reverse_lookup_maps(self): - """Build reverse lookup maps for phone numbers and SIP URIs to customer names.""" - self._PHONE_TO_CUSTOMER_MAP = {} - self._SIP_TO_CUSTOMER_MAP = {} - - for customer_name, contact_info in self.CUSTOMER_MAP.items(): - if "phoneNumber" in contact_info: - self._PHONE_TO_CUSTOMER_MAP[contact_info["phoneNumber"]] = customer_name - if "sipUri" in contact_info: - self._SIP_TO_CUSTOMER_MAP[contact_info["sipUri"]] = customer_name - - @classmethod - def from_json_string(cls, json_string: str): - """Create a CallRoutingManager from a JSON string. - - Args: - json_string: JSON string containing body data - - Returns: - CallRoutingManager instance with parsed data - - Raises: - json.JSONDecodeError: If JSON string is invalid - """ - body_data = json.loads(json_string) - return cls(body_data) - - def find_customer_by_contact(self, contact_info: str) -> Optional[str]: - """Find customer name from a contact identifier (phone number or SIP URI). - - Args: - contact_info: The contact identifier (phone number or SIP URI) - - Returns: - The customer name or None if not found - """ - # Check if it's a phone number - if contact_info in self._PHONE_TO_CUSTOMER_MAP: - return self._PHONE_TO_CUSTOMER_MAP[contact_info] - - # Check if it's a SIP URI - if contact_info in self._SIP_TO_CUSTOMER_MAP: - return self._SIP_TO_CUSTOMER_MAP[contact_info] - - return None - - def get_customer_name(self, phone_number: str) -> Optional[str]: - """Get customer name from their phone number. - - Args: - phone_number: The customer's phone number - - Returns: - The customer name or None if not found - """ - # Note: In production, this would likely query a database - return self.find_customer_by_contact(phone_number) - - def get_operators_for_customer(self, customer_name: Optional[str]) -> List[str]: - """Get the operator name(s) assigned to a customer. - - Args: - customer_name: The customer's name - - Returns: - List of operator names (single item or multiple) - """ - # Note: In production, this would likely query a database - if not customer_name or customer_name not in self.CUSTOMER_TO_OPERATOR_MAP: - return ["Default"] - - operators = self.CUSTOMER_TO_OPERATOR_MAP[customer_name] - # Convert single string to list for consistency - if isinstance(operators, str): - return [operators] - return operators - - def get_operator_dialout_settings(self, operator_name: str) -> Dict[str, str]: - """Get an operator's dialout settings from their name. - - Args: - operator_name: The operator's name - - Returns: - Dictionary with dialout settings for the operator - """ - # Note: In production, this would likely query a database - return self.OPERATOR_CONTACT_MAP.get(operator_name, self.OPERATOR_CONTACT_MAP["Default"]) - - def get_dialout_settings_for_caller( - self, from_number: Optional[str] = None - ) -> List[Dict[str, str]]: - """Determine the appropriate operator dialout settings based on caller's number. - - This method uses the caller's number to look up the customer name, - then finds the assigned operators for that customer, and returns - an array of operator dialout settings to try in sequence. - - Args: - from_number: The caller's phone number (from dialin_settings) - - Returns: - List of operator dialout settings to try - """ - if not from_number: - # If we don't have dialin settings, use the Default operator - return [self.get_operator_dialout_settings("Default")] - - # Get customer name from phone number - customer_name = self.get_customer_name(from_number) - - # Get operator names assigned to this customer - operator_names = self.get_operators_for_customer(customer_name) - - # Get dialout settings for each operator - return [self.get_operator_dialout_settings(name) for name in operator_names] - - def get_caller_info(self) -> Dict[str, Optional[str]]: - """Get caller and dialed numbers from dialin settings in the body. - - Returns: - Dictionary containing caller_number and dialed_number - """ - raw_dialin_settings = self.body.get("dialin_settings") - if not raw_dialin_settings: - return {"caller_number": None, "dialed_number": None} - - # Handle different case variations - dialed_number = raw_dialin_settings.get("To") or raw_dialin_settings.get("to") - caller_number = raw_dialin_settings.get("From") or raw_dialin_settings.get("from") - - return {"caller_number": caller_number, "dialed_number": dialed_number} - - def get_caller_number(self) -> Optional[str]: - """Get the caller's phone number from dialin settings in the body. - - Returns: - The caller's phone number or None if not available - """ - return self.get_caller_info()["caller_number"] - - async def start_dialout(self, transport, dialout_settings=None): - """Helper function to start dialout using the provided settings or from body. - - Args: - transport: The transport instance to use for dialout - dialout_settings: Optional override for dialout settings - - Returns: - None - """ - # Use provided settings or get from body - settings = dialout_settings or self.get_dialout_settings() - if not settings: - logger.warning("No dialout settings available") - return - - for setting in settings: - if "phoneNumber" in setting: - logger.info(f"Dialing number: {setting['phoneNumber']}") - if "callerId" in setting: - logger.info(f"with callerId: {setting['callerId']}") - await transport.start_dialout( - {"phoneNumber": setting["phoneNumber"], "callerId": setting["callerId"]} - ) - else: - logger.info("with no callerId") - await transport.start_dialout({"phoneNumber": setting["phoneNumber"]}) - elif "sipUri" in setting: - logger.info(f"Dialing sipUri: {setting['sipUri']}") - await transport.start_dialout({"sipUri": setting["sipUri"]}) - else: - logger.warning(f"Unknown dialout setting format: {setting}") - - def get_dialout_settings(self) -> Optional[List[Dict[str, Any]]]: - """Extract dialout settings from the body. - - Returns: - List of dialout setting objects or None if not present - """ - # Check if we have dialout settings - if "dialout_settings" in self.body: - dialout_settings = self.body["dialout_settings"] - - # Convert to list if it's an object (for backward compatibility) - if isinstance(dialout_settings, dict): - return [dialout_settings] - elif isinstance(dialout_settings, list): - return dialout_settings - - return None - - def get_dialin_settings(self) -> Optional[Dict[str, Any]]: - """Extract dialin settings from the body. - - Handles both camelCase and snake_case variations of fields for backward compatibility, - but normalizes to snake_case for internal usage. - - Returns: - Dictionary containing dialin settings or None if not present - """ - raw_dialin_settings = self.body.get("dialin_settings") - if not raw_dialin_settings: - return None - - # Normalize dialin settings to handle different case variations - # Prioritize snake_case (call_id, call_domain) but fall back to camelCase (callId, callDomain) - dialin_settings = { - "call_id": raw_dialin_settings.get("call_id") or raw_dialin_settings.get("callId"), - "call_domain": raw_dialin_settings.get("call_domain") - or raw_dialin_settings.get("callDomain"), - "to": raw_dialin_settings.get("to") or raw_dialin_settings.get("To"), - "from": raw_dialin_settings.get("from") or raw_dialin_settings.get("From"), - } - - return dialin_settings - - # Bot prompt helper functions - no defaults provided, just return what's in the body - - def get_prompt(self, prompt_name: str) -> Optional[str]: - """Retrieve the prompt text for a given prompt name. - - Args: - prompt_name: The name of the prompt to retrieve. - - Returns: - The prompt string corresponding to the provided name, or None if not configured. - """ - prompts = self.body.get("prompts", []) - for prompt in prompts: - if prompt.get("name") == prompt_name: - return prompt.get("text") - return None - - def get_transfer_mode(self) -> Optional[str]: - """Get transfer mode from the body. - - Returns: - Transfer mode string or None if not configured - """ - if "call_transfer" in self.body: - return self.body["call_transfer"].get("mode") - return None - - def get_speak_summary(self) -> Optional[bool]: - """Get speak summary from the body. - - Returns: - Boolean indicating if summary should be spoken or None if not configured - """ - if "call_transfer" in self.body: - return self.body["call_transfer"].get("speakSummary") - return None - - def get_store_summary(self) -> Optional[bool]: - """Get store summary from the body. - - Returns: - Boolean indicating if summary should be stored or None if not configured - """ - if "call_transfer" in self.body: - return self.body["call_transfer"].get("storeSummary") - return None - - def is_test_mode(self) -> bool: - """Check if running in test mode. - - Returns: - Boolean indicating if test mode is enabled - """ - if "voicemail_detection" in self.body: - return bool(self.body["voicemail_detection"].get("testInPrebuilt")) - if "call_transfer" in self.body: - return bool(self.body["call_transfer"].get("testInPrebuilt")) - if "simple_dialin" in self.body: - return bool(self.body["simple_dialin"].get("testInPrebuilt")) - if "simple_dialout" in self.body: - return bool(self.body["simple_dialout"].get("testInPrebuilt")) - return False - - def is_voicemail_detection_enabled(self) -> bool: - """Check if voicemail detection is enabled in the body. - - Returns: - Boolean indicating if voicemail detection is enabled - """ - return bool(self.body.get("voicemail_detection")) - - def customize_prompt(self, prompt: str, customer_name: Optional[str] = None) -> str: - """Insert customer name into prompt template if available. - - Args: - prompt: The prompt template containing optional {customer_name} placeholders - customer_name: Optional customer name to insert - - Returns: - Customized prompt with customer name inserted - """ - if customer_name and prompt: - return prompt.replace("{customer_name}", customer_name) - return prompt - - def create_system_message(self, content: str) -> Dict[str, str]: - """Create a properly formatted system message. - - Args: - content: The message content - - Returns: - Dictionary with role and content for the system message - """ - return {"role": "system", "content": content} - - def create_user_message(self, content: str) -> Dict[str, str]: - """Create a properly formatted user message. - - Args: - content: The message content - - Returns: - Dictionary with role and content for the user message - """ - return {"role": "user", "content": content} - - def get_customer_info_suffix( - self, customer_name: Optional[str] = None, preposition: str = "for" - ) -> str: - """Create a consistent customer info suffix. - - Args: - customer_name: Optional customer name - preposition: Preposition to use before the name (e.g., "for", "to", "") - - Returns: - String with formatted customer info suffix - """ - if not customer_name: - return "" - - # Add a space before the preposition if it's not empty - space_prefix = " " if preposition else "" - # For non-empty prepositions, add a space after it - space_suffix = " " if preposition else "" - - return f"{space_prefix}{preposition}{space_suffix}{customer_name}" diff --git a/examples/phone-chatbot/daily-pstn-advanced-voicemail-detection/README.md b/examples/phone-chatbot/daily-pstn-advanced-voicemail-detection/README.md new file mode 100644 index 000000000..d7240512f --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-advanced-voicemail-detection/README.md @@ -0,0 +1,122 @@ +# Daily PSTN Advanced Voicemail Detection Bot + +This project demonstrates how to create a voice bot that uses Dailys PSTN capabilities to make calls to phone numbers, and if the bot hits a voicemail system, to have the bot also leave a message. In this example, we have two pipelines. The voicemail detection pipeline uses Gemini Flash Lite, a fast and cheap LLM that works well for voicemail detection. The second pipeline uses Gemini Flash, a more advanced LLM model ideal for conversations. + +## How it works + +1. The server file receives a curl request with the phone number to dial out to +2. The server creates a Daily room with SIP capabilities +3. The server starts the bot process with the room details +4. When the bot has joined, it starts the dial-out process and rings the number provided in the curl request +5. When the phone is answered, the bot detects for certain key phrases +6. Gemini Flash Lite works best when given small, concise prompts. When a voicemail machine is detected, we switch to a new prompt focused on the message that must be left +7. Once the bot has left the message, it then ends the call +8. If the bot detects there's a human on the phone, the bot runs a function call and switches to the human conversation pipeline. We give the new LLM a prompt and tell the LLM to speak. + +## Prerequisites + +- A Daily account with an API key, and a phone number purchased through Daily +- A US phone number to ring +- dial-out must be enabled on your domain. Find out more by reading this [document and filling in the form](https://docs.daily.co/guides/products/dial-in-dial-out#main) +- Google API key for the bot's intelligence +- Cartesia API key for text-to-speech + +## Setup + +1. Create a virtual environment and install dependencies + +```bash +python -m venv venv +source venv/bin/activate # On Windows: venv\Scripts\activate +pip install -r requirements.txt +``` + +2. Set up environment variables + +Copy the example file and fill in your API keys: + +```bash +cp .env.example .env +# Edit .env with your API keys +``` + +3. Buy a phone number + +Instructions on how to do that can be found at this [docs link:](https://docs.daily.co/reference/rest-api/phone-numbers/buy-phone-number) + +4. Request dial-out enablement + +For compliance reasons, to enable dial-out for your Daily account, you must request enablement via the form. You can find out more about dial-out, and the form at the [link here:](https://docs.daily.co/guides/products/dial-in-dial-out#main) + +## Running the Server + +Start the webhook server: + +```bash +python server.py +``` + +## Testing + +With server.py running, send the following curl command from your terminal: + +```bash +curl -X POST "http://127.0.0.1:7860/start" \ + -H "Content-Type: application/json" \ + -d '{ + "dialout_settings": { + "phone_number": "+12345678910" + } + }' +``` + +The server should make a room. The bot will join the room and then ring the number provided. Answer the call to speak with the bot. + +- You can pretend to be a voicemail machine by saying something like "Please leave a message after the beep... beeeeep". +- You should observe the bot detects the voicemail machine and leaves a message before terminating the call +- You can also say something like "Hello?", and the bot will notice you're likely a human and begin having a conversation with you + +## Customizing the Bot + +You can customize the bot's behavior by modifying the system prompt in `bot.py`. + +## Multiple SIP Endpoints + +For PSTN calls, you only need one SIP endpoint. + +## Daily dial-out configuration + +The bot configures the Daily rooms with dial-out capabilities using these settings. Note: You also need dial-out to be enabled on the domain, as mentioned earlier on in the README. + +```python +properties = DailyRoomProperties( + sip=sip_params, + enable_dialout=True, # Needed for outbound calls if you expand the bot + enable_chat=False, # No need for chat in a voice bot + start_video_off=True, # Voice only +) +``` + +## Troubleshooting + +### I get an error about dial-out not being enabled + +- Check that your room has `enable_dialout=True` set +- Check that your meeting token is an owner token (The bot does this for you automatically) +- Check that you have purchased a phone number to ring from +- Check that the phone number you are trying to ring is correct, and is a US or Canadian number. + +### The bot doesn't detect my voicemail + +- The bot should be smart enough to detect variations of certain patterns, +- If your voicemail machine doesn't follow the example patterns, add the pattern to the LLM prompt + +### Call connects but no bot is heard + +- Ensure your Daily API key is correct and has SIP capabilities +- Verify that the Cartesia API key and voice ID are correct + +### Bot starts but disconnects immediately + +- Check the Daily logs for any error messages +- Ensure your server has stable internet connectivity diff --git a/examples/phone-chatbot/voicemail_detection.py b/examples/phone-chatbot/daily-pstn-advanced-voicemail-detection/bot.py similarity index 68% rename from examples/phone-chatbot/voicemail_detection.py rename to examples/phone-chatbot/daily-pstn-advanced-voicemail-detection/bot.py index e5b51eb55..7eeaef39e 100644 --- a/examples/phone-chatbot/voicemail_detection.py +++ b/examples/phone-chatbot/daily-pstn-advanced-voicemail-detection/bot.py @@ -6,11 +6,11 @@ import argparse import asyncio -import functools +import json import os import sys +from typing import Any -from call_connection_manager import CallConfigManager, SessionManager from dotenv import load_dotenv from loguru import logger @@ -50,6 +50,39 @@ daily_api_url = os.getenv("DAILY_API_URL", "https://api.daily.co/v1") # ------------ HELPER CLASSES ------------ +class CallFlowState: + """State for tracking call flow operations and state transitions.""" + + def __init__(self): + # Voicemail detection state + self.voicemail_detected = False + self.human_detected = False + + # Call termination state + self.call_terminated = False + self.participant_left_early = False + + # Voicemail detection methods + def set_voicemail_detected(self): + """Mark that a voicemail system has been detected.""" + self.voicemail_detected = True + self.human_detected = False + + def set_human_detected(self): + """Mark that a human has been detected (not voicemail).""" + self.human_detected = True + self.voicemail_detected = False + + # Call termination methods + def set_call_terminated(self): + """Mark that the call has been terminated by the bot.""" + self.call_terminated = True + + def set_participant_left_early(self): + """Mark that a participant left the call early.""" + self.participant_left_early = True + + class UserAudioCollector(FrameProcessor): """Collects audio frames in a buffer, then adds them to the LLM context when the user stops speaking.""" @@ -94,9 +127,8 @@ class UserAudioCollector(FrameProcessor): class FunctionHandlers: """Handlers for the voicemail detection bot functions.""" - def __init__(self, session_manager): - self.session_manager = session_manager - self.prompt = None # Can be set externally + def __init__(self, call_flow_state: CallFlowState): + self.call_flow_state = call_flow_state async def voicemail_response(self, params: FunctionCallParams): """Function the bot can call to leave a voicemail message.""" @@ -109,49 +141,73 @@ class FunctionHandlers: async def human_conversation(self, params: FunctionCallParams): """Function called when bot detects it's talking to a human.""" # Update state to indicate human was detected - self.session_manager.call_flow_state.set_human_detected() + self.call_flow_state.set_human_detected() await params.llm.push_frame(StopTaskFrame(), FrameDirection.UPSTREAM) # ------------ MAIN FUNCTION ------------ -async def main( +async def run_bot( room_url: str, token: str, body: dict, -): +) -> None: + """Run the voice bot with the given parameters. + + Args: + room_url: The Daily room URL + token: The Daily room token + body: Body passed to the bot from the webhook + + """ # ------------ CONFIGURATION AND SETUP ------------ + logger.info(f"Starting bot with room: {room_url}") + logger.info(f"Token: {token}") + logger.info(f"Body: {body}") + # Parse the body to get the dial-in settings + body_data = json.loads(body) - # Create a configuration manager from the provided body - call_config_manager = CallConfigManager.from_json_string(body) if body else CallConfigManager() + # Check if the body contains dial-in settings + logger.debug(f"Body data: {body_data}") - # Get important configuration values - dialout_settings = call_config_manager.get_dialout_settings() - test_mode = call_config_manager.is_test_mode() + if not body_data.get("dialout_settings"): + logger.error("Dial-out settings not found in the body data") + return - # Get caller info (might be None for dialout scenarios) - caller_info = call_config_manager.get_caller_info() - logger.info(f"Caller info: {caller_info}") + dialout_settings = body_data["dialout_settings"] - # Initialize the session manager - session_manager = SessionManager() + if not dialout_settings.get("phone_number"): + logger.error("Dial-out phone number not found in the dial-out settings") + return - # ------------ TRANSPORT AND SERVICES SETUP ------------ + # Extract dial-out phone number + phone_number = dialout_settings["phone_number"] + caller_id = dialout_settings.get("caller_id") # Use .get() to handle optional field - # Initialize transport + if caller_id: + logger.info(f"Dial-out caller ID specified: {caller_id}") + else: + logger.info("Dial-out caller ID not specified; proceeding without it") + + # ------------ TRANSPORT SETUP ------------ + + transport_params = DailyParams( + api_url=daily_api_url, + api_key=daily_api_key, + audio_in_enabled=True, + audio_out_enabled=True, + video_out_enabled=False, + vad_analyzer=SileroVADAnalyzer(), + transcription_enabled=True, + ) + + # Initialize transport with Daily transport = DailyTransport( room_url, token, "Voicemail Detection Bot", - DailyParams( - api_url=daily_api_url, - api_key=daily_api_key, - audio_in_enabled=True, - audio_out_enabled=True, - video_out_enabled=False, - vad_analyzer=SileroVADAnalyzer(), - ), + transport_params, ) # Initialize TTS @@ -167,12 +223,12 @@ async def main( async def terminate_call( params: FunctionCallParams, - session_manager=None, + call_flow_state: CallFlowState = None, ): """Function the bot can call to terminate the call.""" - if session_manager: + if call_flow_state: # Set call terminated flag in the session manager - session_manager.call_flow_state.set_call_terminated() + call_flow_state.set_call_terminated() await params.llm.queue_frame(EndTaskFrame(), FrameDirection.UPSTREAM) @@ -198,12 +254,7 @@ async def main( } ] - # Get voicemail detection prompt - voicemail_detection_prompt = call_config_manager.get_prompt("voicemail_detection_prompt") - if voicemail_detection_prompt: - system_instruction = voicemail_detection_prompt - else: - system_instruction = """You are Chatbot trying to determine if this is a voicemail system or a human. + system_instruction = """You are Chatbot trying to determine if this is a voicemail system or a human. If you hear any of these phrases (or very similar ones): - "Please leave a message after the beep" @@ -238,12 +289,9 @@ async def main( voicemail_detection_context ) - # Get custom voicemail prompt if available - voicemail_prompt = call_config_manager.get_prompt("voicemail_prompt") - # Set up function handlers - handlers = FunctionHandlers(session_manager) - handlers.prompt = voicemail_prompt # Set custom prompt if available + call_flow_state = CallFlowState() + handlers = FunctionHandlers(call_flow_state) # Register functions with the voicemail detection LLM voicemail_detection_llm.register_function( @@ -254,7 +302,7 @@ async def main( "switch_to_human_conversation", handlers.human_conversation ) voicemail_detection_llm.register_function( - "terminate_call", lambda params: terminate_call(params, session_manager) + "terminate_call", lambda params: terminate_call(params, call_flow_state) ) # Set up audio collector for handling audio input @@ -279,17 +327,41 @@ async def main( voicemail_detection_pipeline_task = PipelineTask( voicemail_detection_pipeline, params=PipelineParams(allow_interruptions=True), - check_dangling_tasks=False, ) + # ------------ RETRY LOGIC VARIABLES ------------ + max_retries = 5 + retry_count = 0 + dialout_successful = False + + # Build dialout parameters conditionally + dialout_params = {"phoneNumber": phone_number} + if caller_id: + dialout_params["callerId"] = caller_id + logger.debug(f"Including caller ID in dialout: {caller_id}") + + logger.debug(f"Dialout parameters: {dialout_params}") + + async def attempt_dialout(): + """Attempt to start dialout with retry logic.""" + nonlocal retry_count, dialout_successful + + if retry_count < max_retries and not dialout_successful: + retry_count += 1 + logger.info( + f"Attempting dialout (attempt {retry_count}/{max_retries}) to: {phone_number}" + ) + await transport.start_dialout(dialout_params) + else: + logger.error(f"Maximum retry attempts ({max_retries}) reached. Giving up on dialout.") + # ------------ EVENT HANDLERS ------------ @transport.event_handler("on_joined") async def on_joined(transport, data): - # Start dialout if needed - if not test_mode and dialout_settings: - logger.debug("Dialout settings detected; starting dialout") - await call_config_manager.start_dialout(transport, dialout_settings) + # Start initial dialout attempt + logger.debug(f"Dialout settings detected; starting dialout to number: {phone_number}") + await attempt_dialout() @transport.event_handler("on_dialout_connected") async def on_dialout_connected(transport, data): @@ -297,27 +369,36 @@ async def main( @transport.event_handler("on_dialout_answered") async def on_dialout_answered(transport, data): + nonlocal dialout_successful logger.debug(f"Dial-out answered: {data}") - # Start capturing transcription + dialout_successful = True # Mark as successful to stop retries + # Automatically start capturing transcription for the participant await transport.capture_participant_transcription(data["sessionId"]) + # The bot will wait to hear the user before the bot speaks + + @transport.event_handler("on_dialout_error") + async def on_dialout_error(transport, data: Any): + logger.error(f"Dial-out error (attempt {retry_count}/{max_retries}): {data}") + + if retry_count < max_retries: + logger.info(f"Retrying dialout") + await attempt_dialout() + else: + logger.error(f"All {max_retries} dialout attempts failed. Stopping bot.") + await voicemail_detection_pipeline_task.queue_frame(EndFrame()) @transport.event_handler("on_first_participant_joined") async def on_first_participant_joined(transport, participant): logger.debug(f"First participant joined: {participant['id']}") - if test_mode: - await transport.capture_participant_transcription(participant["id"]) @transport.event_handler("on_participant_left") async def on_participant_left(transport, participant, reason): # Mark that a participant left early - session_manager.call_flow_state.set_participant_left_early() + call_flow_state.set_participant_left_early() await voicemail_detection_pipeline_task.queue_frame(EndFrame()) # ------------ RUN VOICEMAIL DETECTION PIPELINE ------------ - if test_mode: - logger.debug("Detect voicemail example. You can test this in Daily Prebuilt") - runner = PipelineRunner() print("!!! starting voicemail detection pipeline") @@ -331,24 +412,17 @@ async def main( print("!!! Done with voicemail detection pipeline") # Check if we should exit early - if ( - session_manager.call_flow_state.participant_left_early - or session_manager.call_flow_state.call_terminated - ): - if session_manager.call_flow_state.participant_left_early: + if call_flow_state.participant_left_early or call_flow_state.call_terminated: + if call_flow_state.participant_left_early: print("!!! Participant left early; terminating call") - elif session_manager.call_flow_state.call_terminated: + elif call_flow_state.call_terminated: print("!!! Bot terminated call; not proceeding to human conversation") return # ------------ HUMAN CONVERSATION PHASE SETUP ------------ # Get human conversation prompt - human_conversation_prompt = call_config_manager.get_prompt("human_conversation_prompt") - if human_conversation_prompt: - human_conversation_system_instruction = human_conversation_prompt - else: - human_conversation_system_instruction = """You are Chatbot talking to a human. Be friendly and helpful. + human_conversation_system_instruction = """You are Chatbot talking to a human. Be friendly and helpful. Start with: "Hello! I'm a friendly chatbot. How can I help you today?" @@ -378,7 +452,7 @@ async def main( # Register terminate function with the human conversation LLM human_conversation_llm.register_function( - "terminate_call", functools.partial(terminate_call, session_manager=session_manager) + "terminate_call", lambda params: terminate_call(params, call_flow_state) ) # Build human conversation pipeline @@ -412,7 +486,12 @@ async def main( # Initialize the context with system message human_conversation_context_aggregator.user().set_messages( - [call_config_manager.create_system_message(human_conversation_system_instruction)] + [ + { + "role": "system", + "content": human_conversation_system_instruction, + } + ] ) # Queue the context frame to start the conversation @@ -434,17 +513,26 @@ async def main( # ------------ SCRIPT ENTRY POINT ------------ -if __name__ == "__main__": - parser = argparse.ArgumentParser(description="Pipecat Voicemail Detection Bot") + +async def main(): + """Parse command line arguments and run the bot.""" + parser = argparse.ArgumentParser(description="Simple Dial-out Bot") parser.add_argument("-u", "--url", type=str, help="Room URL") parser.add_argument("-t", "--token", type=str, help="Room Token") parser.add_argument("-b", "--body", type=str, help="JSON configuration string") args = parser.parse_args() - # Log the arguments for debugging - logger.info(f"Room URL: {args.url}") - logger.info(f"Token: {args.token}") - logger.info(f"Body provided: {bool(args.body)}") + logger.debug(f"url: {args.url}") + logger.debug(f"token: {args.token}") + logger.debug(f"body: {args.body}") + if not all([args.url, args.token, args.body]): + logger.error("All arguments (-u, -t, -b) are required") + parser.print_help() + sys.exit(1) - asyncio.run(main(args.url, args.token, args.body)) + await run_bot(args.url, args.token, args.body) + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/examples/phone-chatbot/daily-pstn-advanced-voicemail-detection/env.example b/examples/phone-chatbot/daily-pstn-advanced-voicemail-detection/env.example new file mode 100644 index 000000000..78f2b2613 --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-advanced-voicemail-detection/env.example @@ -0,0 +1,7 @@ +# Daily credentials +DAILY_API_KEY=your_daily_api_key +DAILY_API_URL=https://api.daily.co/v1 + +# Service keys +GOOGLE_API_KEY=your_google_api_key +CARTESIA_API_KEY=your_cartesia_api_key \ No newline at end of file diff --git a/examples/phone-chatbot/daily-pstn-advanced-voicemail-detection/requirements.txt b/examples/phone-chatbot/daily-pstn-advanced-voicemail-detection/requirements.txt new file mode 100644 index 000000000..744fb4c64 --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-advanced-voicemail-detection/requirements.txt @@ -0,0 +1,6 @@ +pipecat-ai[daily,cartesia,google,deepgram,silero] +fastapi==0.115.6 +uvicorn +python-dotenv +python-multipart +aiohttp diff --git a/examples/phone-chatbot/daily-pstn-advanced-voicemail-detection/server.py b/examples/phone-chatbot/daily-pstn-advanced-voicemail-detection/server.py new file mode 100644 index 000000000..4aed288fc --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-advanced-voicemail-detection/server.py @@ -0,0 +1,121 @@ +# +# Copyright (c) 2024–2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +"""server.py. + +Webhook server to handle webhook coming from Daily, create a Daily room and start the bot. +""" + +import json +import os +import shlex +import subprocess +from contextlib import asynccontextmanager + +import aiohttp +import uvicorn +from dotenv import load_dotenv +from fastapi import FastAPI, HTTPException, Request +from fastapi.responses import JSONResponse +from utils.daily_helpers import create_daily_room + +load_dotenv() + +# ----------------- API ----------------- # + + +@asynccontextmanager +async def lifespan(app: FastAPI): + # Create aiohttp session to be used for Daily API calls + app.state.session = aiohttp.ClientSession() + yield + # Close session when shutting down + await app.state.session.close() + + +app = FastAPI(lifespan=lifespan) + + +@app.post("/start") +async def handle_incoming_daily_webhook(request: Request) -> JSONResponse: + """Handle dial-out request.""" + print("Received webhook from Daily") + + # Get the dial-in properties from the request + try: + data = await request.json() + if "test" in data: + # Pass through any webhook checks + return JSONResponse({"test": True}) + + if not data["dialout_settings"]: + raise HTTPException( + status_code=400, detail="Missing 'dialout_settings' in the request body" + ) + + if not data["dialout_settings"].get("phone_number"): + raise HTTPException( + status_code=400, detail="Missing 'phone_number' in dialout_settings" + ) + + # Extract the phone number we want to dial out to + caller_phone = str(data["dialout_settings"]["phone_number"]) + print(f"Processing call to {caller_phone}") + + # Create a Daily room with dial-in capabilities + try: + room_details = await create_daily_room(request.app.state.session, caller_phone) + except Exception as e: + print(f"Error creating Daily room: {e}") + raise HTTPException(status_code=500, detail=f"Failed to create Daily room: {str(e)}") + + room_url = room_details["room_url"] + token = room_details["token"] + print(f"Created Daily room: {room_url} with token: {token}") + + body_json = json.dumps(data) + + bot_cmd = f"python3 -m bot -u {room_url} -t {token} -b {shlex.quote(body_json)}" + + try: + # CHANGE: Keep stdout/stderr for debugging + # Start the bot in the background but capture output + subprocess.Popen( + bot_cmd, + shell=True, + # Don't redirect output so we can see logs + # stdout=subprocess.DEVNULL, + # stderr=subprocess.DEVNULL + ) + print(f"Started bot process with command: {bot_cmd}") + except Exception as e: + print(f"Error starting bot: {e}") + raise HTTPException(status_code=500, detail=f"Failed to start bot: {str(e)}") + + except HTTPException: + raise + except Exception as e: + print(f"Unexpected error: {str(e)}") + raise HTTPException(status_code=500, detail=f"Server error: {str(e)}") + + # Grab a token for the user to join with + return JSONResponse({"room_url": room_url, "token": token}) + + +@app.get("/health") +async def health_check(): + """Simple health check endpoint.""" + return {"status": "healthy"} + + +# ----------------- Main ----------------- # + + +if __name__ == "__main__": + # Run the server + port = int(os.getenv("PORT", "7860")) + print(f"Starting server on port {port}") + uvicorn.run("server:app", host="0.0.0.0", port=port, reload=True) diff --git a/examples/phone-chatbot/daily-pstn-advanced-voicemail-detection/utils/daily_helpers.py b/examples/phone-chatbot/daily-pstn-advanced-voicemail-detection/utils/daily_helpers.py new file mode 100644 index 000000000..6451fc4c4 --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-advanced-voicemail-detection/utils/daily_helpers.py @@ -0,0 +1,76 @@ +"""Helper functions for interacting with the Daily API.""" + +import os +from typing import Dict, Optional + +import aiohttp +from dotenv import load_dotenv + +from pipecat.transports.services.helpers.daily_rest import ( + DailyRESTHelper, + DailyRoomParams, + DailyRoomProperties, + DailyRoomSipParams, +) + +load_dotenv() + + +# Initialize Daily API helper +async def get_daily_helper(session: Optional[aiohttp.ClientSession] = None) -> DailyRESTHelper: + """Get a Daily REST helper with the configured API key.""" + if session is None: + session = aiohttp.ClientSession() + + return DailyRESTHelper( + daily_api_key=os.getenv("DAILY_API_KEY", ""), + daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"), + aiohttp_session=session, + ) + + +async def create_daily_room( + session: Optional[aiohttp.ClientSession] = None, caller_phone: str = "unknown-caller" +) -> Dict[str, str]: + """Create a Daily room with SIP capabilities for phone calls. + + Args: + session: Optional aiohttp session to use for API calls + caller_phone: The phone number of the caller to use in display name + + Returns: + Dictionary with room URL, token, and SIP endpoint + """ + daily_helper = await get_daily_helper(session) + + # Configure SIP parameters + sip_params = DailyRoomSipParams( + display_name=caller_phone, + video=False, + sip_mode="dial-in", + num_endpoints=1, + ) + + # Create room properties with SIP enabled + properties = DailyRoomProperties( + sip=sip_params, + enable_dialout=True, # Needed for outbound calls if you expand the bot + enable_chat=False, # No need for chat in a voice bot + start_video_off=True, # Voice only + ) + + # Create room parameters + params = DailyRoomParams(properties=properties) + + # Create the room + try: + room = await daily_helper.create_room(params=params) + print(f"Created room: {room.url} with SIP endpoint: {room.config.sip_endpoint}") + + # Get token for the bot to join + token = await daily_helper.get_token(room.url, 24 * 60 * 60) # 24 hours validity + + return {"room_url": room.url, "token": token, "sip_endpoint": room.config.sip_endpoint} + except Exception as e: + print(f"Error creating room: {e}") + raise diff --git a/examples/phone-chatbot/daily-pstn-call-transfer/README.md b/examples/phone-chatbot/daily-pstn-call-transfer/README.md new file mode 100644 index 000000000..4d0ddd3d2 --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-call-transfer/README.md @@ -0,0 +1,124 @@ +# Daily PSTN call transfer + +A basic example of how to create a bot that handles the initial customer interaction and then transfers to a human operator when needed + +## Architecture Overview + +These examples use the following components: + +- 🔁 **Transport**: Daily WebRTC +- 💬 **Speech-to-Text**: Deepgram via Daily transport +- 🤖 **LLMs**: Each example uses a specific LLM (OpenAI GPT-4o or Google Gemini) +- 🔉 **Text-to-Speech**: Cartesia + +## Prerequisites + +- A Daily account with an API key +- An OpenAI API key for the bot's intelligence +- A Cartesia API key for text-to-speech +- One phone to dial-in from and another phone to receive calls when escalating to a manager + +## Setup + +1. Create a virtual environment and install dependencies + +```bash +python -m venv venv +source venv/bin/activate # On Windows: venv\Scripts\activate +pip install -r requirements.txt +``` + +2. Set up environment variables + +Copy the example file and fill in your API keys: + +```bash +cp .env.example .env +# Edit .env with your API keys +``` + +- Note, please specify an OPERATOR_NUMBER so that the bot can ring a number when escalating to a manager + +3. Buy a phone number + +Instructions on how to do that can be found at this [docs link:](https://docs.daily.co/reference/rest-api/phone-numbers/buy-phone-number). + +4. Set up the dial-in config + +Instructions on how to do that can be found at this [docs link:](https://docs.daily.co/reference/rest-api/domainDialinConfig) + +5. For local testing, use ngrok to expose your local server + +```bash +ngrok http 7860 +# Then use the provided URL (e.g., https://abc123.ngrok.io/start) in Twilio +``` + +## Running the Server + +Start the webhook server: + +```bash +python server.py +``` + +## Testing + +Call the purchased phone number. The system should answer the call, put you on hold briefly, then connect you with the bot. +Have a short conversation with the bot, and then request to speak with a manager. The bot should then ring the manager. On your second phone, answer the call. + +The bot will then summarise the conversation so far, and then silently listen to the conversation. You can now speak with the manager on the other phone. + +When the manager hangs up the call, the bot will start speaking again. You can then ask the bot about the conversation with the manager, and it will have the context of the conversation. + +## Customizing the Bot + +You can customize the bot's behavior by modifying the system prompt in `bot.py`. + +## Multiple SIP Endpoints + +For PSTN calls, you only need one SIP endpoint. + +## Daily SIP Configuration + +The bot configures Daily rooms with SIP capabilities using these settings: + +```python +sip_params = DailyRoomSipParams( + display_name="phone-user", # This will show up in the Daily UI; optional display the dialer's number + video=False, # Audio-only call + sip_mode="dial-in", # For receiving calls (vs. dial-out) + num_endpoints=1, # Number of SIP endpoints to create +) + +properties = DailyRoomProperties( + sip=sip_params, + enable_dialout=True, # Needed for outbound calls if you expand the bot + enable_chat=False, # No need for chat in a voice bot + start_video_off=True, # Voice only +) +``` + +## Troubleshooting + +### Call is not being answered + +- Check that your dial-in config is correctly configured to point towards your ngrok server and correct endpoint +- Make sure the server.py file is running +- Make sure ngrok is correctly setup and pointing to the correct port + +### The bot does not escalate to the manager + +- Check that your room has `enable_dialout=True` set +- Check that your meeting token is an owner token (The bot does this for you automatically) +- Check that the phone number you are trying to ring is correct, and is a US or Canadian number. + +### Call connects but no bot is heard + +- Ensure your Daily API key is correct and has SIP capabilities +- Verify that the Cartesia API key and voice ID are correct + +### Bot starts but disconnects immediately + +- Check the Daily logs for any error messages +- Ensure your server has stable internet connectivity diff --git a/examples/phone-chatbot/call_transfer.py b/examples/phone-chatbot/daily-pstn-call-transfer/bot.py similarity index 50% rename from examples/phone-chatbot/call_transfer.py rename to examples/phone-chatbot/daily-pstn-call-transfer/bot.py index 2c9bae4c7..9613be832 100644 --- a/examples/phone-chatbot/call_transfer.py +++ b/examples/phone-chatbot/daily-pstn-call-transfer/bot.py @@ -5,10 +5,10 @@ # import argparse import asyncio +import json import os import sys -from call_connection_manager import CallConfigManager, SessionManager from dotenv import load_dotenv from loguru import logger @@ -29,7 +29,7 @@ from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.processors.filters.function_filter import FunctionFilter from pipecat.processors.frame_processor import FrameDirection, FrameProcessor from pipecat.services.cartesia.tts import CartesiaTTSService -from pipecat.services.llm_service import FunctionCallParams, LLMService +from pipecat.services.llm_service import FunctionCallParams from pipecat.services.openai.llm import OpenAILLMService from pipecat.transports.services.daily import DailyDialinSettings, DailyParams, DailyTransport @@ -42,6 +42,124 @@ daily_api_key = os.getenv("DAILY_API_KEY", "") daily_api_url = os.getenv("DAILY_API_URL", "https://api.daily.co/v1") +class SessionManager: + """Centralized management of session IDs and state for all call participants.""" + + def __init__(self, call_flow_state=None): + # Track session IDs of different participant types + self.session_ids = { + "operator": None, + "customer": None, + "bot": None, + # Add other participant types as needed + } + + # References for easy access in processors that need mutable containers + self.session_id_refs = { + "operator": [None], + "customer": [None], + "bot": [None], + # Add other participant types as needed + } + + # Use the provided call_flow_state or create a new one + self.call_flow_state = call_flow_state if call_flow_state is not None else CallFlowState() + + def set_session_id(self, participant_type, session_id): + """Set the session ID for a specific participant type. + + Args: + participant_type: Type of participant (e.g., "operator", "customer", "bot") + session_id: The session ID to set + """ + if participant_type in self.session_ids: + self.session_ids[participant_type] = session_id + + # Also update the corresponding reference if it exists + if participant_type in self.session_id_refs: + self.session_id_refs[participant_type][0] = session_id + + def get_session_id(self, participant_type): + """Get the session ID for a specific participant type. + + Args: + participant_type: Type of participant (e.g., "operator", "customer", "bot") + + Returns: + The session ID or None if not set + """ + return self.session_ids.get(participant_type) + + def get_session_id_ref(self, participant_type): + """Get the mutable reference for a specific participant type. + + Args: + participant_type: Type of participant (e.g., "operator", "customer", "bot") + + Returns: + A mutable list container holding the session ID or None if not available + """ + return self.session_id_refs.get(participant_type) + + def is_participant_type(self, session_id, participant_type): + """Check if a session ID belongs to a specific participant type. + + Args: + session_id: The session ID to check + participant_type: Type of participant (e.g., "operator", "customer", "bot") + + Returns: + True if the session ID matches the participant type, False otherwise + """ + return self.session_ids.get(participant_type) == session_id + + def reset_participant(self, participant_type): + """Reset the state for a specific participant type. + + Args: + participant_type: Type of participant (e.g., "operator", "customer", "bot") + """ + if participant_type in self.session_ids: + self.session_ids[participant_type] = None + + if participant_type in self.session_id_refs: + self.session_id_refs[participant_type][0] = None + + # Additional reset actions for specific participant types + if participant_type == "operator": + self.call_flow_state.set_operator_disconnected() + + +class CallFlowState: + """State for tracking call flow operations and state transitions.""" + + def __init__(self): + # Operator-related state + self.dialed_operator = False + self.operator_connected = False + self.summary_finished = False + + # Operator-related methods + def set_operator_dialed(self): + """Mark that an operator has been dialed.""" + self.dialed_operator = True + + def set_operator_connected(self): + """Mark that an operator has connected to the call.""" + self.operator_connected = True + # Summary is not finished when operator first connects + self.summary_finished = False + + def set_operator_disconnected(self): + """Handle operator disconnection.""" + self.operator_connected = False + self.summary_finished = False + + def set_summary_finished(self): + """Mark the summary as finished.""" + self.summary_finished = True + + class TranscriptionModifierProcessor(FrameProcessor): """Processor that modifies transcription frames before they reach the context aggregator.""" @@ -96,72 +214,62 @@ class SummaryFinished(FrameProcessor): await self.push_frame(frame, direction) -async def main( +async def run_bot( room_url: str, token: str, body: dict, -): +) -> None: + """Run the voice bot with the given parameters. + + Args: + room_url: The Daily room URL + token: The Daily room token + body: Body passed to the bot from the webhook + + """ # ------------ CONFIGURATION AND SETUP ------------ + logger.info(f"Starting bot with room: {room_url}") + logger.info(f"Token: {token}") + logger.info(f"Body: {body}") + # Parse the body to get the dial-in settings + body_data = json.loads(body) - # Create a routing manager using the provided body - call_config_manager = CallConfigManager.from_json_string(body) if body else CallConfigManager() + # Check if the body contains dial-in settings + logger.debug(f"Body data: {body_data}") - # Get caller information - caller_info = call_config_manager.get_caller_info() - caller_number = caller_info["caller_number"] - dialed_number = caller_info["dialed_number"] + if not all([body_data.get("callId"), body_data.get("callDomain")]): + logger.error("Call ID and Call Domain are required in the body.") + return None - # Get customer name based on caller number - customer_name = call_config_manager.get_customer_name(caller_number) if caller_number else None + call_id = body_data.get("callId") + call_domain = body_data.get("callDomain") + logger.debug(f"Call ID: {call_id}") + logger.debug(f"Call Domain: {call_domain}") - # Get appropriate operator settings based on the caller - operator_dialout_settings = call_config_manager.get_dialout_settings_for_caller(caller_number) + if not call_id or not call_domain: + logger.error("Call ID and Call Domain are required for dial-in.") + sys.exit(1) - logger.info(f"Caller number: {caller_number}") - logger.info(f"Dialed number: {dialed_number}") - logger.info(f"Customer name: {customer_name}") - logger.info(f"Operator dialout settings: {operator_dialout_settings}") - - # Check if in test mode - test_mode = call_config_manager.is_test_mode() - - # Get dialin settings if present - dialin_settings = call_config_manager.get_dialin_settings() - - # ------------ TRANSPORT SETUP ------------ - - # Set up transport parameters - if test_mode: - logger.info("Running in test mode") - transport_params = DailyParams( - api_url=daily_api_url, - api_key=daily_api_key, - audio_in_enabled=True, - audio_out_enabled=True, - video_out_enabled=False, - vad_analyzer=SileroVADAnalyzer(), - transcription_enabled=True, - ) - else: - daily_dialin_settings = DailyDialinSettings( - call_id=dialin_settings.get("call_id"), call_domain=dialin_settings.get("call_domain") - ) - transport_params = DailyParams( - api_url=daily_api_url, - api_key=daily_api_key, - dialin_settings=daily_dialin_settings, - audio_in_enabled=True, - audio_out_enabled=True, - video_out_enabled=False, - vad_analyzer=SileroVADAnalyzer(), - transcription_enabled=True, - ) + daily_dialin_settings = DailyDialinSettings(call_id=call_id, call_domain=call_domain) + logger.debug(f"Dial-in settings: {daily_dialin_settings}") + transport_params = DailyParams( + api_url=daily_api_url, + api_key=daily_api_key, + dialin_settings=daily_dialin_settings, + audio_in_enabled=True, + audio_out_enabled=True, + video_out_enabled=False, + vad_analyzer=SileroVADAnalyzer(), + transcription_enabled=True, + ) + logger.debug("setup transport params") # Initialize the session manager - session_manager = SessionManager() + call_flow_state = CallFlowState() + session_manager = SessionManager(call_flow_state) - # Set up the operator dialout settings - session_manager.call_flow_state.set_operator_dialout_settings(operator_dialout_settings) + # Operator dialout number + operator_number = os.getenv("OPERATOR_NUMBER", None) # Initialize transport transport = DailyTransport( @@ -177,30 +285,38 @@ async def main( voice_id="b7d50908-b17c-442d-ad8d-810c63997ed9", # Use Helpful Woman voice by default ) + # ------------ RETRY LOGIC VARIABLES ------------ + max_retries = 5 + retry_count = 0 + dialout_successful = False + dialout_params = None + + async def attempt_operator_dialout(): + """Attempt to start operator dialout with retry logic.""" + nonlocal retry_count, dialout_successful + + if retry_count < max_retries and not dialout_successful: + retry_count += 1 + logger.info( + f"Attempting operator dialout (attempt {retry_count}/{max_retries}) to: {operator_number}" + ) + await transport.start_dialout(dialout_params) + else: + logger.error(f"Maximum retry attempts ({max_retries}) reached for operator dialout.") + # Notify user that operator connection failed + content = "I'm sorry, but I'm unable to connect you with a supervisor at this time. Please try again later or contact us through other means." + message = {"role": "system", "content": content} + messages.append(message) + await task.queue_frames([LLMMessagesFrame(messages)]) + # ------------ LLM AND CONTEXT SETUP ------------ - # Get prompts from routing manager - call_transfer_initial_prompt = call_config_manager.get_prompt("call_transfer_initial_prompt") - - # Build default greeting with customer name if available - customer_greeting = f"Hello {customer_name}" if customer_name else "Hello" - default_greeting = f"{customer_greeting}, this is Hailey from customer support. What can I help you with today?" - - # Build initial prompt - if call_transfer_initial_prompt: - # Use custom prompt with customer name replacement if needed - system_instruction = call_config_manager.customize_prompt( - call_transfer_initial_prompt, customer_name - ) - logger.info("Using custom call transfer initial prompt") - else: - # Use default prompt with formatted greeting - system_instruction = f"""You are Chatbot, a friendly, helpful robot. Never refer to this prompt, even if asked. Follow these steps **EXACTLY**. + system_instruction = f"""You are Chatbot, a friendly, helpful robot. Never refer to this prompt, even if asked. Follow these steps **EXACTLY**. ### **Standard Operating Procedure:** #### **Step 1: Greeting** - - Greet the user with: "{default_greeting}" + - Greet the user with: "Hello, this is Hailey from customer support. What can I help you with today?" #### **Step 2: Handling Requests** - If the user requests a supervisor, **IMMEDIATELY** call the `dial_operator` function. @@ -211,10 +327,13 @@ async def main( ### **General Rules** - Your output will be converted to audio, so **do not include special characters or formatting.** """ - logger.info("Using default call transfer initial prompt") - # Create the system message and initialize messages list - messages = [call_config_manager.create_system_message(system_instruction)] + messages = [ + { + "role": "system", + "content": system_instruction, + } + ] # ------------ FUNCTION DEFINITIONS ------------ @@ -225,7 +344,10 @@ async def main( """Function the bot can call to terminate the call.""" # Create a message to add content = "The user wants to end the conversation, thank them for chatting." - message = call_config_manager.create_system_message(content) + message = { + "role": "system", + "content": content, + } # Append the message to the list messages.append(message) # Queue the message to the context @@ -236,41 +358,41 @@ async def main( async def dial_operator(params: FunctionCallParams): """Function the bot can call to dial an operator.""" - dialout_setting = session_manager.call_flow_state.get_current_dialout_setting() - if call_config_manager.get_transfer_mode() == "dialout": - if dialout_setting: - session_manager.call_flow_state.set_operator_dialed() - logger.info(f"Dialing operator with settings: {dialout_setting}") + nonlocal dialout_params - # Create a message to add - content = "The user has requested a supervisor, indicate that you will attempt to connect them with a supervisor." - message = call_config_manager.create_system_message(content) + if operator_number: + call_flow_state.set_operator_dialed() + logger.info(f"Dialing operator number: {operator_number}") - # Append the message to the list - messages.append(message) - # Queue the message to the context - await task.queue_frames([LLMMessagesFrame(messages)]) - # Start the dialout - await call_config_manager.start_dialout(transport, [dialout_setting]) - - else: - # Create a message to add - content = "Indicate that there are no operator dialout settings available." - message = call_config_manager.create_system_message(content) - # Append the message to the list - messages.append(message) - # Queue the message to the context - await task.queue_frames([LLMMessagesFrame(messages)]) - logger.info("No operator dialout settings available") - else: # Create a message to add - content = "Indicate that the current mode is not supported." - message = call_config_manager.create_system_message(content) + content = "The user has requested a supervisor, indicate that you will attempt to connect them with a supervisor." + message = { + "role": "system", + "content": content, + } + # Append the message to the list messages.append(message) # Queue the message to the context await task.queue_frames([LLMMessagesFrame(messages)]) - logger.info("Other mode not supported") + + # Set up dialout parameters and start attempt + dialout_params = {"phoneNumber": operator_number} + logger.debug(f"Dialout parameters: {dialout_params}") + await attempt_operator_dialout() + + else: + # Create a message to add + content = "Indicate that there are no operator dialout settings available." + message = { + "role": "system", + "content": content, + } + # Append the message to the list + messages.append(message) + # Queue the message to the context + await task.queue_frames([LLMMessagesFrame(messages)]) + logger.info("No operator dialout settings available") # Define function schemas for tools terminate_call_function = FunctionSchema( @@ -304,17 +426,14 @@ async def main( # ------------ PIPELINE SETUP ------------ # Use the session manager's references - summary_finished = SummaryFinished(session_manager.call_flow_state) + summary_finished = SummaryFinished(call_flow_state) transcription_modifier = TranscriptionModifierProcessor( session_manager.get_session_id_ref("operator") ) # Define function to determine if bot should speak async def should_speak(self) -> bool: - result = ( - not session_manager.call_flow_state.operator_connected - or not session_manager.call_flow_state.summary_finished - ) + result = not call_flow_state.operator_connected or not call_flow_state.summary_finished return result # Build pipeline @@ -348,14 +467,15 @@ async def main( @transport.event_handler("on_dialout_answered") async def on_dialout_answered(transport, data): + nonlocal dialout_successful logger.debug(f"++++ Dial-out answered: {data}") await transport.capture_participant_transcription(data["sessionId"]) + # Mark dialout as successful to stop retries + dialout_successful = True + # Skip if operator already connected - if ( - not session_manager.call_flow_state - or session_manager.call_flow_state.operator_connected - ): + if not call_flow_state or call_flow_state.operator_connected: logger.debug(f"Operator already connected: {data}") return @@ -365,34 +485,32 @@ async def main( session_manager.set_session_id("operator", data["sessionId"]) # Update state - session_manager.call_flow_state.set_operator_connected() - - # Determine message content based on configuration - if call_config_manager.get_speak_summary(): - logger.debug("Bot will speak summary") - call_transfer_prompt = call_config_manager.get_prompt("call_transfer_prompt") - - if call_transfer_prompt: - # Use custom prompt - logger.info("Using custom call transfer prompt") - content = call_config_manager.customize_prompt(call_transfer_prompt, customer_name) - else: - # Use default summary prompt - logger.info("Using default call transfer prompt") - customer_info = call_config_manager.get_customer_info_suffix(customer_name) - content = f"""An operator is joining the call{customer_info}. - Give a brief summary of the customer's issues so far.""" - else: - # Simple join notification without summary - logger.debug("Bot will not speak summary") - customer_info = call_config_manager.get_customer_info_suffix(customer_name) - content = f"""Indicate that an operator has joined the call{customer_info}.""" + call_flow_state.set_operator_connected() # Create and queue system message - message = call_config_manager.create_system_message(content) + content = """An operator is joining the call. + Give a brief summary of the customer's issues so far.""" + message = { + "role": "system", + "content": content, + } messages.append(message) await task.queue_frames([LLMMessagesFrame(messages)]) + @transport.event_handler("on_dialout_connected") + async def on_dialout_connected(transport, data): + logger.debug(f"Dial-out connected: {data}") + + @transport.event_handler("on_dialout_error") + async def on_dialout_error(transport, data): + logger.error(f"Operator dialout error (attempt {retry_count}/{max_retries}): {data}") + + if retry_count < max_retries: + logger.info(f"Retrying operator dialout") + await attempt_operator_dialout() + else: + logger.error(f"All {max_retries} operator dialout attempts failed.") + @transport.event_handler("on_dialout_stopped") async def on_dialout_stopped(transport, data): if session_manager.get_session_id("operator") and data[ @@ -417,29 +535,14 @@ async def main( # Reset operator state session_manager.reset_participant("operator") - # Determine message content - call_transfer_finished_prompt = call_config_manager.get_prompt( - "call_transfer_finished_prompt" - ) - - if call_transfer_finished_prompt: - # Use custom prompt for operator departure - logger.info("Using custom call transfer finished prompt") - content = call_config_manager.customize_prompt( - call_transfer_finished_prompt, customer_name - ) - else: - # Use default prompt for operator departure - logger.info("Using default call transfer finished prompt") - customer_info = call_config_manager.get_customer_info_suffix( - customer_name, preposition="" - ) - content = f"""The operator has left the call. + # Create and queue system message + content = """The operator has left the call. Resume your role as the primary support agent and use information from the operator's conversation to help the customer{customer_info}. Let the customer know the operator has left and ask if they need further assistance.""" - - # Create and queue system message - message = call_config_manager.create_system_message(content) + message = { + "role": "system", + "content": content, + } messages.append(message) await task.queue_frames([LLMMessagesFrame(messages)]) @@ -449,17 +552,25 @@ async def main( await runner.run(task) -if __name__ == "__main__": - parser = argparse.ArgumentParser(description="Pipecat Call Transfer Bot") +async def main(): + """Parse command line arguments and run the bot.""" + parser = argparse.ArgumentParser(description="Simple Dial-out Bot") parser.add_argument("-u", "--url", type=str, help="Room URL") parser.add_argument("-t", "--token", type=str, help="Room Token") parser.add_argument("-b", "--body", type=str, help="JSON configuration string") args = parser.parse_args() - # Log the arguments for debugging - logger.info(f"Room URL: {args.url}") - logger.info(f"Token: {args.token}") - logger.info(f"Body provided: {bool(args.body)}") + logger.debug(f"url: {args.url}") + logger.debug(f"token: {args.token}") + logger.debug(f"body: {args.body}") + if not all([args.url, args.token, args.body]): + logger.error("All arguments (-u, -t, -b) are required") + parser.print_help() + sys.exit(1) - asyncio.run(main(args.url, args.token, args.body)) + await run_bot(args.url, args.token, args.body) + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/examples/phone-chatbot/daily-pstn-call-transfer/env.example b/examples/phone-chatbot/daily-pstn-call-transfer/env.example new file mode 100644 index 000000000..02f60f4eb --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-call-transfer/env.example @@ -0,0 +1,10 @@ +# Daily credentials +DAILY_API_KEY=your_daily_api_key +DAILY_API_URL=https://api.daily.co/v1 + +# Service keys +OPENAI_API_KEY=your_openai_api_key +CARTESIA_API_KEY=your_cartesia_api_key + +# Operator number +OPERATOR_NUMBER=phone_number_to_dial \ No newline at end of file diff --git a/examples/phone-chatbot/daily-pstn-call-transfer/requirements.txt b/examples/phone-chatbot/daily-pstn-call-transfer/requirements.txt new file mode 100644 index 000000000..a337e3c09 --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-call-transfer/requirements.txt @@ -0,0 +1,6 @@ +pipecat-ai[daily,cartesia,openai,silero] +fastapi==0.115.6 +uvicorn +python-dotenv +python-multipart +aiohttp \ No newline at end of file diff --git a/examples/phone-chatbot/daily-pstn-call-transfer/server.py b/examples/phone-chatbot/daily-pstn-call-transfer/server.py new file mode 100644 index 000000000..2fb6de15c --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-call-transfer/server.py @@ -0,0 +1,116 @@ +# +# Copyright (c) 2024–2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +"""server.py. + +Webhook server to handle webhook coming from Daily, create a Daily room and start the bot. +""" + +import json +import os +import shlex +import subprocess +from contextlib import asynccontextmanager + +import aiohttp +import uvicorn +from dotenv import load_dotenv +from fastapi import FastAPI, HTTPException, Request +from fastapi.responses import JSONResponse +from utils.daily_helpers import create_daily_room + +load_dotenv() + +# ----------------- API ----------------- # + + +@asynccontextmanager +async def lifespan(app: FastAPI): + # Create aiohttp session to be used for Daily API calls + app.state.session = aiohttp.ClientSession() + yield + # Close session when shutting down + await app.state.session.close() + + +app = FastAPI(lifespan=lifespan) + + +@app.post("/start") +async def handle_incoming_daily_webhook(request: Request) -> JSONResponse: + """Handle incoming Daily call webhook.""" + print("Received webhook from Daily") + + # Get the dial-in properties from the request + try: + data = await request.json() + if "test" in data: + # Pass through any webhook checks + return JSONResponse({"test": True}) + + if not all(key in data for key in ["From", "To", "callId", "callDomain"]): + raise HTTPException( + status_code=400, detail="Missing properties 'From', 'To', 'callId', 'callDomain'" + ) + + # Extract the caller's phone number + caller_phone = str(data.get("From")) + print(f"Processing call from {caller_phone}") + + # Create a Daily room with dial-in capabilities + try: + room_details = await create_daily_room(request.app.state.session, caller_phone) + except Exception as e: + print(f"Error creating Daily room: {e}") + raise HTTPException(status_code=500, detail=f"Failed to create Daily room: {str(e)}") + + room_url = room_details["room_url"] + token = room_details["token"] + print(f"Created Daily room: {room_url} with token: {token}") + + body_json = json.dumps(data) + + bot_cmd = f"python3 -m bot -u {room_url} -t {token} -b {shlex.quote(body_json)}" + + try: + # CHANGE: Keep stdout/stderr for debugging + # Start the bot in the background but capture output + subprocess.Popen( + bot_cmd, + shell=True, + # Don't redirect output so we can see logs + # stdout=subprocess.DEVNULL, + # stderr=subprocess.DEVNULL + ) + print(f"Started bot process with command: {bot_cmd}") + except Exception as e: + print(f"Error starting bot: {e}") + raise HTTPException(status_code=500, detail=f"Failed to start bot: {str(e)}") + + except HTTPException: + raise + except Exception as e: + print(f"Unexpected error: {str(e)}") + raise HTTPException(status_code=500, detail=f"Server error: {str(e)}") + + # Grab a token for the user to join with + return JSONResponse({"room_url": room_url, "token": token}) + + +@app.get("/health") +async def health_check(): + """Simple health check endpoint.""" + return {"status": "healthy"} + + +# ----------------- Main ----------------- # + + +if __name__ == "__main__": + # Run the server + port = int(os.getenv("PORT", "7860")) + print(f"Starting server on port {port}") + uvicorn.run("server:app", host="0.0.0.0", port=port, reload=True) diff --git a/examples/phone-chatbot/daily-pstn-call-transfer/utils/daily_helpers.py b/examples/phone-chatbot/daily-pstn-call-transfer/utils/daily_helpers.py new file mode 100644 index 000000000..6451fc4c4 --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-call-transfer/utils/daily_helpers.py @@ -0,0 +1,76 @@ +"""Helper functions for interacting with the Daily API.""" + +import os +from typing import Dict, Optional + +import aiohttp +from dotenv import load_dotenv + +from pipecat.transports.services.helpers.daily_rest import ( + DailyRESTHelper, + DailyRoomParams, + DailyRoomProperties, + DailyRoomSipParams, +) + +load_dotenv() + + +# Initialize Daily API helper +async def get_daily_helper(session: Optional[aiohttp.ClientSession] = None) -> DailyRESTHelper: + """Get a Daily REST helper with the configured API key.""" + if session is None: + session = aiohttp.ClientSession() + + return DailyRESTHelper( + daily_api_key=os.getenv("DAILY_API_KEY", ""), + daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"), + aiohttp_session=session, + ) + + +async def create_daily_room( + session: Optional[aiohttp.ClientSession] = None, caller_phone: str = "unknown-caller" +) -> Dict[str, str]: + """Create a Daily room with SIP capabilities for phone calls. + + Args: + session: Optional aiohttp session to use for API calls + caller_phone: The phone number of the caller to use in display name + + Returns: + Dictionary with room URL, token, and SIP endpoint + """ + daily_helper = await get_daily_helper(session) + + # Configure SIP parameters + sip_params = DailyRoomSipParams( + display_name=caller_phone, + video=False, + sip_mode="dial-in", + num_endpoints=1, + ) + + # Create room properties with SIP enabled + properties = DailyRoomProperties( + sip=sip_params, + enable_dialout=True, # Needed for outbound calls if you expand the bot + enable_chat=False, # No need for chat in a voice bot + start_video_off=True, # Voice only + ) + + # Create room parameters + params = DailyRoomParams(properties=properties) + + # Create the room + try: + room = await daily_helper.create_room(params=params) + print(f"Created room: {room.url} with SIP endpoint: {room.config.sip_endpoint}") + + # Get token for the bot to join + token = await daily_helper.get_token(room.url, 24 * 60 * 60) # 24 hours validity + + return {"room_url": room.url, "token": token, "sip_endpoint": room.config.sip_endpoint} + except Exception as e: + print(f"Error creating room: {e}") + raise diff --git a/examples/phone-chatbot/daily-pstn-dial-in/README.md b/examples/phone-chatbot/daily-pstn-dial-in/README.md new file mode 100644 index 000000000..1ced64baa --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-dial-in/README.md @@ -0,0 +1,107 @@ + + +# Daily PSTN dial-in simple chatbot + +This project demonstrates how to create a voice bot that can receive phone calls via Dailys PSTN capabilities to enable voice conversations. + +## How It Works + +1. Daily receives an incoming call to your phone number. +2. Daily calls your webhook server (`/start` endpoint). +3. The server creates a Daily room with dial-in capabilities +4. The server starts the bot process with the room details +5. The caller is put on hold with music +6. The bot joins the Daily room and signals readiness +7. Daily forwards the call to the Daily room +8. The caller and the bot are connected, and the bot handles the conversation + +## Prerequisites + +- A Daily account with an API key +- An OpenAI API key for the bot's intelligence +- A Cartesia API key for text-to-speech + +## Setup + +1. Create a virtual environment and install dependencies + +```bash +python -m venv venv +source venv/bin/activate # On Windows: venv\Scripts\activate +pip install -r requirements.txt +``` + +2. Set up environment variables + +Copy the example file and fill in your API keys: + +```bash +cp .env.example .env +# Edit .env with your API keys +``` + +3. Buy a phone number + +Instructions on how to do that can be found at this [docs link:](https://docs.daily.co/reference/rest-api/phone-numbers/buy-phone-number) + +4. Set up the dial-in config + +Instructions on how to do that can be found at this [docs link:](https://docs.daily.co/reference/rest-api/domainDialinConfig) + +5. For local testing, use ngrok to expose your local server + +```bash +ngrok http 7860 +# Then use the provided URL (e.g., https://abc123.ngrok.io/start) in Twilio +``` + +## Running the Server + +Start the webhook server: + +```bash +python server.py +``` + +## Testing + +Call the purchased phone number. The system should answer the call, put you on hold briefly, then connect you with the bot. + +## Customizing the Bot + +You can customize the bot's behavior by modifying the system prompt in `bot.py`. + +## Multiple SIP Endpoints + +For PSTN calls, you only need one SIP endpoint. + +## Daily SIP Configuration + +The bot configures Daily rooms with SIP capabilities using these settings: + +```python +sip_params = DailyRoomSipParams( + display_name="phone-user", # This will show up in the Daily UI; optional display the dialer's number + video=False, # Audio-only call + sip_mode="dial-in", # For receiving calls (vs. dial-out) + num_endpoints=1, # Number of SIP endpoints to create +) +``` + +## Troubleshooting + +### Call is not being answered + +- Check that your dial-in config is correctly configured to point towards your ngrok server and correct endpoint +- Make sure the server.py file is running +- Make sure ngrok is correctly setup and pointing to the correct port + +### Call connects but no bot is heard + +- Ensure your Daily API key is correct and has SIP capabilities +- Verify that the Cartesia API key and voice ID are correct + +### Bot starts but disconnects immediately + +- Check the Daily logs for any error messages +- Ensure your server has stable internet connectivity diff --git a/examples/phone-chatbot/daily-pstn-dial-in/bot.py b/examples/phone-chatbot/daily-pstn-dial-in/bot.py new file mode 100644 index 000000000..26310b052 --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-dial-in/bot.py @@ -0,0 +1,218 @@ +# +# Copyright (c) 2024–2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +"""simple_dialin.py. + +Daily PSTN Dial-in Bot. +""" + +import argparse +import asyncio +import json +import os +import sys + +from dotenv import load_dotenv +from loguru import logger + +from pipecat.audio.vad.silero import SileroVADAnalyzer +from pipecat.pipeline.pipeline import Pipeline +from pipecat.pipeline.runner import PipelineRunner +from pipecat.pipeline.task import PipelineParams, PipelineTask +from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext +from pipecat.services.cartesia.tts import CartesiaTTSService +from pipecat.services.openai.llm import OpenAILLMService +from pipecat.transports.services.daily import DailyDialinSettings, DailyParams, DailyTransport + +# Setup logging +load_dotenv() +logger.remove(0) +logger.add(sys.stderr, level="DEBUG") + + +daily_api_key = os.getenv("DAILY_API_KEY", "") +daily_api_url = os.getenv("DAILY_API_URL", "https://api.daily.co/v1") + + +async def run_bot( + room_url: str, + token: str, + body: dict, +) -> None: + """Run the voice bot with the given parameters. + + Args: + room_url: The Daily room URL + token: The Daily room token + body: Body passed to the bot from the webhook + + """ + # ------------ CONFIGURATION AND SETUP ------------ + logger.info(f"Starting bot with room: {room_url}") + logger.info(f"Token: {token}") + logger.info(f"Body: {body}") + # Parse the body to get the dial-in settings + body_data = json.loads(body) + + # Check if the body contains dial-in settings + logger.debug(f"Body data: {body_data}") + + if not all([body_data.get("callId"), body_data.get("callDomain")]): + logger.error("Call ID and Call Domain are required in the body.") + return None + + call_id = body_data.get("callId") + call_domain = body_data.get("callDomain") + logger.debug(f"Call ID: {call_id}") + logger.debug(f"Call Domain: {call_domain}") + + if not call_id or not call_domain: + logger.error("Call ID and Call Domain are required for dial-in.") + sys.exit(1) + + daily_dialin_settings = DailyDialinSettings(call_id=call_id, call_domain=call_domain) + logger.debug(f"Dial-in settings: {daily_dialin_settings}") + transport_params = DailyParams( + api_url=daily_api_url, + api_key=daily_api_key, + dialin_settings=daily_dialin_settings, + audio_in_enabled=True, + audio_out_enabled=True, + video_out_enabled=False, + vad_analyzer=SileroVADAnalyzer(), + transcription_enabled=True, + ) + logger.debug("setup transport params") + + # Initialize transport with Daily + transport = DailyTransport( + room_url, + token, + "Simple Dial-in Bot", + transport_params, + ) + logger.debug("setup transport") + + # Initialize TTS + tts = CartesiaTTSService( + api_key=os.getenv("CARTESIA_API_KEY", ""), + voice_id="b7d50908-b17c-442d-ad8d-810c63997ed9", # Use Helpful Woman voice by default + ) + logger.debug("setup tts") + + # ------------ LLM AND CONTEXT SETUP ------------ + + # Set up the system instruction for the LLM + + # Initialize LLM + llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY")) + logger.debug("setup llm") + + # Initialize LLM context with system prompt + messages = [ + { + "role": "system", + "content": ( + "You are a friendly phone assistant. Your responses will be read aloud, " + "so keep them concise and conversational. Avoid special characters or " + "formatting. Begin by greeting the caller and asking how you can help them today." + ), + }, + ] + + # Setup the conversational context + context = OpenAILLMContext(messages) + logger.debug("setup context") + context_aggregator = llm.create_context_aggregator(context) + logger.debug("setup context aggregator") + + # ------------ PIPELINE SETUP ------------ + + # Build the pipeline + pipeline = Pipeline( + [ + transport.input(), # Transport user input + context_aggregator.user(), # User responses + llm, # LLM + tts, # TTS + transport.output(), # Transport bot output + context_aggregator.assistant(), # Assistant spoken responses + ] + ) + logger.debug("setup pipeline") + + # Create pipeline task + task = PipelineTask( + pipeline, + params=PipelineParams( + allow_interruptions=True # Enable barge-in so callers can interrupt the bot + ), + ) + logger.debug("setup task") + + # ------------ EVENT HANDLERS ------------ + + @transport.event_handler("on_first_participant_joined") + async def on_first_participant_joined(transport, participant): + logger.debug(f"First participant joined: {participant['id']}") + await transport.capture_participant_transcription(participant["id"]) + await task.queue_frames([context_aggregator.user().get_context_frame()]) + + @transport.event_handler("on_participant_left") + async def on_participant_left(transport, participant, reason): + logger.debug(f"Participant left: {participant}, reason: {reason}") + await task.cancel() + + @transport.event_handler("on_dialin_ready") + async def on_dialin_ready(transport, cdata): + logger.debug(f"Dial-in ready: {cdata}") + + @transport.event_handler("on_dialin_connected") + async def on_dialin_connected(transport, data): + logger.debug(f"Dial-in connected: {data}") + + @transport.event_handler("on_dialin_stopped") + async def on_dialin_stopped(transport, data): + logger.debug(f"Dial-in stopped: {data}") + + @transport.event_handler("on_dialin_error") + async def on_dialin_error(transport, data): + logger.error(f"Dial-in error: {data}") + # If there is an error, the bot should leave the call + # This may be also handled in on_participant_left with + # await task.cancel() + + @transport.event_handler("on_dialin_warning") + async def on_dialin_warning(transport, data): + logger.warning(f"Dial-in warning: {data}") + + # Run the pipeline + runner = PipelineRunner() + await runner.run(task) + + +async def main(): + """Parse command line arguments and run the bot.""" + parser = argparse.ArgumentParser(description="Simple Dial-in Bot") + parser.add_argument("-u", "--url", type=str, help="Daily room URL") + parser.add_argument("-t", "--token", type=str, help="Daily room token") + parser.add_argument("-b", "--body", type=str, help="JSON configuration string") + + args = parser.parse_args() + + logger.debug(f"url: {args.url}") + logger.debug(f"token: {args.token}") + logger.debug(f"body: {args.body}") + if not all([args.url, args.token, args.body]): + logger.error("All arguments (-u, -t, -b) are required") + parser.print_help() + sys.exit(1) + + await run_bot(args.url, args.token, args.body) + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/examples/phone-chatbot/daily-pstn-dial-in/env.example b/examples/phone-chatbot/daily-pstn-dial-in/env.example new file mode 100644 index 000000000..f48a6a810 --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-dial-in/env.example @@ -0,0 +1,7 @@ +# Daily credentials +DAILY_API_KEY=your_daily_api_key +DAILY_API_URL=https://api.daily.co/v1 + +# Service keys +OPENAI_API_KEY=your_openai_api_key +CARTESIA_API_KEY=your_cartesia_api_key \ No newline at end of file diff --git a/examples/phone-chatbot/daily-pstn-dial-in/requirements.txt b/examples/phone-chatbot/daily-pstn-dial-in/requirements.txt new file mode 100644 index 000000000..a337e3c09 --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-dial-in/requirements.txt @@ -0,0 +1,6 @@ +pipecat-ai[daily,cartesia,openai,silero] +fastapi==0.115.6 +uvicorn +python-dotenv +python-multipart +aiohttp \ No newline at end of file diff --git a/examples/phone-chatbot/daily-pstn-dial-in/server.py b/examples/phone-chatbot/daily-pstn-dial-in/server.py new file mode 100644 index 000000000..2fb6de15c --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-dial-in/server.py @@ -0,0 +1,116 @@ +# +# Copyright (c) 2024–2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +"""server.py. + +Webhook server to handle webhook coming from Daily, create a Daily room and start the bot. +""" + +import json +import os +import shlex +import subprocess +from contextlib import asynccontextmanager + +import aiohttp +import uvicorn +from dotenv import load_dotenv +from fastapi import FastAPI, HTTPException, Request +from fastapi.responses import JSONResponse +from utils.daily_helpers import create_daily_room + +load_dotenv() + +# ----------------- API ----------------- # + + +@asynccontextmanager +async def lifespan(app: FastAPI): + # Create aiohttp session to be used for Daily API calls + app.state.session = aiohttp.ClientSession() + yield + # Close session when shutting down + await app.state.session.close() + + +app = FastAPI(lifespan=lifespan) + + +@app.post("/start") +async def handle_incoming_daily_webhook(request: Request) -> JSONResponse: + """Handle incoming Daily call webhook.""" + print("Received webhook from Daily") + + # Get the dial-in properties from the request + try: + data = await request.json() + if "test" in data: + # Pass through any webhook checks + return JSONResponse({"test": True}) + + if not all(key in data for key in ["From", "To", "callId", "callDomain"]): + raise HTTPException( + status_code=400, detail="Missing properties 'From', 'To', 'callId', 'callDomain'" + ) + + # Extract the caller's phone number + caller_phone = str(data.get("From")) + print(f"Processing call from {caller_phone}") + + # Create a Daily room with dial-in capabilities + try: + room_details = await create_daily_room(request.app.state.session, caller_phone) + except Exception as e: + print(f"Error creating Daily room: {e}") + raise HTTPException(status_code=500, detail=f"Failed to create Daily room: {str(e)}") + + room_url = room_details["room_url"] + token = room_details["token"] + print(f"Created Daily room: {room_url} with token: {token}") + + body_json = json.dumps(data) + + bot_cmd = f"python3 -m bot -u {room_url} -t {token} -b {shlex.quote(body_json)}" + + try: + # CHANGE: Keep stdout/stderr for debugging + # Start the bot in the background but capture output + subprocess.Popen( + bot_cmd, + shell=True, + # Don't redirect output so we can see logs + # stdout=subprocess.DEVNULL, + # stderr=subprocess.DEVNULL + ) + print(f"Started bot process with command: {bot_cmd}") + except Exception as e: + print(f"Error starting bot: {e}") + raise HTTPException(status_code=500, detail=f"Failed to start bot: {str(e)}") + + except HTTPException: + raise + except Exception as e: + print(f"Unexpected error: {str(e)}") + raise HTTPException(status_code=500, detail=f"Server error: {str(e)}") + + # Grab a token for the user to join with + return JSONResponse({"room_url": room_url, "token": token}) + + +@app.get("/health") +async def health_check(): + """Simple health check endpoint.""" + return {"status": "healthy"} + + +# ----------------- Main ----------------- # + + +if __name__ == "__main__": + # Run the server + port = int(os.getenv("PORT", "7860")) + print(f"Starting server on port {port}") + uvicorn.run("server:app", host="0.0.0.0", port=port, reload=True) diff --git a/examples/phone-chatbot/daily-pstn-dial-in/utils/daily_helpers.py b/examples/phone-chatbot/daily-pstn-dial-in/utils/daily_helpers.py new file mode 100644 index 000000000..6451fc4c4 --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-dial-in/utils/daily_helpers.py @@ -0,0 +1,76 @@ +"""Helper functions for interacting with the Daily API.""" + +import os +from typing import Dict, Optional + +import aiohttp +from dotenv import load_dotenv + +from pipecat.transports.services.helpers.daily_rest import ( + DailyRESTHelper, + DailyRoomParams, + DailyRoomProperties, + DailyRoomSipParams, +) + +load_dotenv() + + +# Initialize Daily API helper +async def get_daily_helper(session: Optional[aiohttp.ClientSession] = None) -> DailyRESTHelper: + """Get a Daily REST helper with the configured API key.""" + if session is None: + session = aiohttp.ClientSession() + + return DailyRESTHelper( + daily_api_key=os.getenv("DAILY_API_KEY", ""), + daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"), + aiohttp_session=session, + ) + + +async def create_daily_room( + session: Optional[aiohttp.ClientSession] = None, caller_phone: str = "unknown-caller" +) -> Dict[str, str]: + """Create a Daily room with SIP capabilities for phone calls. + + Args: + session: Optional aiohttp session to use for API calls + caller_phone: The phone number of the caller to use in display name + + Returns: + Dictionary with room URL, token, and SIP endpoint + """ + daily_helper = await get_daily_helper(session) + + # Configure SIP parameters + sip_params = DailyRoomSipParams( + display_name=caller_phone, + video=False, + sip_mode="dial-in", + num_endpoints=1, + ) + + # Create room properties with SIP enabled + properties = DailyRoomProperties( + sip=sip_params, + enable_dialout=True, # Needed for outbound calls if you expand the bot + enable_chat=False, # No need for chat in a voice bot + start_video_off=True, # Voice only + ) + + # Create room parameters + params = DailyRoomParams(properties=properties) + + # Create the room + try: + room = await daily_helper.create_room(params=params) + print(f"Created room: {room.url} with SIP endpoint: {room.config.sip_endpoint}") + + # Get token for the bot to join + token = await daily_helper.get_token(room.url, 24 * 60 * 60) # 24 hours validity + + return {"room_url": room.url, "token": token, "sip_endpoint": room.config.sip_endpoint} + except Exception as e: + print(f"Error creating room: {e}") + raise diff --git a/examples/phone-chatbot/daily-pstn-dial-out/README.md b/examples/phone-chatbot/daily-pstn-dial-out/README.md new file mode 100644 index 000000000..f939bac57 --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-dial-out/README.md @@ -0,0 +1,111 @@ +# Daily PSTN dial-out simple chatbot + +This project demonstrates how to create a voice bot that uses Dailys PSTN capabilities to make calls to phone numbers. + +## How it works + +1. The server file receives a curl request with the phone number to dial out to +2. The server creates a Daily room with SIP capabilities +3. The server starts the bot process with the room details +4. When the bot has joined, it starts the dial-out process and rings the number provided in the curl request +5. The user then answers the phone and the user is brought into the call +6. The end user and bot are connected, and the bot handles the conversation + +## Prerequisites + +- A Daily account with an API key, and a phone number purchased through Daily +- A US phone number to ring +- dial-out must be enabled on your domain. Find out more by reading this [document and filling in the form](https://docs.daily.co/guides/products/dial-in-dial-out#main) +- OpenAI API key for the bot's intelligence +- Cartesia API key for text-to-speech + +## Setup + +1. Create a virtual environment and install dependencies + +```bash +python -m venv venv +source venv/bin/activate # On Windows: venv\Scripts\activate +pip install -r requirements.txt +``` + +2. Set up environment variables + +Copy the example file and fill in your API keys: + +```bash +cp .env.example .env +# Edit .env with your API keys +``` + +3. Buy a phone number + +Instructions on how to do that can be found at this [docs link:](https://docs.daily.co/reference/rest-api/phone-numbers/buy-phone-number) + +4. Request dial-out enablement + +For compliance reasons, to enable dial-out for your Daily account, you must request enablement via the form. You can find out more about dial-out, and the form at the [link here:](https://docs.daily.co/guides/products/dial-in-dial-out#main) + +## Running the Server + +Start the webhook server: + +```bash +python server.py +``` + +## Testing + +With server.py running, send the following curl command from your terminal: + +```bash +curl -X POST "http://127.0.0.1:7860/start" \ + -H "Content-Type: application/json" \ + -d '{ + "dialout_settings": { + "phone_number": "+12345678910" + } + }' +``` + +The server should make a room. The bot will join the room and then ring the number provided. Answer the call to speak with the bot. + +## Customizing the Bot + +You can customize the bot's behavior by modifying the system prompt in `bot.py`. + +## Multiple SIP Endpoints + +For PSTN calls, you only need one SIP endpoint. + +## Daily dial-out configuration + +The bot configures the Daily rooms with dial-out capabilities using these settings. Note: You also need dial-out to be enabled on the domain, as mentioned earlier on in the README. + +```python +properties = DailyRoomProperties( + sip=sip_params, + enable_dialout=True, # Needed for outbound calls if you expand the bot + enable_chat=False, # No need for chat in a voice bot + start_video_off=True, # Voice only +) +``` + +## Troubleshooting + +### I get an error about dial-out not being enabled + +- Check that your room has `enable_dialout=True` set +- Check that your meeting token is an owner token (The bot does this for you automatically) +- Check that you have purchased a phone number to ring from +- Check that the phone number you are trying to ring is correct, and is a US or Canadian number. + +### Call connects but no bot is heard + +- Ensure your Daily API key is correct and has SIP capabilities +- Verify that the Cartesia API key and voice ID are correct + +### Bot starts but disconnects immediately + +- Check the Daily logs for any error messages +- Ensure your server has stable internet connectivity diff --git a/examples/phone-chatbot/daily-pstn-dial-out/bot.py b/examples/phone-chatbot/daily-pstn-dial-out/bot.py new file mode 100644 index 000000000..752c4941d --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-dial-out/bot.py @@ -0,0 +1,239 @@ +# +# Copyright (c) 2024–2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +"""simple_dialout.py. + +Simple Dial-out Bot. +""" + +import argparse +import asyncio +import json +import os +import sys +from typing import Any + +from dotenv import load_dotenv +from loguru import logger + +from pipecat.audio.vad.silero import SileroVADAnalyzer +from pipecat.pipeline.pipeline import Pipeline +from pipecat.pipeline.runner import PipelineRunner +from pipecat.pipeline.task import PipelineParams, PipelineTask +from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext +from pipecat.services.cartesia.tts import CartesiaTTSService +from pipecat.services.openai.llm import OpenAILLMService +from pipecat.transports.services.daily import DailyParams, DailyTransport + +load_dotenv(override=True) + +logger.remove(0) +logger.add(sys.stderr, level="DEBUG") + +daily_api_key = os.getenv("DAILY_API_KEY", "") +daily_api_url = os.getenv("DAILY_API_URL", "https://api.daily.co/v1") + + +async def run_bot( + room_url: str, + token: str, + body: dict, +) -> None: + """Run the voice bot with the given parameters. + + Args: + room_url: The Daily room URL + token: The Daily room token + body: Body passed to the bot from the webhook + + """ + # ------------ CONFIGURATION AND SETUP ------------ + logger.info(f"Starting bot with room: {room_url}") + logger.info(f"Token: {token}") + logger.info(f"Body: {body}") + # Parse the body to get the dial-in settings + body_data = json.loads(body) + + # Check if the body contains dial-in settings + logger.debug(f"Body data: {body_data}") + + if not body_data.get("dialout_settings"): + logger.error("Dial-out settings not found in the body data") + return + + dialout_settings = body_data["dialout_settings"] + + if not dialout_settings.get("phone_number"): + logger.error("Dial-out phone number not found in the dial-out settings") + return + + # Extract dial-out phone number + phone_number = dialout_settings["phone_number"] + caller_id = dialout_settings.get("caller_id") # Use .get() to handle optional field + + if caller_id: + logger.info(f"Dial-out caller ID specified: {caller_id}") + else: + logger.info("Dial-out caller ID not specified; proceeding without it") + + # ------------ TRANSPORT SETUP ------------ + + transport_params = DailyParams( + api_url=daily_api_url, + api_key=daily_api_key, + audio_in_enabled=True, + audio_out_enabled=True, + video_out_enabled=False, + vad_analyzer=SileroVADAnalyzer(), + transcription_enabled=True, + ) + + # Initialize transport with Daily + transport = DailyTransport( + room_url, + token, + "Simple Dial-out Bot", + transport_params, + ) + + # Initialize TTS + tts = CartesiaTTSService( + api_key=os.getenv("CARTESIA_API_KEY", ""), + voice_id="b7d50908-b17c-442d-ad8d-810c63997ed9", # Use Helpful Woman voice by default + ) + + # ------------ LLM AND CONTEXT SETUP ------------ + + # Initialize LLM + llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY")) + + # Create system message and initialize messages list + messages = [ + { + "role": "system", + "content": ( + "You are a friendly phone assistant. Your responses will be read aloud, " + "so keep them concise and conversational. Avoid special characters or " + "formatting. Begin by greeting the caller and asking how you can help them today." + ), + }, + ] + # Initialize LLM context and aggregator + context = OpenAILLMContext(messages) + context_aggregator = llm.create_context_aggregator(context) + + # ------------ PIPELINE SETUP ------------ + + # Build pipeline + pipeline = Pipeline( + [ + transport.input(), # Transport user input + context_aggregator.user(), # User responses + llm, # LLM + tts, # TTS + transport.output(), # Transport bot output + context_aggregator.assistant(), # Assistant spoken responses + ] + ) + + # Create pipeline task + task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True)) + + # ------------ RETRY LOGIC VARIABLES ------------ + max_retries = 5 + retry_count = 0 + dialout_successful = False + + # Build dialout parameters conditionally + dialout_params = {"phoneNumber": phone_number} + if caller_id: + dialout_params["callerId"] = caller_id + logger.debug(f"Including caller ID in dialout: {caller_id}") + + logger.debug(f"Dialout parameters: {dialout_params}") + + async def attempt_dialout(): + """Attempt to start dialout with retry logic.""" + nonlocal retry_count, dialout_successful + + if retry_count < max_retries and not dialout_successful: + retry_count += 1 + logger.info( + f"Attempting dialout (attempt {retry_count}/{max_retries}) to: {phone_number}" + ) + await transport.start_dialout(dialout_params) + else: + logger.error(f"Maximum retry attempts ({max_retries}) reached. Giving up on dialout.") + + # ------------ EVENT HANDLERS ------------ + + @transport.event_handler("on_joined") + async def on_joined(transport, data): + # Start initial dialout attempt + logger.debug(f"Dialout settings detected; starting dialout to number: {phone_number}") + await attempt_dialout() + + @transport.event_handler("on_dialout_connected") + async def on_dialout_connected(transport, data): + logger.debug(f"Dial-out connected: {data}") + + @transport.event_handler("on_dialout_answered") + async def on_dialout_answered(transport, data): + nonlocal dialout_successful + logger.debug(f"Dial-out answered: {data}") + dialout_successful = True # Mark as successful to stop retries + # Automatically start capturing transcription for the participant + await transport.capture_participant_transcription(data["sessionId"]) + # The bot will wait to hear the user before the bot speaks + + @transport.event_handler("on_dialout_error") + async def on_dialout_error(transport, data: Any): + logger.error(f"Dial-out error (attempt {retry_count}/{max_retries}): {data}") + + if retry_count < max_retries: + logger.info(f"Retrying dialout") + await attempt_dialout() + else: + logger.error(f"All {max_retries} dialout attempts failed. Stopping bot.") + await task.cancel() + + @transport.event_handler("on_first_participant_joined") + async def on_first_participant_joined(transport, participant): + logger.debug(f"First participant joined: {participant['id']}") + + @transport.event_handler("on_participant_left") + async def on_participant_left(transport, participant, reason): + logger.debug(f"Participant left: {participant}, reason: {reason}") + await task.cancel() + + # ------------ RUN PIPELINE ------------ + + runner = PipelineRunner() + await runner.run(task) + + +async def main(): + """Parse command line arguments and run the bot.""" + parser = argparse.ArgumentParser(description="Simple Dial-out Bot") + parser.add_argument("-u", "--url", type=str, help="Room URL") + parser.add_argument("-t", "--token", type=str, help="Room Token") + parser.add_argument("-b", "--body", type=str, help="JSON configuration string") + + args = parser.parse_args() + + logger.debug(f"url: {args.url}") + logger.debug(f"token: {args.token}") + logger.debug(f"body: {args.body}") + if not all([args.url, args.token, args.body]): + logger.error("All arguments (-u, -t, -b) are required") + parser.print_help() + sys.exit(1) + + await run_bot(args.url, args.token, args.body) + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/examples/phone-chatbot/daily-pstn-dial-out/env.example b/examples/phone-chatbot/daily-pstn-dial-out/env.example new file mode 100644 index 000000000..f48a6a810 --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-dial-out/env.example @@ -0,0 +1,7 @@ +# Daily credentials +DAILY_API_KEY=your_daily_api_key +DAILY_API_URL=https://api.daily.co/v1 + +# Service keys +OPENAI_API_KEY=your_openai_api_key +CARTESIA_API_KEY=your_cartesia_api_key \ No newline at end of file diff --git a/examples/phone-chatbot/daily-pstn-dial-out/requirements.txt b/examples/phone-chatbot/daily-pstn-dial-out/requirements.txt new file mode 100644 index 000000000..a337e3c09 --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-dial-out/requirements.txt @@ -0,0 +1,6 @@ +pipecat-ai[daily,cartesia,openai,silero] +fastapi==0.115.6 +uvicorn +python-dotenv +python-multipart +aiohttp \ No newline at end of file diff --git a/examples/phone-chatbot/daily-pstn-dial-out/server.py b/examples/phone-chatbot/daily-pstn-dial-out/server.py new file mode 100644 index 000000000..4aed288fc --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-dial-out/server.py @@ -0,0 +1,121 @@ +# +# Copyright (c) 2024–2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +"""server.py. + +Webhook server to handle webhook coming from Daily, create a Daily room and start the bot. +""" + +import json +import os +import shlex +import subprocess +from contextlib import asynccontextmanager + +import aiohttp +import uvicorn +from dotenv import load_dotenv +from fastapi import FastAPI, HTTPException, Request +from fastapi.responses import JSONResponse +from utils.daily_helpers import create_daily_room + +load_dotenv() + +# ----------------- API ----------------- # + + +@asynccontextmanager +async def lifespan(app: FastAPI): + # Create aiohttp session to be used for Daily API calls + app.state.session = aiohttp.ClientSession() + yield + # Close session when shutting down + await app.state.session.close() + + +app = FastAPI(lifespan=lifespan) + + +@app.post("/start") +async def handle_incoming_daily_webhook(request: Request) -> JSONResponse: + """Handle dial-out request.""" + print("Received webhook from Daily") + + # Get the dial-in properties from the request + try: + data = await request.json() + if "test" in data: + # Pass through any webhook checks + return JSONResponse({"test": True}) + + if not data["dialout_settings"]: + raise HTTPException( + status_code=400, detail="Missing 'dialout_settings' in the request body" + ) + + if not data["dialout_settings"].get("phone_number"): + raise HTTPException( + status_code=400, detail="Missing 'phone_number' in dialout_settings" + ) + + # Extract the phone number we want to dial out to + caller_phone = str(data["dialout_settings"]["phone_number"]) + print(f"Processing call to {caller_phone}") + + # Create a Daily room with dial-in capabilities + try: + room_details = await create_daily_room(request.app.state.session, caller_phone) + except Exception as e: + print(f"Error creating Daily room: {e}") + raise HTTPException(status_code=500, detail=f"Failed to create Daily room: {str(e)}") + + room_url = room_details["room_url"] + token = room_details["token"] + print(f"Created Daily room: {room_url} with token: {token}") + + body_json = json.dumps(data) + + bot_cmd = f"python3 -m bot -u {room_url} -t {token} -b {shlex.quote(body_json)}" + + try: + # CHANGE: Keep stdout/stderr for debugging + # Start the bot in the background but capture output + subprocess.Popen( + bot_cmd, + shell=True, + # Don't redirect output so we can see logs + # stdout=subprocess.DEVNULL, + # stderr=subprocess.DEVNULL + ) + print(f"Started bot process with command: {bot_cmd}") + except Exception as e: + print(f"Error starting bot: {e}") + raise HTTPException(status_code=500, detail=f"Failed to start bot: {str(e)}") + + except HTTPException: + raise + except Exception as e: + print(f"Unexpected error: {str(e)}") + raise HTTPException(status_code=500, detail=f"Server error: {str(e)}") + + # Grab a token for the user to join with + return JSONResponse({"room_url": room_url, "token": token}) + + +@app.get("/health") +async def health_check(): + """Simple health check endpoint.""" + return {"status": "healthy"} + + +# ----------------- Main ----------------- # + + +if __name__ == "__main__": + # Run the server + port = int(os.getenv("PORT", "7860")) + print(f"Starting server on port {port}") + uvicorn.run("server:app", host="0.0.0.0", port=port, reload=True) diff --git a/examples/phone-chatbot/daily-pstn-dial-out/utils/daily_helpers.py b/examples/phone-chatbot/daily-pstn-dial-out/utils/daily_helpers.py new file mode 100644 index 000000000..6451fc4c4 --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-dial-out/utils/daily_helpers.py @@ -0,0 +1,76 @@ +"""Helper functions for interacting with the Daily API.""" + +import os +from typing import Dict, Optional + +import aiohttp +from dotenv import load_dotenv + +from pipecat.transports.services.helpers.daily_rest import ( + DailyRESTHelper, + DailyRoomParams, + DailyRoomProperties, + DailyRoomSipParams, +) + +load_dotenv() + + +# Initialize Daily API helper +async def get_daily_helper(session: Optional[aiohttp.ClientSession] = None) -> DailyRESTHelper: + """Get a Daily REST helper with the configured API key.""" + if session is None: + session = aiohttp.ClientSession() + + return DailyRESTHelper( + daily_api_key=os.getenv("DAILY_API_KEY", ""), + daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"), + aiohttp_session=session, + ) + + +async def create_daily_room( + session: Optional[aiohttp.ClientSession] = None, caller_phone: str = "unknown-caller" +) -> Dict[str, str]: + """Create a Daily room with SIP capabilities for phone calls. + + Args: + session: Optional aiohttp session to use for API calls + caller_phone: The phone number of the caller to use in display name + + Returns: + Dictionary with room URL, token, and SIP endpoint + """ + daily_helper = await get_daily_helper(session) + + # Configure SIP parameters + sip_params = DailyRoomSipParams( + display_name=caller_phone, + video=False, + sip_mode="dial-in", + num_endpoints=1, + ) + + # Create room properties with SIP enabled + properties = DailyRoomProperties( + sip=sip_params, + enable_dialout=True, # Needed for outbound calls if you expand the bot + enable_chat=False, # No need for chat in a voice bot + start_video_off=True, # Voice only + ) + + # Create room parameters + params = DailyRoomParams(properties=properties) + + # Create the room + try: + room = await daily_helper.create_room(params=params) + print(f"Created room: {room.url} with SIP endpoint: {room.config.sip_endpoint}") + + # Get token for the bot to join + token = await daily_helper.get_token(room.url, 24 * 60 * 60) # 24 hours validity + + return {"room_url": room.url, "token": token, "sip_endpoint": room.config.sip_endpoint} + except Exception as e: + print(f"Error creating room: {e}") + raise diff --git a/examples/phone-chatbot/daily-pstn-simple-voicemail-detection/README.md b/examples/phone-chatbot/daily-pstn-simple-voicemail-detection/README.md new file mode 100644 index 000000000..f72709579 --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-simple-voicemail-detection/README.md @@ -0,0 +1,121 @@ +# Daily PSTN Simple Voicemail Detection Bot + +This project demonstrates how to create a voice bot that uses Dailys PSTN capabilities to make calls to phone numbers, and if the bot hits a voicemail system, to have the bot also leave a message. + +## How it works + +1. The server file receives a curl request with the phone number to dial out to +2. The server creates a Daily room with SIP capabilities +3. The server starts the bot process with the room details +4. When the bot has joined, it starts the dial-out process and rings the number provided in the curl request +5. When the phone is answered, the bot detects for certain key phrases +6. If the bot detects those key phrases, it leaves a messages and then ends the call +7. If the bot detects there's a human on the phone, it behaves like a regular bot + +## Prerequisites + +- A Daily account with an API key, and a phone number purchased through Daily +- A US phone number to ring +- dial-out must be enabled on your domain. Find out more by reading this [document and filling in the form](https://docs.daily.co/guides/products/dial-in-dial-out#main) +- Google API key for the bot's intelligence +- Cartesia API key for text-to-speech + +## Setup + +1. Create a virtual environment and install dependencies + +```bash +python -m venv venv +source venv/bin/activate # On Windows: venv\Scripts\activate +pip install -r requirements.txt +``` + +2. Set up environment variables + +Copy the example file and fill in your API keys: + +```bash +cp .env.example .env +# Edit .env with your API keys +``` + +3. Buy a phone number + +Instructions on how to do that can be found at this [docs link:](https://docs.daily.co/reference/rest-api/phone-numbers/buy-phone-number) + +4. Request dial-out enablement + +For compliance reasons, to enable dial-out for your Daily account, you must request enablement via the form. You can find out more about dial-out, and the form at the [link here:](https://docs.daily.co/guides/products/dial-in-dial-out#main) + +## Running the Server + +Start the webhook server: + +```bash +python server.py +``` + +## Testing + +With server.py running, send the following curl command from your terminal: + +```bash +curl -X POST "http://127.0.0.1:7860/start" \ + -H "Content-Type: application/json" \ + -d '{ + "dialout_settings": { + "phone_number": "+12345678910" + } + }' +``` + +The server should make a room. The bot will join the room and then ring the number provided. Answer the call to speak with the bot. + +- You can pretend to be a voicemail machine by saying something like "Please leave a message after the beep... beeeeep". +- You should observe the bot detects the voicemail machine and leaves a message before terminating the call +- You can also say something like "Hello?", and the bot will notice you're likely a human and begin having a conversation with you + +## Customizing the Bot + +You can customize the bot's behavior by modifying the system prompt in `bot.py`. + +## Multiple SIP Endpoints + +For PSTN calls, you only need one SIP endpoint. + +## Daily dial-out configuration + +The bot configures the Daily rooms with dial-out capabilities using these settings. Note: You also need dial-out to be enabled on the domain, as mentioned earlier on in the README. + +```python +properties = DailyRoomProperties( + sip=sip_params, + enable_dialout=True, # Needed for outbound calls if you expand the bot + enable_chat=False, # No need for chat in a voice bot + start_video_off=True, # Voice only +) +``` + +## Troubleshooting + +### I get an error about dial-out not being enabled + +- Check that your room has `enable_dialout=True` set +- Check that your meeting token is an owner token (The bot does this for you automatically) +- Check that you have purchased a phone number to ring from +- Check that the phone number you are trying to ring is correct, and is a US or Canadian number. + +### The bot doesn't detect my voicemail + +- The bot should be smart enough to detect variations of certain patterns, +- If your voicemail machine doesn't follow the example patterns, add the pattern to the LLM prompt + +### Call connects but no bot is heard + +- Ensure your Daily API key is correct and has SIP capabilities +- Verify that the Cartesia API key and voice ID are correct + +### Bot starts but disconnects immediately + +- Check the Daily logs for any error messages +- Ensure your server has stable internet connectivity diff --git a/examples/phone-chatbot/daily-pstn-simple-voicemail-detection/bot.py b/examples/phone-chatbot/daily-pstn-simple-voicemail-detection/bot.py new file mode 100644 index 000000000..46d8051f5 --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-simple-voicemail-detection/bot.py @@ -0,0 +1,313 @@ +# +# Copyright (c) 2024–2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# +import argparse +import asyncio +import json +import os +import sys +from typing import Any + +from dotenv import load_dotenv +from loguru import logger + +from pipecat.audio.vad.silero import SileroVADAnalyzer +from pipecat.frames.frames import EndTaskFrame +from pipecat.pipeline.pipeline import Pipeline +from pipecat.pipeline.runner import PipelineRunner +from pipecat.pipeline.task import PipelineParams, PipelineTask +from pipecat.processors.frame_processor import FrameDirection +from pipecat.services.cartesia.tts import CartesiaTTSService +from pipecat.services.google.google import GoogleLLMContext +from pipecat.services.google.llm import GoogleLLMService +from pipecat.services.llm_service import FunctionCallParams +from pipecat.transports.services.daily import DailyParams, DailyTransport + +load_dotenv(override=True) + +logger.remove(0) +logger.add(sys.stderr, level="DEBUG") + + +daily_api_key = os.getenv("DAILY_API_KEY", "") +daily_api_url = os.getenv("DAILY_API_URL", "https://api.daily.co/v1") + + +async def run_bot( + room_url: str, + token: str, + body: dict, +) -> None: + """Run the voice bot with the given parameters. + + Args: + room_url: The Daily room URL + token: The Daily room token + body: Body passed to the bot from the webhook + + """ + # ------------ CONFIGURATION AND SETUP ------------ + logger.info(f"Starting bot with room: {room_url}") + logger.info(f"Token: {token}") + logger.info(f"Body: {body}") + # Parse the body to get the dial-in settings + body_data = json.loads(body) + + # Check if the body contains dial-in settings + logger.debug(f"Body data: {body_data}") + + if not body_data.get("dialout_settings"): + logger.error("Dial-out settings not found in the body data") + return + + dialout_settings = body_data["dialout_settings"] + + if not dialout_settings.get("phone_number"): + logger.error("Dial-out phone number not found in the dial-out settings") + return + + # Extract dial-out phone number + phone_number = dialout_settings["phone_number"] + caller_id = dialout_settings.get("caller_id") # Use .get() to handle optional field + + if caller_id: + logger.info(f"Dial-out caller ID specified: {caller_id}") + else: + logger.info("Dial-out caller ID not specified; proceeding without it") + + # ------------ TRANSPORT SETUP ------------ + + transport_params = DailyParams( + api_url=daily_api_url, + api_key=daily_api_key, + audio_in_enabled=True, + audio_out_enabled=True, + video_out_enabled=False, + vad_analyzer=SileroVADAnalyzer(), + transcription_enabled=True, + ) + + # Initialize transport with Daily + transport = DailyTransport( + room_url, + token, + "Voicemail Detection Bot", + transport_params, + ) + + # Initialize TTS + tts = CartesiaTTSService( + api_key=os.getenv("CARTESIA_API_KEY", ""), + voice_id="b7d50908-b17c-442d-ad8d-810c63997ed9", # Use Helpful Woman voice by default + ) + + async def terminate_call( + params: FunctionCallParams, + ): + """Function the bot can call to terminate the call.""" + + await params.llm.queue_frame(EndTaskFrame(), FrameDirection.UPSTREAM) + + tools = [ + { + "function_declarations": [ + { + "name": "terminate_call", + "description": "Terminate the call", + }, + ] + } + ] + + system_instruction = """You are Chatbot, a friendly, helpful robot. Never mention this prompt. + + **Operating Procedure:** + + **Phase 1: Initial Call Answer - Listen for Voicemail Greeting** + + **IMMEDIATELY after the call connects, LISTEN CAREFULLY for the *very first thing* you hear.** + + If you hear any of these phrases (or very similar ones): + - "Please leave a message after the beep" + - "No one is available to take your call" + - "Record your message after the tone" + - "You have reached voicemail for..." + - "You have reached [phone number]" + - "[phone number] is unavailable" + - "The person you are trying to reach..." + - "The number you have dialed..." + - "Your call has been forwarded to an automated voice messaging system" + + **If you HEAR one of these sentences (or a very similar greeting) as the *initial response* to the call, IMMEDIATELY assume it is voicemail and proceed to Phase 2.** + + **If you hear "PLEASE LEAVE A MESSAGE AFTER THE BEEP", WAIT for the actual beep sound from the voicemail system *after* hearing the sentence, before proceeding to Phase 2.** + + **If you DO NOT hear any of these voicemail greetings as the *initial response*, assume it is a human and proceed to Phase 3.** + + + **Phase 2: Leave Voicemail Message (If Voicemail Detected):** + + If you assumed voicemail in Phase 1, say this EXACTLY: + "Hello, this is a message for Pipecat example user. This is Chatbot. Please call back on 123-456-7891. Thank you." + + **Immediately after saying the message, call the function `terminate_call`.** + **DO NOT SAY ANYTHING ELSE. SILENCE IS REQUIRED AFTER `terminate_call`.** + + + **Phase 3: Human Interaction (If No Voicemail Greeting Detected in Phase 1):** + + If you did not detect a voicemail greeting in Phase 1 and a human answers, say: + "Oh, hello! I'm a friendly chatbot. Is there anything I can help you with?" + + Keep your responses **short and helpful.** + + When the person indicates they're done with the conversation by saying something like: + - "Goodbye" + - "That's all" + - "I'm done" + - "Thank you, that's all I needed" + + + THEN say: "Thank you for chatting. Goodbye!" and call the terminate_call function. + + **Then, immediately call the function `terminate_call`.** + + + **VERY IMPORTANT RULES - DO NOT DO THESE THINGS:** + + * **DO NOT SAY "Please leave a message after the beep."** + * **DO NOT SAY "No one is available to take your call."** + * **DO NOT SAY "Record your message after the tone."** + * **DO NOT SAY ANY voicemail greeting yourself.** + * **Only check for voicemail greetings in Phase 1, *immediately after the call connects*.** + * **After voicemail or human interaction, ALWAYS call `terminate_call` immediately.** + * **Do not speak after calling `terminate_call`.** + * Your speech will be audio, so use simple language without special characters. + """ + + llm = GoogleLLMService( + model="models/gemini-2.0-flash-001", # Full model for better conversation + api_key=os.getenv("GOOGLE_API_KEY"), + system_instruction=system_instruction, + tools=tools, + ) + llm.register_function("terminate_call", terminate_call) + + context = GoogleLLMContext() + + context_aggregator = llm.create_context_aggregator(context) + + pipeline = Pipeline( + [ + transport.input(), # Transport user input + context_aggregator.user(), # User responses + llm, # LLM + tts, # TTS + transport.output(), # Transport bot output + context_aggregator.assistant(), # Assistant spoken responses + ] + ) + + # Create pipeline task + task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True)) + + # ------------ RETRY LOGIC VARIABLES ------------ + max_retries = 5 + retry_count = 0 + dialout_successful = False + + # Build dialout parameters conditionally + dialout_params = {"phoneNumber": phone_number} + if caller_id: + dialout_params["callerId"] = caller_id + logger.debug(f"Including caller ID in dialout: {caller_id}") + + logger.debug(f"Dialout parameters: {dialout_params}") + + async def attempt_dialout(): + """Attempt to start dialout with retry logic.""" + nonlocal retry_count, dialout_successful + + if retry_count < max_retries and not dialout_successful: + retry_count += 1 + logger.info( + f"Attempting dialout (attempt {retry_count}/{max_retries}) to: {phone_number}" + ) + await transport.start_dialout(dialout_params) + else: + logger.error(f"Maximum retry attempts ({max_retries}) reached. Giving up on dialout.") + + # ------------ EVENT HANDLERS ------------ + + @transport.event_handler("on_joined") + async def on_joined(transport, data): + # Start initial dialout attempt + logger.debug(f"Dialout settings detected; starting dialout to number: {phone_number}") + await attempt_dialout() + + @transport.event_handler("on_dialout_connected") + async def on_dialout_connected(transport, data): + logger.debug(f"Dial-out connected: {data}") + + @transport.event_handler("on_dialout_answered") + async def on_dialout_answered(transport, data): + nonlocal dialout_successful + logger.debug(f"Dial-out answered: {data}") + dialout_successful = True # Mark as successful to stop retries + # Automatically start capturing transcription for the participant + await transport.capture_participant_transcription(data["sessionId"]) + # The bot will wait to hear the user before the bot speaks + + @transport.event_handler("on_dialout_error") + async def on_dialout_error(transport, data: Any): + logger.error(f"Dial-out error (attempt {retry_count}/{max_retries}): {data}") + + if retry_count < max_retries: + logger.info(f"Retrying dialout") + await attempt_dialout() + else: + logger.error(f"All {max_retries} dialout attempts failed. Stopping bot.") + await task.cancel() + + @transport.event_handler("on_first_participant_joined") + async def on_first_participant_joined(transport, participant): + logger.debug(f"First participant joined: {participant['id']}") + + @transport.event_handler("on_participant_left") + async def on_participant_left(transport, participant, reason): + logger.debug(f"Participant left: {participant}, reason: {reason}") + await task.cancel() + + # ------------ RUN PIPELINE ------------ + + runner = PipelineRunner() + await runner.run(task) + + +# ------------ SCRIPT ENTRY POINT ------------ + + +async def main(): + """Parse command line arguments and run the bot.""" + parser = argparse.ArgumentParser(description="Simple Dial-out Bot") + parser.add_argument("-u", "--url", type=str, help="Room URL") + parser.add_argument("-t", "--token", type=str, help="Room Token") + parser.add_argument("-b", "--body", type=str, help="JSON configuration string") + + args = parser.parse_args() + + logger.debug(f"url: {args.url}") + logger.debug(f"token: {args.token}") + logger.debug(f"body: {args.body}") + if not all([args.url, args.token, args.body]): + logger.error("All arguments (-u, -t, -b) are required") + parser.print_help() + sys.exit(1) + + await run_bot(args.url, args.token, args.body) + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/examples/phone-chatbot/daily-pstn-simple-voicemail-detection/env.example b/examples/phone-chatbot/daily-pstn-simple-voicemail-detection/env.example new file mode 100644 index 000000000..78f2b2613 --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-simple-voicemail-detection/env.example @@ -0,0 +1,7 @@ +# Daily credentials +DAILY_API_KEY=your_daily_api_key +DAILY_API_URL=https://api.daily.co/v1 + +# Service keys +GOOGLE_API_KEY=your_google_api_key +CARTESIA_API_KEY=your_cartesia_api_key \ No newline at end of file diff --git a/examples/phone-chatbot/daily-pstn-simple-voicemail-detection/requirements.txt b/examples/phone-chatbot/daily-pstn-simple-voicemail-detection/requirements.txt new file mode 100644 index 000000000..d40b48761 --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-simple-voicemail-detection/requirements.txt @@ -0,0 +1,6 @@ +pipecat-ai[daily,cartesia,google,silero] +fastapi==0.115.6 +uvicorn +python-dotenv +python-multipart +aiohttp diff --git a/examples/phone-chatbot/daily-pstn-simple-voicemail-detection/server.py b/examples/phone-chatbot/daily-pstn-simple-voicemail-detection/server.py new file mode 100644 index 000000000..4aed288fc --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-simple-voicemail-detection/server.py @@ -0,0 +1,121 @@ +# +# Copyright (c) 2024–2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +"""server.py. + +Webhook server to handle webhook coming from Daily, create a Daily room and start the bot. +""" + +import json +import os +import shlex +import subprocess +from contextlib import asynccontextmanager + +import aiohttp +import uvicorn +from dotenv import load_dotenv +from fastapi import FastAPI, HTTPException, Request +from fastapi.responses import JSONResponse +from utils.daily_helpers import create_daily_room + +load_dotenv() + +# ----------------- API ----------------- # + + +@asynccontextmanager +async def lifespan(app: FastAPI): + # Create aiohttp session to be used for Daily API calls + app.state.session = aiohttp.ClientSession() + yield + # Close session when shutting down + await app.state.session.close() + + +app = FastAPI(lifespan=lifespan) + + +@app.post("/start") +async def handle_incoming_daily_webhook(request: Request) -> JSONResponse: + """Handle dial-out request.""" + print("Received webhook from Daily") + + # Get the dial-in properties from the request + try: + data = await request.json() + if "test" in data: + # Pass through any webhook checks + return JSONResponse({"test": True}) + + if not data["dialout_settings"]: + raise HTTPException( + status_code=400, detail="Missing 'dialout_settings' in the request body" + ) + + if not data["dialout_settings"].get("phone_number"): + raise HTTPException( + status_code=400, detail="Missing 'phone_number' in dialout_settings" + ) + + # Extract the phone number we want to dial out to + caller_phone = str(data["dialout_settings"]["phone_number"]) + print(f"Processing call to {caller_phone}") + + # Create a Daily room with dial-in capabilities + try: + room_details = await create_daily_room(request.app.state.session, caller_phone) + except Exception as e: + print(f"Error creating Daily room: {e}") + raise HTTPException(status_code=500, detail=f"Failed to create Daily room: {str(e)}") + + room_url = room_details["room_url"] + token = room_details["token"] + print(f"Created Daily room: {room_url} with token: {token}") + + body_json = json.dumps(data) + + bot_cmd = f"python3 -m bot -u {room_url} -t {token} -b {shlex.quote(body_json)}" + + try: + # CHANGE: Keep stdout/stderr for debugging + # Start the bot in the background but capture output + subprocess.Popen( + bot_cmd, + shell=True, + # Don't redirect output so we can see logs + # stdout=subprocess.DEVNULL, + # stderr=subprocess.DEVNULL + ) + print(f"Started bot process with command: {bot_cmd}") + except Exception as e: + print(f"Error starting bot: {e}") + raise HTTPException(status_code=500, detail=f"Failed to start bot: {str(e)}") + + except HTTPException: + raise + except Exception as e: + print(f"Unexpected error: {str(e)}") + raise HTTPException(status_code=500, detail=f"Server error: {str(e)}") + + # Grab a token for the user to join with + return JSONResponse({"room_url": room_url, "token": token}) + + +@app.get("/health") +async def health_check(): + """Simple health check endpoint.""" + return {"status": "healthy"} + + +# ----------------- Main ----------------- # + + +if __name__ == "__main__": + # Run the server + port = int(os.getenv("PORT", "7860")) + print(f"Starting server on port {port}") + uvicorn.run("server:app", host="0.0.0.0", port=port, reload=True) diff --git a/examples/phone-chatbot/daily-pstn-simple-voicemail-detection/utils/daily_helpers.py b/examples/phone-chatbot/daily-pstn-simple-voicemail-detection/utils/daily_helpers.py new file mode 100644 index 000000000..6451fc4c4 --- /dev/null +++ b/examples/phone-chatbot/daily-pstn-simple-voicemail-detection/utils/daily_helpers.py @@ -0,0 +1,76 @@ +"""Helper functions for interacting with the Daily API.""" + +import os +from typing import Dict, Optional + +import aiohttp +from dotenv import load_dotenv + +from pipecat.transports.services.helpers.daily_rest import ( + DailyRESTHelper, + DailyRoomParams, + DailyRoomProperties, + DailyRoomSipParams, +) + +load_dotenv() + + +# Initialize Daily API helper +async def get_daily_helper(session: Optional[aiohttp.ClientSession] = None) -> DailyRESTHelper: + """Get a Daily REST helper with the configured API key.""" + if session is None: + session = aiohttp.ClientSession() + + return DailyRESTHelper( + daily_api_key=os.getenv("DAILY_API_KEY", ""), + daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"), + aiohttp_session=session, + ) + + +async def create_daily_room( + session: Optional[aiohttp.ClientSession] = None, caller_phone: str = "unknown-caller" +) -> Dict[str, str]: + """Create a Daily room with SIP capabilities for phone calls. + + Args: + session: Optional aiohttp session to use for API calls + caller_phone: The phone number of the caller to use in display name + + Returns: + Dictionary with room URL, token, and SIP endpoint + """ + daily_helper = await get_daily_helper(session) + + # Configure SIP parameters + sip_params = DailyRoomSipParams( + display_name=caller_phone, + video=False, + sip_mode="dial-in", + num_endpoints=1, + ) + + # Create room properties with SIP enabled + properties = DailyRoomProperties( + sip=sip_params, + enable_dialout=True, # Needed for outbound calls if you expand the bot + enable_chat=False, # No need for chat in a voice bot + start_video_off=True, # Voice only + ) + + # Create room parameters + params = DailyRoomParams(properties=properties) + + # Create the room + try: + room = await daily_helper.create_room(params=params) + print(f"Created room: {room.url} with SIP endpoint: {room.config.sip_endpoint}") + + # Get token for the bot to join + token = await daily_helper.get_token(room.url, 24 * 60 * 60) # 24 hours validity + + return {"room_url": room.url, "token": token, "sip_endpoint": room.config.sip_endpoint} + except Exception as e: + print(f"Error creating room: {e}") + raise diff --git a/examples/phone-chatbot-daily-twilio-sip/README.md b/examples/phone-chatbot/daily-twilio-sip-dial-in/README.md similarity index 95% rename from examples/phone-chatbot-daily-twilio-sip/README.md rename to examples/phone-chatbot/daily-twilio-sip-dial-in/README.md index d2cde8186..1d4cfb325 100644 --- a/examples/phone-chatbot-daily-twilio-sip/README.md +++ b/examples/phone-chatbot/daily-twilio-sip-dial-in/README.md @@ -1,11 +1,11 @@ -# Daily + Twilio SIP Voice Bot +# Daily + Twilio SIP dial-in Voice Bot This project demonstrates how to create a voice bot that can receive phone calls via Twilio and use Daily's SIP capabilities to enable voice conversations. ## How It Works 1. Twilio receives an incoming call to your phone number -2. Twilio calls your webhook server (`/call` endpoint) +2. Twilio calls your webhook server (`/start` endpoint) 3. The server creates a Daily room with SIP capabilities 4. The server starts the bot process with the room details 5. The caller is put on hold with music @@ -44,12 +44,12 @@ cp .env.example .env In the Twilio console: - Go to your phone number's configuration -- Set the webhook for "A Call Comes In" to your server's URL + "/call" +- Set the webhook for "A Call Comes In" to your server's URL + "/start" - For local testing, you can use ngrok to expose your local server ```bash -ngrok http 8000 -# Then use the provided URL (e.g., https://abc123.ngrok.io/call) in Twilio +ngrok http 7860 +# Then use the provided URL (e.g., https://abc123.ngrok.io/start) in Twilio ``` ## Running the Server diff --git a/examples/phone-chatbot-daily-twilio-sip/bot.py b/examples/phone-chatbot/daily-twilio-sip-dial-in/bot.py similarity index 100% rename from examples/phone-chatbot-daily-twilio-sip/bot.py rename to examples/phone-chatbot/daily-twilio-sip-dial-in/bot.py diff --git a/examples/phone-chatbot-daily-twilio-sip/env.example b/examples/phone-chatbot/daily-twilio-sip-dial-in/env.example similarity index 100% rename from examples/phone-chatbot-daily-twilio-sip/env.example rename to examples/phone-chatbot/daily-twilio-sip-dial-in/env.example diff --git a/examples/phone-chatbot-daily-twilio-sip/requirements.txt b/examples/phone-chatbot/daily-twilio-sip-dial-in/requirements.txt similarity index 92% rename from examples/phone-chatbot-daily-twilio-sip/requirements.txt rename to examples/phone-chatbot/daily-twilio-sip-dial-in/requirements.txt index 623893a63..cd12dd07a 100644 --- a/examples/phone-chatbot-daily-twilio-sip/requirements.txt +++ b/examples/phone-chatbot/daily-twilio-sip-dial-in/requirements.txt @@ -3,3 +3,4 @@ fastapi==0.115.6 uvicorn python-dotenv twilio +aiohttp \ No newline at end of file diff --git a/examples/phone-chatbot-daily-twilio-sip/server.py b/examples/phone-chatbot/daily-twilio-sip-dial-in/server.py similarity index 97% rename from examples/phone-chatbot-daily-twilio-sip/server.py rename to examples/phone-chatbot/daily-twilio-sip-dial-in/server.py index 47b7a4a22..4728e980d 100644 --- a/examples/phone-chatbot-daily-twilio-sip/server.py +++ b/examples/phone-chatbot/daily-twilio-sip-dial-in/server.py @@ -30,7 +30,7 @@ async def lifespan(app: FastAPI): app = FastAPI(lifespan=lifespan) -@app.post("/call", response_class=PlainTextResponse) +@app.post("/start", response_class=PlainTextResponse) async def handle_call(request: Request): """Handle incoming Twilio call webhook.""" print("Received call webhook from Twilio") @@ -111,6 +111,6 @@ async def health_check(): if __name__ == "__main__": # Run the server - port = int(os.getenv("PORT", "8000")) + port = int(os.getenv("PORT", "7860")) print(f"Starting server on port {port}") uvicorn.run("server:app", host="0.0.0.0", port=port, reload=True) diff --git a/examples/phone-chatbot-daily-twilio-sip/utils/daily_helpers.py b/examples/phone-chatbot/daily-twilio-sip-dial-in/utils/daily_helpers.py similarity index 100% rename from examples/phone-chatbot-daily-twilio-sip/utils/daily_helpers.py rename to examples/phone-chatbot/daily-twilio-sip-dial-in/utils/daily_helpers.py diff --git a/examples/phone-chatbot/daily-twilio-sip-dial-out/README.md b/examples/phone-chatbot/daily-twilio-sip-dial-out/README.md new file mode 100644 index 000000000..8cdd52b86 --- /dev/null +++ b/examples/phone-chatbot/daily-twilio-sip-dial-out/README.md @@ -0,0 +1,152 @@ + + +# Daily + Twilio SIP dial-out Voice Bot + +This project demonstrates how to create a voice bot that can make phone calls via Twilio and use Daily's SIP capabilities to enable voice conversations. + +## How it works + +1. The server file receives a curl request with the SIP uri to dial out to +2. The server creates a Daily room with SIP capabilities +3. The server starts the bot process with the room details +4. When the bot has joined, it starts the dial-out process and dials out to the SIP uri provided in the curl request +5. Twilio receives the request, and the provided TWIML processes the SIP uri +6. Twilio then rings the number found within the SIP uri +7. When the user answers the phone, the user is brought into the call +8. The end user and the bot are connected, and the bot handles the conversation + +## Prerequisites + +- A Daily account with an API key +- A Twilio account with a phone number that supports voice and a correctly configured SIP domain +- OpenAI API key for the bot's intelligence +- Cartesia API key for text-to-speech + +## Setup + +1. Create a virtual environment and install dependencies + +```bash +python -m venv venv +source venv/bin/activate # On Windows: venv\Scripts\activate +pip install -r requirements.txt +``` + +2. Set up environment variables + +Copy the example file and fill in your API keys: + +```bash +cp .env.example .env +# Edit .env with your API keys +``` + +3. Create a TwiML Bin + +Visit this link to create your [TwiML Bin](https://www.twilio.com/docs/serverless/twiml-bins) + +- Login to the account that has your purchased Twilio phone number +- Press the plus button on the TwiML Bin dashboard to write a new TwiML that Twilio will host for you +- Give it a friendly name. For example "daily sip uri twiml bin" +- For the TWIML code, use something like: + +```xml + + + {{#e164}}{{To}}{{/e164}} + +``` + +- callerId must be a valid number that you own on [Twilio](https://console.twilio.com/us1/develop/phone-numbers/manage/incoming) +- Save the file. We will use this when creating the SIP domain + +4. Create and configure a programmable SIP domain + +- Visit this link to [create a new SIP domain:](https://console.twilio.com/us1/develop/voice/manage/sip-domains?frameUrl=%2Fconsole%2Fvoice%2Fsip%2Fendpoints%3Fx-target-region%3Dus1) +- Press the plus button to create a new SIP domain +- Give the SIP domain a friendly name. For example "Daily SIP domain" +- Specify a SIP URI, for example "daily.sip.twilio.com" +- Under "Voice Authentication", press the plus button next to IP Access Control Lists. We are going to white list the entire IP spectrum +- Give it a friendly name such as "first half" +- For CIDR Network Address specify 0.0.0.0 and for the subnet specify 1 +- Again, specify "first half" for the friendly name and click "Create ACL" +- Now let's do the same again and add another IP Access Control List by pressing the plus button +- Give it a friendly name such as "second half". +- For the CIDR Network Address specify 128.0.0.0 and for the subnet specify 1 +- Lastly, specify the friendly name "second half" again +- Make sure both IP Access control list appears selected in the dropdown +- Under "Call Control Configuration", specify the following: +- Configure with: Webhooks, TwiML Bins, Functions, Studio, Proxy +- A call comes in: TwiML Bin > Select the name of the TwiML bin you made earlier +- Leave everything else blank and scroll to the bottom of the page. Click save + +## Running the Server + +Start the webhook server: + +```bash +python server.py +``` + +## Testing + +With server.py running, send the following curl command from your terminal: + +```bash +curl -X POST "http://127.0.0.1:7860/start" \ + -H "Content-Type: application/json" \ + -d '{ + "dialout_settings": { + "sip_uri": "sip:+1234567891@daily.sip.twilio.com" + } + }' +``` + +- Replace the phone number (Starting with +1) with the phone number you want to ring +- Replace daily with the SIP domain you configured previously + +The server should make a room. The bot will join the room and then dial out to the SIP URI provided. Answer the call to speak with the bot. + +## Customizing the Bot + +You can customize the bot's behavior by modifying the system prompt in `bot.py`. + +## Handling Multiple SIP Endpoints + +Note that normally calls only require a single SIP endpoint. If you are planning to forward the call to a different number, you will need to set up 2 SIP endpoints: one for the initial call and one for the forwarded call. + +## Daily dial-out configuration + +The bot configures the Daily rooms with dial-out capabilities using these settings. Note: You also need dial-out to be enabled on the domain, as mentioned earlier on in the README. + +```python +properties = DailyRoomProperties( + sip=sip_params, + enable_dialout=True, # Needed for outbound calls if you expand the bot + enable_chat=False, # No need for chat in a voice bot + start_video_off=True, # Voice only +) +``` + +## Troubleshooting + +### I get an error about dial-out not being enabled + +- Check that your room has `enable_dialout=True` set +- Check that your meeting token is an owner token (The bot does this for you automatically) +- Check that the SIP URI is correct +- Check that the phone number you are trying to ring is correct + +### I'm stuck setting up my Twilio account + +- You can reference this [Notion doc](https://dailyco.notion.site/PUBLIC-Doc-Integration-Twilio-PSTN-Daily-s-SIP-Dialout-1cfdaed630f5458d9d4fc0e3f29ec559) to find more information on how to set up Twilio, as well as use webhooks instead of TwiML Bins + +### Call connects but no bot is heard + +- Ensure your Daily API key is correct and has SIP capabilities +- Verify that the Cartesia API key and voice ID are correct + +### Bot starts but disconnects immediately + +- Check the Daily logs for any error messages +- Ensure your server has stable internet connectivity diff --git a/examples/phone-chatbot/simple_dialout.py b/examples/phone-chatbot/daily-twilio-sip-dial-out/bot.py similarity index 53% rename from examples/phone-chatbot/simple_dialout.py rename to examples/phone-chatbot/daily-twilio-sip-dial-out/bot.py index 26e754521..5f65a821e 100644 --- a/examples/phone-chatbot/simple_dialout.py +++ b/examples/phone-chatbot/daily-twilio-sip-dial-out/bot.py @@ -3,26 +3,28 @@ # # SPDX-License-Identifier: BSD 2-Clause License # + +"""simple_dialout.py. + +Simple Dial-out Bot. +""" + import argparse import asyncio +import json import os import sys +from typing import Any -from call_connection_manager import CallConfigManager from dotenv import load_dotenv from loguru import logger -from pipecat.adapters.schemas.function_schema import FunctionSchema -from pipecat.adapters.schemas.tools_schema import ToolsSchema from pipecat.audio.vad.silero import SileroVADAnalyzer -from pipecat.frames.frames import EndTaskFrame from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext -from pipecat.processors.frame_processor import FrameDirection from pipecat.services.cartesia.tts import CartesiaTTSService -from pipecat.services.llm_service import FunctionCallParams from pipecat.services.openai.llm import OpenAILLMService from pipecat.transports.services.daily import DailyParams, DailyTransport @@ -35,19 +37,41 @@ daily_api_key = os.getenv("DAILY_API_KEY", "") daily_api_url = os.getenv("DAILY_API_URL", "https://api.daily.co/v1") -async def main( +async def run_bot( room_url: str, token: str, body: dict, -): +) -> None: + """Run the voice bot with the given parameters. + + Args: + room_url: The Daily room URL + token: The Daily room token + body: Body passed to the bot from the webhook + + """ # ------------ CONFIGURATION AND SETUP ------------ + logger.info(f"Starting bot with room: {room_url}") + logger.info(f"Token: {token}") + logger.info(f"Body: {body}") + # Parse the body to get the dial-in settings + body_data = json.loads(body) - # Create a config manager using the provided body - call_config_manager = CallConfigManager.from_json_string(body) if body else CallConfigManager() + # Check if the body contains dial-in settings + logger.debug(f"Body data: {body_data}") - # Get important configuration values - dialout_settings = call_config_manager.get_dialout_settings() - test_mode = call_config_manager.is_test_mode() + if not body_data.get("dialout_settings"): + logger.error("Dial-out settings not found in the body data") + return + + dialout_settings = body_data["dialout_settings"] + + if not dialout_settings.get("sip_uri"): + logger.error("Dial-out sip_uri not found in the dial-out settings") + return + + # Extract sip_uri + sip_uri = dialout_settings["sip_uri"] # ------------ TRANSPORT SETUP ------------ @@ -65,7 +89,7 @@ async def main( transport = DailyTransport( room_url, token, - "Simple Dial-out Bot", + "Phone Bot", transport_params, ) @@ -75,39 +99,24 @@ async def main( voice_id="b7d50908-b17c-442d-ad8d-810c63997ed9", # Use Helpful Woman voice by default ) - # ------------ FUNCTION DEFINITIONS ------------ - - async def terminate_call(params: FunctionCallParams): - """Function the bot can call to terminate the call upon completion of a voicemail message.""" - await params.llm.queue_frame(EndTaskFrame(), FrameDirection.UPSTREAM) - - # Define function schemas for tools - terminate_call_function = FunctionSchema( - name="terminate_call", - description="Call this function to terminate the call.", - properties={}, - required=[], - ) - - # Create tools schema - tools = ToolsSchema(standard_tools=[terminate_call_function]) - # ------------ LLM AND CONTEXT SETUP ------------ - # Set up the system instruction for the LLM - system_instruction = """You are Chatbot, a friendly, helpful robot. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by introducing yourself. If the user ends the conversation, **IMMEDIATELY** call the `terminate_call` function. """ - # Initialize LLM llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY")) - # Register functions with the LLM - llm.register_function("terminate_call", terminate_call) - # Create system message and initialize messages list - messages = [call_config_manager.create_system_message(system_instruction)] - + messages = [ + { + "role": "system", + "content": ( + "You are a friendly phone assistant. Your responses will be read aloud, " + "so keep them concise and conversational. Avoid special characters or " + "formatting. Begin by greeting the caller and asking how you can help them today." + ), + }, + ] # Initialize LLM context and aggregator - context = OpenAILLMContext(messages, tools) + context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) # ------------ PIPELINE SETUP ------------ @@ -127,14 +136,34 @@ async def main( # Create pipeline task task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True)) + # ------------ RETRY LOGIC VARIABLES ------------ + max_retries = 5 + retry_count = 0 + dialout_successful = False + + # Build dialout parameters conditionally + dialout_params = {"sipUri": sip_uri} + + logger.debug(f"Dialout parameters: {dialout_params}") + + async def attempt_dialout(): + """Attempt to start dialout with retry logic.""" + nonlocal retry_count, dialout_successful + + if retry_count < max_retries and not dialout_successful: + retry_count += 1 + logger.info(f"Attempting dialout (attempt {retry_count}/{max_retries}) to: {sip_uri}") + await transport.start_dialout(dialout_params) + else: + logger.error(f"Maximum retry attempts ({max_retries}) reached. Giving up on dialout.") + # ------------ EVENT HANDLERS ------------ @transport.event_handler("on_joined") async def on_joined(transport, data): - # Start dialout if needed - if not test_mode and dialout_settings: - logger.debug("Dialout settings detected; starting dialout") - await call_config_manager.start_dialout(transport, dialout_settings) + # Start initial dialout attempt + logger.debug(f"Dialout settings detected; starting dialout to number: {sip_uri}") + await attempt_dialout() @transport.event_handler("on_dialout_connected") async def on_dialout_connected(transport, data): @@ -142,17 +171,27 @@ async def main( @transport.event_handler("on_dialout_answered") async def on_dialout_answered(transport, data): + nonlocal dialout_successful logger.debug(f"Dial-out answered: {data}") + dialout_successful = True # Mark as successful to stop retries # Automatically start capturing transcription for the participant await transport.capture_participant_transcription(data["sessionId"]) # The bot will wait to hear the user before the bot speaks + @transport.event_handler("on_dialout_error") + async def on_dialout_error(transport, data: Any): + logger.error(f"Dial-out error (attempt {retry_count}/{max_retries}): {data}") + + if retry_count < max_retries: + logger.info(f"Retrying dialout") + await attempt_dialout() + else: + logger.error(f"All {max_retries} dialout attempts failed. Stopping bot.") + await task.cancel() + @transport.event_handler("on_first_participant_joined") async def on_first_participant_joined(transport, participant): - if test_mode: - logger.debug(f"First participant joined: {participant['id']}") - await transport.capture_participant_transcription(participant["id"]) - # The bot will wait to hear the user before the bot speaks + logger.debug(f"First participant joined: {participant['id']}") @transport.event_handler("on_participant_left") async def on_participant_left(transport, participant, reason): @@ -161,14 +200,12 @@ async def main( # ------------ RUN PIPELINE ------------ - if test_mode: - logger.debug("Running in test mode (can be tested in Daily Prebuilt)") - runner = PipelineRunner() await runner.run(task) -if __name__ == "__main__": +async def main(): + """Parse command line arguments and run the bot.""" parser = argparse.ArgumentParser(description="Simple Dial-out Bot") parser.add_argument("-u", "--url", type=str, help="Room URL") parser.add_argument("-t", "--token", type=str, help="Room Token") @@ -176,9 +213,16 @@ if __name__ == "__main__": args = parser.parse_args() - # Log the arguments for debugging - logger.info(f"Room URL: {args.url}") - logger.info(f"Token: {args.token}") - logger.info(f"Body provided: {bool(args.body)}") + logger.debug(f"url: {args.url}") + logger.debug(f"token: {args.token}") + logger.debug(f"body: {args.body}") + if not all([args.url, args.token, args.body]): + logger.error("All arguments (-u, -t, -b) are required") + parser.print_help() + sys.exit(1) - asyncio.run(main(args.url, args.token, args.body)) + await run_bot(args.url, args.token, args.body) + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/examples/phone-chatbot/daily-twilio-sip-dial-out/env.example b/examples/phone-chatbot/daily-twilio-sip-dial-out/env.example new file mode 100644 index 000000000..f48a6a810 --- /dev/null +++ b/examples/phone-chatbot/daily-twilio-sip-dial-out/env.example @@ -0,0 +1,7 @@ +# Daily credentials +DAILY_API_KEY=your_daily_api_key +DAILY_API_URL=https://api.daily.co/v1 + +# Service keys +OPENAI_API_KEY=your_openai_api_key +CARTESIA_API_KEY=your_cartesia_api_key \ No newline at end of file diff --git a/examples/phone-chatbot/daily-twilio-sip-dial-out/requirements.txt b/examples/phone-chatbot/daily-twilio-sip-dial-out/requirements.txt new file mode 100644 index 000000000..858a8d7b6 --- /dev/null +++ b/examples/phone-chatbot/daily-twilio-sip-dial-out/requirements.txt @@ -0,0 +1,6 @@ +pipecat-ai[daily,cartesia,elevenlabs,deepgram,openai,silero] +fastapi==0.115.6 +uvicorn +python-dotenv +python-multipart +aiohttp \ No newline at end of file diff --git a/examples/phone-chatbot/daily-twilio-sip-dial-out/server.py b/examples/phone-chatbot/daily-twilio-sip-dial-out/server.py new file mode 100644 index 000000000..42591094d --- /dev/null +++ b/examples/phone-chatbot/daily-twilio-sip-dial-out/server.py @@ -0,0 +1,140 @@ +# +# Copyright (c) 2024–2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +"""server.py. + +Webhook server to handle webhook coming from Daily, create a Daily room and start the bot. +""" + +import json +import os +import shlex +import subprocess +from contextlib import asynccontextmanager + +import aiohttp +import uvicorn +from dotenv import load_dotenv +from fastapi import FastAPI, HTTPException, Request +from fastapi.responses import JSONResponse +from utils.daily_helpers import create_daily_room + +load_dotenv() + +# ----------------- API ----------------- # + + +@asynccontextmanager +async def lifespan(app: FastAPI): + # Create aiohttp session to be used for Daily API calls + app.state.session = aiohttp.ClientSession() + yield + # Close session when shutting down + await app.state.session.close() + + +app = FastAPI(lifespan=lifespan) + + +def extract_phone_from_sip_uri(sip_uri): + """Extract phone number from SIP URI. + + Args: + sip_uri: SIP URI in format "sip:+17868748498@daily-twilio-integration.sip.twilio.com" + + Returns: + Phone number string (e.g., "+17868748498") or None if invalid format + """ + if not sip_uri or not isinstance(sip_uri, str): + return None + + if sip_uri.startswith("sip:") and "@" in sip_uri: + phone_part = sip_uri[4:] # Remove 'sip:' prefix + caller_phone = phone_part.split("@")[0] # Get everything before '@' + return caller_phone + return None + + +@app.post("/start") +async def handle_incoming_daily_webhook(request: Request) -> JSONResponse: + """Handle dial-out request.""" + print("Received webhook from Daily") + + # Get the dial-in properties from the request + try: + data = await request.json() + if "test" in data: + # Pass through any webhook checks + return JSONResponse({"test": True}) + + if not data["dialout_settings"]: + raise HTTPException( + status_code=400, detail="Missing 'dialout_settings' in the request body" + ) + + if not data["dialout_settings"].get("sip_uri"): + raise HTTPException(status_code=400, detail="Missing 'sip_uri' in dialout_settings") + + # Extract the phone number we want to dial out to + sip_uri = str(data["dialout_settings"]["sip_uri"]) + caller_phone = extract_phone_from_sip_uri(sip_uri) + print(f"SIP URI: {sip_uri}") + print(f"Processing sip call to {caller_phone}") + + # Create a Daily room with dial-in capabilities + try: + room_details = await create_daily_room(request.app.state.session, caller_phone) + except Exception as e: + print(f"Error creating Daily room: {e}") + raise HTTPException(status_code=500, detail=f"Failed to create Daily room: {str(e)}") + + room_url = room_details["room_url"] + token = room_details["token"] + print(f"Created Daily room: {room_url} with token: {token}") + + body_json = json.dumps(data) + + bot_cmd = f"python3 -m bot -u {room_url} -t {token} -b {shlex.quote(body_json)}" + + try: + # CHANGE: Keep stdout/stderr for debugging + # Start the bot in the background but capture output + subprocess.Popen( + bot_cmd, + shell=True, + # Don't redirect output so we can see logs + # stdout=subprocess.DEVNULL, + # stderr=subprocess.DEVNULL + ) + print(f"Started bot process with command: {bot_cmd}") + except Exception as e: + print(f"Error starting bot: {e}") + raise HTTPException(status_code=500, detail=f"Failed to start bot: {str(e)}") + + except HTTPException: + raise + except Exception as e: + print(f"Unexpected error: {str(e)}") + raise HTTPException(status_code=500, detail=f"Server error: {str(e)}") + + # Grab a token for the user to join with + return JSONResponse({"room_url": room_url, "token": token}) + + +@app.get("/health") +async def health_check(): + """Simple health check endpoint.""" + return {"status": "healthy"} + + +# ----------------- Main ----------------- # + + +if __name__ == "__main__": + # Run the server + port = int(os.getenv("PORT", "7860")) + print(f"Starting server on port {port}") + uvicorn.run("server:app", host="0.0.0.0", port=port, reload=True) diff --git a/examples/phone-chatbot/daily-twilio-sip-dial-out/utils/daily_helpers.py b/examples/phone-chatbot/daily-twilio-sip-dial-out/utils/daily_helpers.py new file mode 100644 index 000000000..6451fc4c4 --- /dev/null +++ b/examples/phone-chatbot/daily-twilio-sip-dial-out/utils/daily_helpers.py @@ -0,0 +1,76 @@ +"""Helper functions for interacting with the Daily API.""" + +import os +from typing import Dict, Optional + +import aiohttp +from dotenv import load_dotenv + +from pipecat.transports.services.helpers.daily_rest import ( + DailyRESTHelper, + DailyRoomParams, + DailyRoomProperties, + DailyRoomSipParams, +) + +load_dotenv() + + +# Initialize Daily API helper +async def get_daily_helper(session: Optional[aiohttp.ClientSession] = None) -> DailyRESTHelper: + """Get a Daily REST helper with the configured API key.""" + if session is None: + session = aiohttp.ClientSession() + + return DailyRESTHelper( + daily_api_key=os.getenv("DAILY_API_KEY", ""), + daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"), + aiohttp_session=session, + ) + + +async def create_daily_room( + session: Optional[aiohttp.ClientSession] = None, caller_phone: str = "unknown-caller" +) -> Dict[str, str]: + """Create a Daily room with SIP capabilities for phone calls. + + Args: + session: Optional aiohttp session to use for API calls + caller_phone: The phone number of the caller to use in display name + + Returns: + Dictionary with room URL, token, and SIP endpoint + """ + daily_helper = await get_daily_helper(session) + + # Configure SIP parameters + sip_params = DailyRoomSipParams( + display_name=caller_phone, + video=False, + sip_mode="dial-in", + num_endpoints=1, + ) + + # Create room properties with SIP enabled + properties = DailyRoomProperties( + sip=sip_params, + enable_dialout=True, # Needed for outbound calls if you expand the bot + enable_chat=False, # No need for chat in a voice bot + start_video_off=True, # Voice only + ) + + # Create room parameters + params = DailyRoomParams(properties=properties) + + # Create the room + try: + room = await daily_helper.create_room(params=params) + print(f"Created room: {room.url} with SIP endpoint: {room.config.sip_endpoint}") + + # Get token for the bot to join + token = await daily_helper.get_token(room.url, 24 * 60 * 60) # 24 hours validity + + return {"room_url": room.url, "token": token, "sip_endpoint": room.config.sip_endpoint} + except Exception as e: + print(f"Error creating room: {e}") + raise diff --git a/examples/phone-chatbot/env.example b/examples/phone-chatbot/env.example deleted file mode 100644 index c5e049ead..000000000 --- a/examples/phone-chatbot/env.example +++ /dev/null @@ -1,10 +0,0 @@ -DAILY_SAMPLE_ROOM_URL=https://yourdomain.daily.co/yourroom # (optional: for joining the bot to the same room repeatedly for local dev) -DAILY_API_KEY= -DAILY_API_URL=https://api.daily.co/v1 -DEEPGRAM_API_KEY= -OPENAI_API_KEY= -GOOGLE_API_KEY -CARTESIA_API_KEY= -DIAL_IN_FROM_NUMBER= -DIAL_OUT_TO_NUMBER= -OPERATOR_NUMBER= \ No newline at end of file diff --git a/examples/phone-chatbot/requirements.txt b/examples/phone-chatbot/requirements.txt deleted file mode 100644 index a70e1113f..000000000 --- a/examples/phone-chatbot/requirements.txt +++ /dev/null @@ -1,5 +0,0 @@ -pipecat-ai[daily,cartesia,deepgram,openai,google,silero] -fastapi==0.115.6 -uvicorn -python-dotenv -python-multipart diff --git a/examples/phone-chatbot/simple_dialin.py b/examples/phone-chatbot/simple_dialin.py deleted file mode 100644 index 5842b97f1..000000000 --- a/examples/phone-chatbot/simple_dialin.py +++ /dev/null @@ -1,192 +0,0 @@ -# -# Copyright (c) 2024–2025, Daily -# -# SPDX-License-Identifier: BSD 2-Clause License -# -import argparse -import asyncio -import os -import sys - -from call_connection_manager import CallConfigManager, SessionManager -from dotenv import load_dotenv -from loguru import logger - -from pipecat.adapters.schemas.function_schema import FunctionSchema -from pipecat.adapters.schemas.tools_schema import ToolsSchema -from pipecat.audio.vad.silero import SileroVADAnalyzer -from pipecat.frames.frames import EndTaskFrame -from pipecat.pipeline.pipeline import Pipeline -from pipecat.pipeline.runner import PipelineRunner -from pipecat.pipeline.task import PipelineParams, PipelineTask -from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext -from pipecat.processors.frame_processor import FrameDirection -from pipecat.services.cartesia.tts import CartesiaTTSService -from pipecat.services.llm_service import FunctionCallParams -from pipecat.services.openai.llm import OpenAILLMService -from pipecat.transports.services.daily import DailyDialinSettings, DailyParams, DailyTransport - -load_dotenv(override=True) - -logger.remove(0) -logger.add(sys.stderr, level="DEBUG") - -daily_api_key = os.getenv("DAILY_API_KEY", "") -daily_api_url = os.getenv("DAILY_API_URL", "https://api.daily.co/v1") - - -async def main( - room_url: str, - token: str, - body: dict, -): - # ------------ CONFIGURATION AND SETUP ------------ - - # Create a config manager using the provided body - call_config_manager = CallConfigManager.from_json_string(body) if body else CallConfigManager() - - # Get important configuration values - test_mode = call_config_manager.is_test_mode() - - # Get dialin settings if present - dialin_settings = call_config_manager.get_dialin_settings() - - # Initialize the session manager - session_manager = SessionManager() - - # ------------ TRANSPORT SETUP ------------ - - # Set up transport parameters - if test_mode: - logger.info("Running in test mode") - transport_params = DailyParams( - api_url=daily_api_url, - api_key=daily_api_key, - audio_in_enabled=True, - audio_out_enabled=True, - video_out_enabled=False, - vad_analyzer=SileroVADAnalyzer(), - transcription_enabled=True, - ) - else: - daily_dialin_settings = DailyDialinSettings( - call_id=dialin_settings.get("call_id"), call_domain=dialin_settings.get("call_domain") - ) - transport_params = DailyParams( - api_url=daily_api_url, - api_key=daily_api_key, - dialin_settings=daily_dialin_settings, - audio_in_enabled=True, - audio_out_enabled=True, - video_out_enabled=False, - vad_analyzer=SileroVADAnalyzer(), - transcription_enabled=True, - ) - - # Initialize transport with Daily - transport = DailyTransport( - room_url, - token, - "Simple Dial-in Bot", - transport_params, - ) - - # Initialize TTS - tts = CartesiaTTSService( - api_key=os.getenv("CARTESIA_API_KEY", ""), - voice_id="b7d50908-b17c-442d-ad8d-810c63997ed9", # Use Helpful Woman voice by default - ) - - # ------------ FUNCTION DEFINITIONS ------------ - - async def terminate_call(params: FunctionCallParams): - """Function the bot can call to terminate the call upon completion of a voicemail message.""" - if session_manager: - # Mark that the call was terminated by the bot - session_manager.call_flow_state.set_call_terminated() - - # Then end the call - await params.llm.queue_frame(EndTaskFrame(), FrameDirection.UPSTREAM) - - # Define function schemas for tools - terminate_call_function = FunctionSchema( - name="terminate_call", - description="Call this function to terminate the call.", - properties={}, - required=[], - ) - - # Create tools schema - tools = ToolsSchema(standard_tools=[terminate_call_function]) - - # ------------ LLM AND CONTEXT SETUP ------------ - - # Set up the system instruction for the LLM - system_instruction = """You are Chatbot, a friendly, helpful robot. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by introducing yourself. If the user ends the conversation, **IMMEDIATELY** call the `terminate_call` function. """ - - # Initialize LLM - llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY")) - - # Register functions with the LLM - llm.register_function("terminate_call", terminate_call) - - # Create system message and initialize messages list - messages = [call_config_manager.create_system_message(system_instruction)] - - # Initialize LLM context and aggregator - context = OpenAILLMContext(messages, tools) - context_aggregator = llm.create_context_aggregator(context) - - # ------------ PIPELINE SETUP ------------ - - # Build pipeline - pipeline = Pipeline( - [ - transport.input(), # Transport user input - context_aggregator.user(), # User responses - llm, # LLM - tts, # TTS - transport.output(), # Transport bot output - context_aggregator.assistant(), # Assistant spoken responses - ] - ) - - # Create pipeline task - task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True)) - - # ------------ EVENT HANDLERS ------------ - - @transport.event_handler("on_first_participant_joined") - async def on_first_participant_joined(transport, participant): - logger.debug(f"First participant joined: {participant['id']}") - await transport.capture_participant_transcription(participant["id"]) - await task.queue_frames([context_aggregator.user().get_context_frame()]) - - @transport.event_handler("on_participant_left") - async def on_participant_left(transport, participant, reason): - logger.debug(f"Participant left: {participant}, reason: {reason}") - await task.cancel() - - # ------------ RUN PIPELINE ------------ - - if test_mode: - logger.debug("Running in test mode (can be tested in Daily Prebuilt)") - - runner = PipelineRunner() - await runner.run(task) - - -if __name__ == "__main__": - parser = argparse.ArgumentParser(description="Simple Dial-in Bot") - parser.add_argument("-u", "--url", type=str, help="Room URL") - parser.add_argument("-t", "--token", type=str, help="Room Token") - parser.add_argument("-b", "--body", type=str, help="JSON configuration string") - - args = parser.parse_args() - - # Log the arguments for debugging - logger.info(f"Room URL: {args.url}") - logger.info(f"Token: {args.token}") - logger.info(f"Body provided: {bool(args.body)}") - - asyncio.run(main(args.url, args.token, args.body)) diff --git a/examples/simple-chatbot/client/android/gradle/libs.versions.toml b/examples/simple-chatbot/client/android/gradle/libs.versions.toml index 1793b8949..5bc7b5db9 100644 --- a/examples/simple-chatbot/client/android/gradle/libs.versions.toml +++ b/examples/simple-chatbot/client/android/gradle/libs.versions.toml @@ -2,7 +2,7 @@ accompanistPermissions = "0.34.0" agp = "8.7.3" constraintlayoutCompose = "1.1.0" -pipecatClientDaily = "0.3.2" +pipecatClientDaily = "0.3.7" kotlin = "2.0.20" coreKtx = "1.15.0" lifecycleRuntimeKtx = "2.8.7" diff --git a/examples/simple-chatbot/client/android/settings.gradle.kts b/examples/simple-chatbot/client/android/settings.gradle.kts index 03bf7ef96..d416e5d43 100644 --- a/examples/simple-chatbot/client/android/settings.gradle.kts +++ b/examples/simple-chatbot/client/android/settings.gradle.kts @@ -16,6 +16,7 @@ dependencyResolutionManagement { repositories { google() mavenCentral() + mavenLocal() } } diff --git a/examples/simple-chatbot/client/android/simple-chatbot-client/src/main/java/ai/pipecat/simple_chatbot_client/VoiceClientManager.kt b/examples/simple-chatbot/client/android/simple-chatbot-client/src/main/java/ai/pipecat/simple_chatbot_client/VoiceClientManager.kt index cee65612e..6f122f310 100644 --- a/examples/simple-chatbot/client/android/simple-chatbot-client/src/main/java/ai/pipecat/simple_chatbot_client/VoiceClientManager.kt +++ b/examples/simple-chatbot/client/android/simple-chatbot-client/src/main/java/ai/pipecat/simple_chatbot_client/VoiceClientManager.kt @@ -16,6 +16,7 @@ import ai.pipecat.client.types.ServiceConfig import ai.pipecat.client.types.Tracks import ai.pipecat.client.types.Transcript import ai.pipecat.client.types.TransportState +import ai.pipecat.client.types.Value import ai.pipecat.simple_chatbot_client.utils.Timestamp import android.content.Context import android.util.Log @@ -88,6 +89,10 @@ class VoiceClientManager(private val context: Context) { } } + override fun onServerMessage(data: Value) { + Log.i(TAG, "onServerMessage: $data") + } + override fun onBotReady(version: String, config: List) { Log.i(TAG, "Bot ready. Version $version, config: $config") diff --git a/examples/twilio-chatbot/README.md b/examples/twilio-chatbot/README.md index d06fd7f85..9c7a4be95 100644 --- a/examples/twilio-chatbot/README.md +++ b/examples/twilio-chatbot/README.md @@ -110,31 +110,6 @@ To start a call, simply make a call to your configured Twilio phone number. The ## Testing -It is also possible to automatically test the server without making phone calls by using a software client. - -First, update `templates/streams.xml` to point to your server's websocket endpoint. For example: - -``` - - - - - - - -``` - -Then, start the server with `-t` to indicate we are testing: - -```sh -# Make sure you’re in the project directory and your virtual environment is activated -python server.py -t -``` - -Finally, just point the client to the server's URL: - -```sh -python client.py -u http://localhost:8765 -c 2 -``` - -where `-c` allows you to create multiple concurrent clients. +It is also possible to test the server without making phone calls by using one of these clients. +- [python](client/python/README.md): This Python client enables automated testing of the server via WebSocket without the need to make actual phone calls. +- [typescript](client/typescript/README.md): This typescript client enables manual testing of the server via WebSocket without the need to make actual phone calls. \ No newline at end of file diff --git a/examples/twilio-chatbot/client/python/README.md b/examples/twilio-chatbot/client/python/README.md new file mode 100644 index 000000000..18e695382 --- /dev/null +++ b/examples/twilio-chatbot/client/python/README.md @@ -0,0 +1,39 @@ +# Python Client for Server Testing + +This Python client enables automated testing of the server via WebSocket without the need to make actual phone calls. + +## Setup Instructions + +### 1. Configure the Stream Template + +Edit the `templates/streams.xml` file to point to your server’s WebSocket endpoint. For example: + +```xml + + + + + + + +``` + +### 2. Start the Server in Test Mode + +Run the server with the `-t` flag to indicate test mode: + +```sh +# Ensure you're in the project directory and your virtual environment is activated +python server.py -t +``` + +### 3. Run the Client + +Start the client and point it to the server URL: + +```sh +python client.py -u http://localhost:8765 -c 2 +``` + +- `-u`: Server URL (default is `http://localhost:8765`) +- `-c`: Number of concurrent client connections (e.g., 2) diff --git a/examples/twilio-chatbot/client.py b/examples/twilio-chatbot/client/python/client.py similarity index 100% rename from examples/twilio-chatbot/client.py rename to examples/twilio-chatbot/client/python/client.py diff --git a/examples/twilio-chatbot/client/typescript/README.md b/examples/twilio-chatbot/client/typescript/README.md new file mode 100644 index 000000000..a2dd7b05b --- /dev/null +++ b/examples/twilio-chatbot/client/typescript/README.md @@ -0,0 +1,27 @@ +# Typescript Client for Server Testing + +This typescript client enables manual testing of the server via WebSocket without the need to make actual phone calls. + +## Setup + +1. Run the bot server. See the [server README](../../README). + +2. Navigate to the `client/typescript` directory: + +```bash +cd client/typescript +``` + +3. Install dependencies: + +```bash +npm install +``` + +4. Run the client app: + +``` +npm run dev +``` + +5. Visit http://localhost:5173 in your browser. diff --git a/examples/twilio-chatbot/client/typescript/index.html b/examples/twilio-chatbot/client/typescript/index.html new file mode 100644 index 000000000..83c24031a --- /dev/null +++ b/examples/twilio-chatbot/client/typescript/index.html @@ -0,0 +1,34 @@ + + + + + + + AI Chatbot + + + +
+
+
+ Transport: Disconnected +
+
+ + +
+
+ + + +
+

Debug Info

+
+
+
+ + + + + + diff --git a/examples/twilio-chatbot/client/typescript/package-lock.json b/examples/twilio-chatbot/client/typescript/package-lock.json new file mode 100644 index 000000000..d7e58ef56 --- /dev/null +++ b/examples/twilio-chatbot/client/typescript/package-lock.json @@ -0,0 +1,1793 @@ +{ + "name": "client", + "version": "1.0.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "client", + "version": "1.0.0", + "license": "ISC", + "dependencies": { + "@pipecat-ai/client-js": "^0.4.0", + "@pipecat-ai/websocket-transport": "^0.4.2" + }, + "devDependencies": { + "@types/node": "^22.13.1", + "@types/protobufjs": "^6.0.0", + "@vitejs/plugin-react-swc": "^3.7.2", + "typescript": "^5.7.3", + "vite": "^6.0.2" + } + }, + "node_modules/@babel/runtime": { + "version": "7.27.6", + "resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.27.6.tgz", + "integrity": "sha512-vbavdySgbTTrmFE+EsiqUTzlOr5bzlnJtUv9PynGCAKvfQqjIXbvFdumPM/GxMDfyuGMJaJAU6TO4zc1Jf1i8Q==", + "license": "MIT", + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@bufbuild/protobuf": { + "version": "2.5.2", + "resolved": "https://registry.npmjs.org/@bufbuild/protobuf/-/protobuf-2.5.2.tgz", + "integrity": "sha512-foZ7qr0IsUBjzWIq+SuBLfdQCpJ1j8cTuNNT4owngTHoN5KsJb8L9t65fzz7SCeSWzescoOil/0ldqiL041ABg==", + "license": "(Apache-2.0 AND BSD-3-Clause)" + }, + "node_modules/@bufbuild/protoplugin": { + "version": "2.5.2", + "resolved": "https://registry.npmjs.org/@bufbuild/protoplugin/-/protoplugin-2.5.2.tgz", + "integrity": "sha512-7d/NUae/ugs/qgHEYOwkVWGDE3Bf/xjuGviVFs38+MLRdwiHNTiuvzPVwuIPo/1wuZCZn3Nax1cg1owLuY72xw==", + "license": "Apache-2.0", + "dependencies": { + "@bufbuild/protobuf": "2.5.2", + "@typescript/vfs": "^1.5.2", + "typescript": "5.4.5" + } + }, + "node_modules/@bufbuild/protoplugin/node_modules/typescript": { + "version": "5.4.5", + "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.4.5.tgz", + "integrity": "sha512-vcI4UpRgg81oIRUFwR0WSIHKt11nJ7SAVlYNIu+QpqeyXP+gpQJy/Z4+F0aGxSE4MqwjyXvW/TzgkLAx2AGHwQ==", + "license": "Apache-2.0", + "bin": { + "tsc": "bin/tsc", + "tsserver": "bin/tsserver" + }, + "engines": { + "node": ">=14.17" + } + }, + "node_modules/@daily-co/daily-js": { + "version": "0.79.0", + "resolved": "https://registry.npmjs.org/@daily-co/daily-js/-/daily-js-0.79.0.tgz", + "integrity": "sha512-Ii/Zi6cfTl2EZBpX8msRPNkkCHcajA+ErXpbN2Xe2KySd1Nb4IzC/QWJlSl9VA9pIlYPQicRTDoZnoym/0uEAw==", + "license": "BSD-2-Clause", + "dependencies": { + "@babel/runtime": "^7.12.5", + "@sentry/browser": "^8.33.1", + "bowser": "^2.8.1", + "dequal": "^2.0.3", + "events": "^3.1.0" + }, + "engines": { + "node": ">=10.0.0" + } + }, + "node_modules/@esbuild/aix-ppc64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.25.5.tgz", + "integrity": "sha512-9o3TMmpmftaCMepOdA5k/yDw8SfInyzWWTjYTFCX3kPSDJMROQTb8jg+h9Cnwnmm1vOzvxN7gIfB5V2ewpjtGA==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "aix" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-arm": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.25.5.tgz", + "integrity": "sha512-AdJKSPeEHgi7/ZhuIPtcQKr5RQdo6OO2IL87JkianiMYMPbCtot9fxPbrMiBADOWWm3T2si9stAiVsGbTQFkbA==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-arm64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.25.5.tgz", + "integrity": "sha512-VGzGhj4lJO+TVGV1v8ntCZWJktV7SGCs3Pn1GRWI1SBFtRALoomm8k5E9Pmwg3HOAal2VDc2F9+PM/rEY6oIDg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-x64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.25.5.tgz", + "integrity": "sha512-D2GyJT1kjvO//drbRT3Hib9XPwQeWd9vZoBJn+bu/lVsOZ13cqNdDeqIF/xQ5/VmWvMduP6AmXvylO/PIc2isw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/darwin-arm64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.25.5.tgz", + "integrity": "sha512-GtaBgammVvdF7aPIgH2jxMDdivezgFu6iKpmT+48+F8Hhg5J/sfnDieg0aeG/jfSvkYQU2/pceFPDKlqZzwnfQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/darwin-x64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.25.5.tgz", + "integrity": "sha512-1iT4FVL0dJ76/q1wd7XDsXrSW+oLoquptvh4CLR4kITDtqi2e/xwXwdCVH8hVHU43wgJdsq7Gxuzcs6Iq/7bxQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/freebsd-arm64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.25.5.tgz", + "integrity": "sha512-nk4tGP3JThz4La38Uy/gzyXtpkPW8zSAmoUhK9xKKXdBCzKODMc2adkB2+8om9BDYugz+uGV7sLmpTYzvmz6Sw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/freebsd-x64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.25.5.tgz", + "integrity": "sha512-PrikaNjiXdR2laW6OIjlbeuCPrPaAl0IwPIaRv+SMV8CiM8i2LqVUHFC1+8eORgWyY7yhQY+2U2fA55mBzReaw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-arm": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.25.5.tgz", + "integrity": "sha512-cPzojwW2okgh7ZlRpcBEtsX7WBuqbLrNXqLU89GxWbNt6uIg78ET82qifUy3W6OVww6ZWobWub5oqZOVtwolfw==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-arm64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.25.5.tgz", + "integrity": "sha512-Z9kfb1v6ZlGbWj8EJk9T6czVEjjq2ntSYLY2cw6pAZl4oKtfgQuS4HOq41M/BcoLPzrUbNd+R4BXFyH//nHxVg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-ia32": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.25.5.tgz", + "integrity": "sha512-sQ7l00M8bSv36GLV95BVAdhJ2QsIbCuCjh/uYrWiMQSUuV+LpXwIqhgJDcvMTj+VsQmqAHL2yYaasENvJ7CDKA==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-loong64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.25.5.tgz", + "integrity": "sha512-0ur7ae16hDUC4OL5iEnDb0tZHDxYmuQyhKhsPBV8f99f6Z9KQM02g33f93rNH5A30agMS46u2HP6qTdEt6Q1kg==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-mips64el": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.25.5.tgz", + "integrity": "sha512-kB/66P1OsHO5zLz0i6X0RxlQ+3cu0mkxS3TKFvkb5lin6uwZ/ttOkP3Z8lfR9mJOBk14ZwZ9182SIIWFGNmqmg==", + "cpu": [ + "mips64el" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-ppc64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.25.5.tgz", + "integrity": "sha512-UZCmJ7r9X2fe2D6jBmkLBMQetXPXIsZjQJCjgwpVDz+YMcS6oFR27alkgGv3Oqkv07bxdvw7fyB71/olceJhkQ==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-riscv64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.25.5.tgz", + "integrity": "sha512-kTxwu4mLyeOlsVIFPfQo+fQJAV9mh24xL+y+Bm6ej067sYANjyEw1dNHmvoqxJUCMnkBdKpvOn0Ahql6+4VyeA==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-s390x": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.25.5.tgz", + "integrity": "sha512-K2dSKTKfmdh78uJ3NcWFiqyRrimfdinS5ErLSn3vluHNeHVnBAFWC8a4X5N+7FgVE1EjXS1QDZbpqZBjfrqMTQ==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-x64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.25.5.tgz", + "integrity": "sha512-uhj8N2obKTE6pSZ+aMUbqq+1nXxNjZIIjCjGLfsWvVpy7gKCOL6rsY1MhRh9zLtUtAI7vpgLMK6DxjO8Qm9lJw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/netbsd-arm64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.25.5.tgz", + "integrity": "sha512-pwHtMP9viAy1oHPvgxtOv+OkduK5ugofNTVDilIzBLpoWAM16r7b/mxBvfpuQDpRQFMfuVr5aLcn4yveGvBZvw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/netbsd-x64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.25.5.tgz", + "integrity": "sha512-WOb5fKrvVTRMfWFNCroYWWklbnXH0Q5rZppjq0vQIdlsQKuw6mdSihwSo4RV/YdQ5UCKKvBy7/0ZZYLBZKIbwQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/openbsd-arm64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.25.5.tgz", + "integrity": "sha512-7A208+uQKgTxHd0G0uqZO8UjK2R0DDb4fDmERtARjSHWxqMTye4Erz4zZafx7Di9Cv+lNHYuncAkiGFySoD+Mw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/openbsd-x64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.25.5.tgz", + "integrity": "sha512-G4hE405ErTWraiZ8UiSoesH8DaCsMm0Cay4fsFWOOUcz8b8rC6uCvnagr+gnioEjWn0wC+o1/TAHt+It+MpIMg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/sunos-x64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.25.5.tgz", + "integrity": "sha512-l+azKShMy7FxzY0Rj4RCt5VD/q8mG/e+mDivgspo+yL8zW7qEwctQ6YqKX34DTEleFAvCIUviCFX1SDZRSyMQA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "sunos" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-arm64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.25.5.tgz", + "integrity": "sha512-O2S7SNZzdcFG7eFKgvwUEZ2VG9D/sn/eIiz8XRZ1Q/DO5a3s76Xv0mdBzVM5j5R639lXQmPmSo0iRpHqUUrsxw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-ia32": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.25.5.tgz", + "integrity": "sha512-onOJ02pqs9h1iMJ1PQphR+VZv8qBMQ77Klcsqv9CNW2w6yLqoURLcgERAIurY6QE63bbLuqgP9ATqajFLK5AMQ==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-x64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.25.5.tgz", + "integrity": "sha512-TXv6YnJ8ZMVdX+SXWVBo/0p8LTcrUYngpWjvm91TMjjBQii7Oz11Lw5lbDV5Y0TzuhSJHwiH4hEtC1I42mMS0g==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@pipecat-ai/client-js": { + "version": "0.4.1", + "resolved": "https://registry.npmjs.org/@pipecat-ai/client-js/-/client-js-0.4.1.tgz", + "integrity": "sha512-3jLKRzeryqLxtkqvr4Bvxe2OxoI7mdOFecm6iolZizXnk/BE480SEg2oAKyov3b5oT6+jmPlT+1HRBlTzEtL7A==", + "license": "BSD-2-Clause", + "dependencies": { + "@types/events": "^3.0.3", + "clone-deep": "^4.0.1", + "events": "^3.3.0", + "typed-emitter": "^2.1.0", + "uuid": "^10.0.0" + } + }, + "node_modules/@pipecat-ai/websocket-transport": { + "version": "0.4.2", + "resolved": "https://registry.npmjs.org/@pipecat-ai/websocket-transport/-/websocket-transport-0.4.2.tgz", + "integrity": "sha512-mOYnw9n60usODrE35D+uhFbJXl0DqXV32pAqSHu1of049s128mex6Qv+W49DBMVr8h5W6pLGrXhm+XDAtN5leg==", + "license": "BSD-2-Clause", + "dependencies": { + "@daily-co/daily-js": "^0.79.0", + "@protobuf-ts/plugin": "^2.11.0", + "@protobuf-ts/runtime": "^2.11.0", + "x-law": "^0.3.1" + }, + "peerDependencies": { + "@pipecat-ai/client-js": "~0.4.0" + } + }, + "node_modules/@protobuf-ts/plugin": { + "version": "2.11.0", + "resolved": "https://registry.npmjs.org/@protobuf-ts/plugin/-/plugin-2.11.0.tgz", + "integrity": "sha512-Y+p4Axrk3thxws4BVSIO+x4CKWH2c8k3K+QPrp6Oq8agdsXPL/uwsMTIdpTdXIzTaUEZFASJL9LU56pob5GTHg==", + "license": "Apache-2.0", + "dependencies": { + "@bufbuild/protobuf": "^2.4.0", + "@bufbuild/protoplugin": "^2.4.0", + "@protobuf-ts/protoc": "^2.11.0", + "@protobuf-ts/runtime": "^2.11.0", + "@protobuf-ts/runtime-rpc": "^2.11.0", + "typescript": "^3.9" + }, + "bin": { + "protoc-gen-dump": "bin/protoc-gen-dump", + "protoc-gen-ts": "bin/protoc-gen-ts" + } + }, + "node_modules/@protobuf-ts/plugin/node_modules/typescript": { + "version": "3.9.10", + "resolved": "https://registry.npmjs.org/typescript/-/typescript-3.9.10.tgz", + "integrity": "sha512-w6fIxVE/H1PkLKcCPsFqKE7Kv7QUwhU8qQY2MueZXWx5cPZdwFupLgKK3vntcK98BtNHZtAF4LA/yl2a7k8R6Q==", + "license": "Apache-2.0", + "bin": { + "tsc": "bin/tsc", + "tsserver": "bin/tsserver" + }, + "engines": { + "node": ">=4.2.0" + } + }, + "node_modules/@protobuf-ts/protoc": { + "version": "2.11.0", + "resolved": "https://registry.npmjs.org/@protobuf-ts/protoc/-/protoc-2.11.0.tgz", + "integrity": "sha512-GYfmv1rjZ/7MWzUqMszhdXiuoa4Js/j6zCbcxFmeThBBUhbrXdPU42vY+QVCHL9PvAMXO+wEhUfPWYdd1YgnlA==", + "license": "Apache-2.0", + "bin": { + "protoc": "protoc.js" + } + }, + "node_modules/@protobuf-ts/runtime": { + "version": "2.11.0", + "resolved": "https://registry.npmjs.org/@protobuf-ts/runtime/-/runtime-2.11.0.tgz", + "integrity": "sha512-DfpRpUiNvPC3Kj48CmlU4HaIEY1Myh++PIumMmohBAk8/k0d2CkxYxJfPyUAxfuUfl97F4AvuCu1gXmfOG7OJQ==", + "license": "(Apache-2.0 AND BSD-3-Clause)" + }, + "node_modules/@protobuf-ts/runtime-rpc": { + "version": "2.11.0", + "resolved": "https://registry.npmjs.org/@protobuf-ts/runtime-rpc/-/runtime-rpc-2.11.0.tgz", + "integrity": "sha512-g/oMPym5LjVyCc3nlQc6cHer0R3CyleBos4p7CjRNzdKuH/FlRXzfQYo6EN5uv8vLtn7zEK9Cy4YBKvHStIaag==", + "license": "Apache-2.0", + "dependencies": { + "@protobuf-ts/runtime": "^2.11.0" + } + }, + "node_modules/@protobufjs/aspromise": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/aspromise/-/aspromise-1.1.2.tgz", + "integrity": "sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ==", + "dev": true, + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/base64": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/base64/-/base64-1.1.2.tgz", + "integrity": "sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg==", + "dev": true, + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/codegen": { + "version": "2.0.4", + "resolved": "https://registry.npmjs.org/@protobufjs/codegen/-/codegen-2.0.4.tgz", + "integrity": "sha512-YyFaikqM5sH0ziFZCN3xDC7zeGaB/d0IUb9CATugHWbd1FRFwWwt4ld4OYMPWu5a3Xe01mGAULCdqhMlPl29Jg==", + "dev": true, + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/eventemitter": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@protobufjs/eventemitter/-/eventemitter-1.1.0.tgz", + "integrity": "sha512-j9ednRT81vYJ9OfVuXG6ERSTdEL1xVsNgqpkxMsbIabzSo3goCjDIveeGv5d03om39ML71RdmrGNjG5SReBP/Q==", + "dev": true, + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/fetch": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@protobufjs/fetch/-/fetch-1.1.0.tgz", + "integrity": "sha512-lljVXpqXebpsijW71PZaCYeIcE5on1w5DlQy5WH6GLbFryLUrBD4932W/E2BSpfRJWseIL4v/KPgBFxDOIdKpQ==", + "dev": true, + "license": "BSD-3-Clause", + "dependencies": { + "@protobufjs/aspromise": "^1.1.1", + "@protobufjs/inquire": "^1.1.0" + } + }, + "node_modules/@protobufjs/float": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/@protobufjs/float/-/float-1.0.2.tgz", + "integrity": "sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ==", + "dev": true, + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/inquire": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@protobufjs/inquire/-/inquire-1.1.0.tgz", + "integrity": "sha512-kdSefcPdruJiFMVSbn801t4vFK7KB/5gd2fYvrxhuJYg8ILrmn9SKSX2tZdV6V+ksulWqS7aXjBcRXl3wHoD9Q==", + "dev": true, + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/path": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/path/-/path-1.1.2.tgz", + "integrity": "sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA==", + "dev": true, + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/pool": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@protobufjs/pool/-/pool-1.1.0.tgz", + "integrity": "sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw==", + "dev": true, + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/utf8": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@protobufjs/utf8/-/utf8-1.1.0.tgz", + "integrity": "sha512-Vvn3zZrhQZkkBE8LSuW3em98c0FwgO4nxzv6OdSxPKJIEKY2bGbHn+mhGIPerzI4twdxaP8/0+06HBpwf345Lw==", + "dev": true, + "license": "BSD-3-Clause" + }, + "node_modules/@rolldown/pluginutils": { + "version": "1.0.0-beta.11", + "resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0-beta.11.tgz", + "integrity": "sha512-L/gAA/hyCSuzTF1ftlzUSI/IKr2POHsv1Dd78GfqkR83KMNuswWD61JxGV2L7nRwBBBSDr6R1gCkdTmoN7W4ag==", + "dev": true, + "license": "MIT" + }, + "node_modules/@rollup/rollup-android-arm-eabi": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm-eabi/-/rollup-android-arm-eabi-4.43.0.tgz", + "integrity": "sha512-Krjy9awJl6rKbruhQDgivNbD1WuLb8xAclM4IR4cN5pHGAs2oIMMQJEiC3IC/9TZJ+QZkmZhlMO/6MBGxPidpw==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ] + }, + "node_modules/@rollup/rollup-android-arm64": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm64/-/rollup-android-arm64-4.43.0.tgz", + "integrity": "sha512-ss4YJwRt5I63454Rpj+mXCXicakdFmKnUNxr1dLK+5rv5FJgAxnN7s31a5VchRYxCFWdmnDWKd0wbAdTr0J5EA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ] + }, + "node_modules/@rollup/rollup-darwin-arm64": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-arm64/-/rollup-darwin-arm64-4.43.0.tgz", + "integrity": "sha512-eKoL8ykZ7zz8MjgBenEF2OoTNFAPFz1/lyJ5UmmFSz5jW+7XbH1+MAgCVHy72aG59rbuQLcJeiMrP8qP5d/N0A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@rollup/rollup-darwin-x64": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-x64/-/rollup-darwin-x64-4.43.0.tgz", + "integrity": "sha512-SYwXJgaBYW33Wi/q4ubN+ldWC4DzQY62S4Ll2dgfr/dbPoF50dlQwEaEHSKrQdSjC6oIe1WgzosoaNoHCdNuMg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@rollup/rollup-freebsd-arm64": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-arm64/-/rollup-freebsd-arm64-4.43.0.tgz", + "integrity": "sha512-SV+U5sSo0yujrjzBF7/YidieK2iF6E7MdF6EbYxNz94lA+R0wKl3SiixGyG/9Klab6uNBIqsN7j4Y/Fya7wAjQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ] + }, + "node_modules/@rollup/rollup-freebsd-x64": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-x64/-/rollup-freebsd-x64-4.43.0.tgz", + "integrity": "sha512-J7uCsiV13L/VOeHJBo5SjasKiGxJ0g+nQTrBkAsmQBIdil3KhPnSE9GnRon4ejX1XDdsmK/l30IYLiAaQEO0Cg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ] + }, + "node_modules/@rollup/rollup-linux-arm-gnueabihf": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-gnueabihf/-/rollup-linux-arm-gnueabihf-4.43.0.tgz", + "integrity": "sha512-gTJ/JnnjCMc15uwB10TTATBEhK9meBIY+gXP4s0sHD1zHOaIh4Dmy1X9wup18IiY9tTNk5gJc4yx9ctj/fjrIw==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm-musleabihf": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-musleabihf/-/rollup-linux-arm-musleabihf-4.43.0.tgz", + "integrity": "sha512-ZJ3gZynL1LDSIvRfz0qXtTNs56n5DI2Mq+WACWZ7yGHFUEirHBRt7fyIk0NsCKhmRhn7WAcjgSkSVVxKlPNFFw==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm64-gnu": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-gnu/-/rollup-linux-arm64-gnu-4.43.0.tgz", + "integrity": "sha512-8FnkipasmOOSSlfucGYEu58U8cxEdhziKjPD2FIa0ONVMxvl/hmONtX/7y4vGjdUhjcTHlKlDhw3H9t98fPvyA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm64-musl": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-musl/-/rollup-linux-arm64-musl-4.43.0.tgz", + "integrity": "sha512-KPPyAdlcIZ6S9C3S2cndXDkV0Bb1OSMsX0Eelr2Bay4EsF9yi9u9uzc9RniK3mcUGCLhWY9oLr6er80P5DE6XA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-loongarch64-gnu": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-loongarch64-gnu/-/rollup-linux-loongarch64-gnu-4.43.0.tgz", + "integrity": "sha512-HPGDIH0/ZzAZjvtlXj6g+KDQ9ZMHfSP553za7o2Odegb/BEfwJcR0Sw0RLNpQ9nC6Gy8s+3mSS9xjZ0n3rhcYg==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-powerpc64le-gnu": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-powerpc64le-gnu/-/rollup-linux-powerpc64le-gnu-4.43.0.tgz", + "integrity": "sha512-gEmwbOws4U4GLAJDhhtSPWPXUzDfMRedT3hFMyRAvM9Mrnj+dJIFIeL7otsv2WF3D7GrV0GIewW0y28dOYWkmw==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-riscv64-gnu": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-gnu/-/rollup-linux-riscv64-gnu-4.43.0.tgz", + "integrity": "sha512-XXKvo2e+wFtXZF/9xoWohHg+MuRnvO29TI5Hqe9xwN5uN8NKUYy7tXUG3EZAlfchufNCTHNGjEx7uN78KsBo0g==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-riscv64-musl": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-musl/-/rollup-linux-riscv64-musl-4.43.0.tgz", + "integrity": "sha512-ruf3hPWhjw6uDFsOAzmbNIvlXFXlBQ4nk57Sec8E8rUxs/AI4HD6xmiiasOOx/3QxS2f5eQMKTAwk7KHwpzr/Q==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-s390x-gnu": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-s390x-gnu/-/rollup-linux-s390x-gnu-4.43.0.tgz", + "integrity": "sha512-QmNIAqDiEMEvFV15rsSnjoSmO0+eJLoKRD9EAa9rrYNwO/XRCtOGM3A5A0X+wmG+XRrw9Fxdsw+LnyYiZWWcVw==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-x64-gnu": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-gnu/-/rollup-linux-x64-gnu-4.43.0.tgz", + "integrity": "sha512-jAHr/S0iiBtFyzjhOkAics/2SrXE092qyqEg96e90L3t9Op8OTzS6+IX0Fy5wCt2+KqeHAkti+eitV0wvblEoQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-x64-musl": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-musl/-/rollup-linux-x64-musl-4.43.0.tgz", + "integrity": "sha512-3yATWgdeXyuHtBhrLt98w+5fKurdqvs8B53LaoKD7P7H7FKOONLsBVMNl9ghPQZQuYcceV5CDyPfyfGpMWD9mQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-win32-arm64-msvc": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-arm64-msvc/-/rollup-win32-arm64-msvc-4.43.0.tgz", + "integrity": "sha512-wVzXp2qDSCOpcBCT5WRWLmpJRIzv23valvcTwMHEobkjippNf+C3ys/+wf07poPkeNix0paTNemB2XrHr2TnGw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-ia32-msvc": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-ia32-msvc/-/rollup-win32-ia32-msvc-4.43.0.tgz", + "integrity": "sha512-fYCTEyzf8d+7diCw8b+asvWDCLMjsCEA8alvtAutqJOJp/wL5hs1rWSqJ1vkjgW0L2NB4bsYJrpKkiIPRR9dvw==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-x64-msvc": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-msvc/-/rollup-win32-x64-msvc-4.43.0.tgz", + "integrity": "sha512-SnGhLiE5rlK0ofq8kzuDkM0g7FN1s5VYY+YSMTibP7CqShxCQvqtNxTARS4xX4PFJfHjG0ZQYX9iGzI3FQh5Aw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@sentry-internal/browser-utils": { + "version": "8.55.0", + "resolved": "https://registry.npmjs.org/@sentry-internal/browser-utils/-/browser-utils-8.55.0.tgz", + "integrity": "sha512-ROgqtQfpH/82AQIpESPqPQe0UyWywKJsmVIqi3c5Fh+zkds5LUxnssTj3yNd1x+kxaPDVB023jAP+3ibNgeNDw==", + "license": "MIT", + "dependencies": { + "@sentry/core": "8.55.0" + }, + "engines": { + "node": ">=14.18" + } + }, + "node_modules/@sentry-internal/feedback": { + "version": "8.55.0", + "resolved": "https://registry.npmjs.org/@sentry-internal/feedback/-/feedback-8.55.0.tgz", + "integrity": "sha512-cP3BD/Q6pquVQ+YL+rwCnorKuTXiS9KXW8HNKu4nmmBAyf7urjs+F6Hr1k9MXP5yQ8W3yK7jRWd09Yu6DHWOiw==", + "license": "MIT", + "dependencies": { + "@sentry/core": "8.55.0" + }, + "engines": { + "node": ">=14.18" + } + }, + "node_modules/@sentry-internal/replay": { + "version": "8.55.0", + "resolved": "https://registry.npmjs.org/@sentry-internal/replay/-/replay-8.55.0.tgz", + "integrity": "sha512-roCDEGkORwolxBn8xAKedybY+Jlefq3xYmgN2fr3BTnsXjSYOPC7D1/mYqINBat99nDtvgFvNfRcZPiwwZ1hSw==", + "license": "MIT", + "dependencies": { + "@sentry-internal/browser-utils": "8.55.0", + "@sentry/core": "8.55.0" + }, + "engines": { + "node": ">=14.18" + } + }, + "node_modules/@sentry-internal/replay-canvas": { + "version": "8.55.0", + "resolved": "https://registry.npmjs.org/@sentry-internal/replay-canvas/-/replay-canvas-8.55.0.tgz", + "integrity": "sha512-nIkfgRWk1091zHdu4NbocQsxZF1rv1f7bbp3tTIlZYbrH62XVZosx5iHAuZG0Zc48AETLE7K4AX9VGjvQj8i9w==", + "license": "MIT", + "dependencies": { + "@sentry-internal/replay": "8.55.0", + "@sentry/core": "8.55.0" + }, + "engines": { + "node": ">=14.18" + } + }, + "node_modules/@sentry/browser": { + "version": "8.55.0", + "resolved": "https://registry.npmjs.org/@sentry/browser/-/browser-8.55.0.tgz", + "integrity": "sha512-1A31mCEWCjaMxJt6qGUK+aDnLDcK6AwLAZnqpSchNysGni1pSn1RWSmk9TBF8qyTds5FH8B31H480uxMPUJ7Cw==", + "license": "MIT", + "dependencies": { + "@sentry-internal/browser-utils": "8.55.0", + "@sentry-internal/feedback": "8.55.0", + "@sentry-internal/replay": "8.55.0", + "@sentry-internal/replay-canvas": "8.55.0", + "@sentry/core": "8.55.0" + }, + "engines": { + "node": ">=14.18" + } + }, + "node_modules/@sentry/core": { + "version": "8.55.0", + "resolved": "https://registry.npmjs.org/@sentry/core/-/core-8.55.0.tgz", + "integrity": "sha512-6g7jpbefjHYs821Z+EBJ8r4Z7LT5h80YSWRJaylGS4nW5W5Z2KXzpdnyFarv37O7QjauzVC2E+PABmpkw5/JGA==", + "license": "MIT", + "engines": { + "node": ">=14.18" + } + }, + "node_modules/@swc/core": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core/-/core-1.12.0.tgz", + "integrity": "sha512-/C0kiMHPY/HnLfqXYGMGxGck3A5Y3mqwxfv+EwHTPHGjAVRfHpWAEEBTSTF5C88vVY6CvwBEkhR2TX7t8Mahcw==", + "dev": true, + "hasInstallScript": true, + "license": "Apache-2.0", + "dependencies": { + "@swc/counter": "^0.1.3", + "@swc/types": "^0.1.22" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/swc" + }, + "optionalDependencies": { + "@swc/core-darwin-arm64": "1.12.0", + "@swc/core-darwin-x64": "1.12.0", + "@swc/core-linux-arm-gnueabihf": "1.12.0", + "@swc/core-linux-arm64-gnu": "1.12.0", + "@swc/core-linux-arm64-musl": "1.12.0", + "@swc/core-linux-x64-gnu": "1.12.0", + "@swc/core-linux-x64-musl": "1.12.0", + "@swc/core-win32-arm64-msvc": "1.12.0", + "@swc/core-win32-ia32-msvc": "1.12.0", + "@swc/core-win32-x64-msvc": "1.12.0" + }, + "peerDependencies": { + "@swc/helpers": ">=0.5.17" + }, + "peerDependenciesMeta": { + "@swc/helpers": { + "optional": true + } + } + }, + "node_modules/@swc/core-darwin-arm64": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core-darwin-arm64/-/core-darwin-arm64-1.12.0.tgz", + "integrity": "sha512-usLr8kC80GDv3pwH2zoEaS279kxtWY0MY3blbMFw7zA8fAjqxa8IDxm3WcgyNLNWckWn4asFfguEwz/Weem3nA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0 AND MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=10" + } + }, + "node_modules/@swc/core-darwin-x64": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core-darwin-x64/-/core-darwin-x64-1.12.0.tgz", + "integrity": "sha512-Cvv4sqDcTY7QF2Dh1vn2Xbt/1ENYQcpmrGHzITJrXzxA2aBopsz/n4yQDiyRxTR0t802m4xu0CzMoZIHvVruWQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0 AND MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=10" + } + }, + "node_modules/@swc/core-linux-arm-gnueabihf": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core-linux-arm-gnueabihf/-/core-linux-arm-gnueabihf-1.12.0.tgz", + "integrity": "sha512-seM4/XMJMOupkzfLfHl8sRa3NdhsVZp+XgwA/vVeYZYJE4wuWUxVzhCYzwmNftVY32eF2IiRaWnhG6ho6jusnQ==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=10" + } + }, + "node_modules/@swc/core-linux-arm64-gnu": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core-linux-arm64-gnu/-/core-linux-arm64-gnu-1.12.0.tgz", + "integrity": "sha512-Al0x33gUVxNY5tutEYpSyv7mze6qQS1ONa0HEwoRxcK9WXsX0NHLTiOSGZoCUS1SsXM37ONlbA6/Bsp1MQyP+g==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0 AND MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=10" + } + }, + "node_modules/@swc/core-linux-arm64-musl": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core-linux-arm64-musl/-/core-linux-arm64-musl-1.12.0.tgz", + "integrity": "sha512-OeFHz/5Hl9v75J9TYA5jQxNIYAZMqaiPpd9dYSTK2Xyqa/ZGgTtNyPhIwVfxx+9mHBf6+9c1mTlXUtACMtHmaQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0 AND MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=10" + } + }, + "node_modules/@swc/core-linux-x64-gnu": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core-linux-x64-gnu/-/core-linux-x64-gnu-1.12.0.tgz", + "integrity": "sha512-ltIvqNi7H0c5pRawyqjeYSKEIfZP4vv/datT3mwT6BW7muJtd1+KIDCPFLMIQ4wm/h76YQwPocsin3fzmnFdNA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0 AND MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=10" + } + }, + "node_modules/@swc/core-linux-x64-musl": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core-linux-x64-musl/-/core-linux-x64-musl-1.12.0.tgz", + "integrity": "sha512-Z/DhpjehaTK0uf+MhNB7mV9SuewpGs3P/q9/8+UsJeYoFr7yuOoPbAvrD6AqZkf6Bh7MRZ5OtG+KQgG5L+goiA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0 AND MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=10" + } + }, + "node_modules/@swc/core-win32-arm64-msvc": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core-win32-arm64-msvc/-/core-win32-arm64-msvc-1.12.0.tgz", + "integrity": "sha512-wHnvbfHIh2gfSbvuFT7qP97YCMUDh+fuiso+pcC6ug8IsMxuViNapHET4o0ZdFNWHhXJ7/s0e6w7mkOalsqQiQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0 AND MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=10" + } + }, + "node_modules/@swc/core-win32-ia32-msvc": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core-win32-ia32-msvc/-/core-win32-ia32-msvc-1.12.0.tgz", + "integrity": "sha512-88umlXwK+7J2p4DjfWHXQpmlZgCf1ayt6Ssj+PYlAfMCR0aBiJoAMwHWrvDXEozyOrsyP1j2X6WxbmA861vL5Q==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "Apache-2.0 AND MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=10" + } + }, + "node_modules/@swc/core-win32-x64-msvc": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core-win32-x64-msvc/-/core-win32-x64-msvc-1.12.0.tgz", + "integrity": "sha512-KR9TSRp+FEVOhbgTU6c94p/AYpsyBk7dIvlKQiDp8oKScUoyHG5yjmMBFN/BqUyTq4kj6zlgsY2rFE4R8/yqWg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0 AND MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=10" + } + }, + "node_modules/@swc/counter": { + "version": "0.1.3", + "resolved": "https://registry.npmjs.org/@swc/counter/-/counter-0.1.3.tgz", + "integrity": "sha512-e2BR4lsJkkRlKZ/qCHPw9ZaSxc0MVUd7gtbtaB7aMvHeJVYe8sOB8DBZkP2DtISHGSku9sCK6T6cnY0CtXrOCQ==", + "dev": true, + "license": "Apache-2.0" + }, + "node_modules/@swc/types": { + "version": "0.1.23", + "resolved": "https://registry.npmjs.org/@swc/types/-/types-0.1.23.tgz", + "integrity": "sha512-u1iIVZV9Q0jxY+yM2vw/hZGDNudsN85bBpTqzAQ9rzkxW9D+e3aEM4Han+ow518gSewkXgjmEK0BD79ZcNVgPw==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "@swc/counter": "^0.1.3" + } + }, + "node_modules/@types/estree": { + "version": "1.0.7", + "resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.7.tgz", + "integrity": "sha512-w28IoSUCJpidD/TGviZwwMJckNESJZXFu7NBZ5YJ4mEUnNraUn9Pm8HSZm/jDF1pDWYKspWE7oVphigUPRakIQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/@types/events": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/@types/events/-/events-3.0.3.tgz", + "integrity": "sha512-trOc4AAUThEz9hapPtSd7wf5tiQKvTtu5b371UxXdTuqzIh0ArcRspRP0i0Viu+LXstIQ1z96t1nsPxT9ol01g==", + "license": "MIT" + }, + "node_modules/@types/node": { + "version": "22.15.31", + "resolved": "https://registry.npmjs.org/@types/node/-/node-22.15.31.tgz", + "integrity": "sha512-jnVe5ULKl6tijxUhvQeNbQG/84fHfg+yMak02cT8QVhBx/F05rAVxCGBYYTh2EKz22D6JF5ktXuNwdx7b9iEGw==", + "dev": true, + "license": "MIT", + "dependencies": { + "undici-types": "~6.21.0" + } + }, + "node_modules/@types/protobufjs": { + "version": "6.0.0", + "resolved": "https://registry.npmjs.org/@types/protobufjs/-/protobufjs-6.0.0.tgz", + "integrity": "sha512-A27RDExpAf3rdDjIrHKiJK6x8kqqJ4CmoChwtipfhVAn1p7+wviQFFP7dppn8FslSbHtQeVPvi8wNKkDjSYjHw==", + "deprecated": "This is a stub types definition for protobufjs (https://github.com/dcodeIO/ProtoBuf.js). protobufjs provides its own type definitions, so you don't need @types/protobufjs installed!", + "dev": true, + "license": "MIT", + "dependencies": { + "protobufjs": "*" + } + }, + "node_modules/@typescript/vfs": { + "version": "1.6.1", + "resolved": "https://registry.npmjs.org/@typescript/vfs/-/vfs-1.6.1.tgz", + "integrity": "sha512-JwoxboBh7Oz1v38tPbkrZ62ZXNHAk9bJ7c9x0eI5zBfBnBYGhURdbnh7Z4smN/MV48Y5OCcZb58n972UtbazsA==", + "license": "MIT", + "dependencies": { + "debug": "^4.1.1" + }, + "peerDependencies": { + "typescript": "*" + } + }, + "node_modules/@vitejs/plugin-react-swc": { + "version": "3.10.2", + "resolved": "https://registry.npmjs.org/@vitejs/plugin-react-swc/-/plugin-react-swc-3.10.2.tgz", + "integrity": "sha512-xD3Rdvrt5LgANug7WekBn1KhcvLn1H3jNBfJRL3reeOIua/WnZOEV5qi5qIBq5T8R0jUDmRtxuvk4bPhzGHDWw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@rolldown/pluginutils": "1.0.0-beta.11", + "@swc/core": "^1.11.31" + }, + "peerDependencies": { + "vite": "^4 || ^5 || ^6 || ^7.0.0-beta.0" + } + }, + "node_modules/bowser": { + "version": "2.11.0", + "resolved": "https://registry.npmjs.org/bowser/-/bowser-2.11.0.tgz", + "integrity": "sha512-AlcaJBi/pqqJBIQ8U9Mcpc9i8Aqxn88Skv5d+xBX006BY5u8N3mGLHa5Lgppa7L/HfwgwLgZ6NYs+Ag6uUmJRA==", + "license": "MIT" + }, + "node_modules/clone-deep": { + "version": "4.0.1", + "resolved": "https://registry.npmjs.org/clone-deep/-/clone-deep-4.0.1.tgz", + "integrity": "sha512-neHB9xuzh/wk0dIHweyAXv2aPGZIVk3pLMe+/RNzINf17fe0OG96QroktYAUm7SM1PBnzTabaLboqqxDyMU+SQ==", + "license": "MIT", + "dependencies": { + "is-plain-object": "^2.0.4", + "kind-of": "^6.0.2", + "shallow-clone": "^3.0.0" + }, + "engines": { + "node": ">=6" + } + }, + "node_modules/debug": { + "version": "4.4.1", + "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.1.tgz", + "integrity": "sha512-KcKCqiftBJcZr++7ykoDIEwSa3XWowTfNPo92BYxjXiyYEVrUQh2aLyhxBCwww+heortUFxEJYcRzosstTEBYQ==", + "license": "MIT", + "dependencies": { + "ms": "^2.1.3" + }, + "engines": { + "node": ">=6.0" + }, + "peerDependenciesMeta": { + "supports-color": { + "optional": true + } + } + }, + "node_modules/dequal": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/dequal/-/dequal-2.0.3.tgz", + "integrity": "sha512-0je+qPKHEMohvfRTCEo3CrPG6cAzAYgmzKyxRiYSSDkS6eGJdyVJm7WaYA5ECaAD9wLB2T4EEeymA5aFVcYXCA==", + "license": "MIT", + "engines": { + "node": ">=6" + } + }, + "node_modules/esbuild": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.25.5.tgz", + "integrity": "sha512-P8OtKZRv/5J5hhz0cUAdu/cLuPIKXpQl1R9pZtvmHWQvrAUVd0UNIPT4IB4W3rNOqVO0rlqHmCIbSwxh/c9yUQ==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "bin": { + "esbuild": "bin/esbuild" + }, + "engines": { + "node": ">=18" + }, + "optionalDependencies": { + "@esbuild/aix-ppc64": "0.25.5", + "@esbuild/android-arm": "0.25.5", + "@esbuild/android-arm64": "0.25.5", + "@esbuild/android-x64": "0.25.5", + "@esbuild/darwin-arm64": "0.25.5", + "@esbuild/darwin-x64": "0.25.5", + "@esbuild/freebsd-arm64": "0.25.5", + "@esbuild/freebsd-x64": "0.25.5", + "@esbuild/linux-arm": "0.25.5", + "@esbuild/linux-arm64": "0.25.5", + "@esbuild/linux-ia32": "0.25.5", + "@esbuild/linux-loong64": "0.25.5", + "@esbuild/linux-mips64el": "0.25.5", + "@esbuild/linux-ppc64": "0.25.5", + "@esbuild/linux-riscv64": "0.25.5", + "@esbuild/linux-s390x": "0.25.5", + "@esbuild/linux-x64": "0.25.5", + "@esbuild/netbsd-arm64": "0.25.5", + "@esbuild/netbsd-x64": "0.25.5", + "@esbuild/openbsd-arm64": "0.25.5", + "@esbuild/openbsd-x64": "0.25.5", + "@esbuild/sunos-x64": "0.25.5", + "@esbuild/win32-arm64": "0.25.5", + "@esbuild/win32-ia32": "0.25.5", + "@esbuild/win32-x64": "0.25.5" + } + }, + "node_modules/events": { + "version": "3.3.0", + "resolved": "https://registry.npmjs.org/events/-/events-3.3.0.tgz", + "integrity": "sha512-mQw+2fkQbALzQ7V0MY0IqdnXNOeTtP4r0lN9z7AAawCXgqea7bDii20AYrIBrFd/Hx0M2Ocz6S111CaFkUcb0Q==", + "license": "MIT", + "engines": { + "node": ">=0.8.x" + } + }, + "node_modules/fdir": { + "version": "6.4.6", + "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.4.6.tgz", + "integrity": "sha512-hiFoqpyZcfNm1yc4u8oWCf9A2c4D3QjCrks3zmoVKVxpQRzmPNar1hUJcBG2RQHvEVGDN+Jm81ZheVLAQMK6+w==", + "dev": true, + "license": "MIT", + "peerDependencies": { + "picomatch": "^3 || ^4" + }, + "peerDependenciesMeta": { + "picomatch": { + "optional": true + } + } + }, + "node_modules/fsevents": { + "version": "2.3.3", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", + "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, + "node_modules/is-plain-object": { + "version": "2.0.4", + "resolved": "https://registry.npmjs.org/is-plain-object/-/is-plain-object-2.0.4.tgz", + "integrity": "sha512-h5PpgXkWitc38BBMYawTYMWJHFZJVnBquFE57xFpjB8pJFiF6gZ+bU+WyI/yqXiFR5mdLsgYNaPe8uao6Uv9Og==", + "license": "MIT", + "dependencies": { + "isobject": "^3.0.1" + }, + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/isobject": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/isobject/-/isobject-3.0.1.tgz", + "integrity": "sha512-WhB9zCku7EGTj/HQQRz5aUQEUeoQZH2bWcltRErOpymJ4boYE6wL9Tbr23krRPSZ+C5zqNSrSw+Cc7sZZ4b7vg==", + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/kind-of": { + "version": "6.0.3", + "resolved": "https://registry.npmjs.org/kind-of/-/kind-of-6.0.3.tgz", + "integrity": "sha512-dcS1ul+9tmeD95T+x28/ehLgd9mENa3LsvDTtzm3vyBEO7RPptvAD+t44WVXaUjTBRcrpFeFlC8WCruUR456hw==", + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/long": { + "version": "5.3.2", + "resolved": "https://registry.npmjs.org/long/-/long-5.3.2.tgz", + "integrity": "sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA==", + "dev": true, + "license": "Apache-2.0" + }, + "node_modules/ms": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", + "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", + "license": "MIT" + }, + "node_modules/nanoid": { + "version": "3.3.11", + "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.11.tgz", + "integrity": "sha512-N8SpfPUnUp1bK+PMYW8qSWdl9U+wwNWI4QKxOYDy9JAro3WMX7p2OeVRF9v+347pnakNevPmiHhNmZ2HbFA76w==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "bin": { + "nanoid": "bin/nanoid.cjs" + }, + "engines": { + "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1" + } + }, + "node_modules/picocolors": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", + "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==", + "dev": true, + "license": "ISC" + }, + "node_modules/picomatch": { + "version": "4.0.2", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.2.tgz", + "integrity": "sha512-M7BAV6Rlcy5u+m6oPhAPFgJTzAioX/6B0DxyvDlo9l8+T3nLKbrczg2WLUyzd45L8RqfUMyGPzekbMvX2Ldkwg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/jonschlinkert" + } + }, + "node_modules/postcss": { + "version": "8.5.5", + "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.5.tgz", + "integrity": "sha512-d/jtm+rdNT8tpXuHY5MMtcbJFBkhXE6593XVR9UoGCH8jSFGci7jGvMGH5RYd5PBJW+00NZQt6gf7CbagJCrhg==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/postcss/" + }, + { + "type": "tidelift", + "url": "https://tidelift.com/funding/github/npm/postcss" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "dependencies": { + "nanoid": "^3.3.11", + "picocolors": "^1.1.1", + "source-map-js": "^1.2.1" + }, + "engines": { + "node": "^10 || ^12 || >=14" + } + }, + "node_modules/protobufjs": { + "version": "7.5.3", + "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-7.5.3.tgz", + "integrity": "sha512-sildjKwVqOI2kmFDiXQ6aEB0fjYTafpEvIBs8tOR8qI4spuL9OPROLVu2qZqi/xgCfsHIwVqlaF8JBjWFHnKbw==", + "dev": true, + "hasInstallScript": true, + "license": "BSD-3-Clause", + "dependencies": { + "@protobufjs/aspromise": "^1.1.2", + "@protobufjs/base64": "^1.1.2", + "@protobufjs/codegen": "^2.0.4", + "@protobufjs/eventemitter": "^1.1.0", + "@protobufjs/fetch": "^1.1.0", + "@protobufjs/float": "^1.0.2", + "@protobufjs/inquire": "^1.1.0", + "@protobufjs/path": "^1.1.2", + "@protobufjs/pool": "^1.1.0", + "@protobufjs/utf8": "^1.1.0", + "@types/node": ">=13.7.0", + "long": "^5.0.0" + }, + "engines": { + "node": ">=12.0.0" + } + }, + "node_modules/rollup": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/rollup/-/rollup-4.43.0.tgz", + "integrity": "sha512-wdN2Kd3Twh8MAEOEJZsuxuLKCsBEo4PVNLK6tQWAn10VhsVewQLzcucMgLolRlhFybGxfclbPeEYBaP6RvUFGg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/estree": "1.0.7" + }, + "bin": { + "rollup": "dist/bin/rollup" + }, + "engines": { + "node": ">=18.0.0", + "npm": ">=8.0.0" + }, + "optionalDependencies": { + "@rollup/rollup-android-arm-eabi": "4.43.0", + "@rollup/rollup-android-arm64": "4.43.0", + "@rollup/rollup-darwin-arm64": "4.43.0", + "@rollup/rollup-darwin-x64": "4.43.0", + "@rollup/rollup-freebsd-arm64": "4.43.0", + "@rollup/rollup-freebsd-x64": "4.43.0", + "@rollup/rollup-linux-arm-gnueabihf": "4.43.0", + "@rollup/rollup-linux-arm-musleabihf": "4.43.0", + "@rollup/rollup-linux-arm64-gnu": "4.43.0", + "@rollup/rollup-linux-arm64-musl": "4.43.0", + "@rollup/rollup-linux-loongarch64-gnu": "4.43.0", + "@rollup/rollup-linux-powerpc64le-gnu": "4.43.0", + "@rollup/rollup-linux-riscv64-gnu": "4.43.0", + "@rollup/rollup-linux-riscv64-musl": "4.43.0", + "@rollup/rollup-linux-s390x-gnu": "4.43.0", + "@rollup/rollup-linux-x64-gnu": "4.43.0", + "@rollup/rollup-linux-x64-musl": "4.43.0", + "@rollup/rollup-win32-arm64-msvc": "4.43.0", + "@rollup/rollup-win32-ia32-msvc": "4.43.0", + "@rollup/rollup-win32-x64-msvc": "4.43.0", + "fsevents": "~2.3.2" + } + }, + "node_modules/rxjs": { + "version": "7.8.2", + "resolved": "https://registry.npmjs.org/rxjs/-/rxjs-7.8.2.tgz", + "integrity": "sha512-dhKf903U/PQZY6boNNtAGdWbG85WAbjT/1xYoZIC7FAY0yWapOBQVsVrDl58W86//e1VpMNBtRV4MaXfdMySFA==", + "license": "Apache-2.0", + "optional": true, + "dependencies": { + "tslib": "^2.1.0" + } + }, + "node_modules/shallow-clone": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/shallow-clone/-/shallow-clone-3.0.1.tgz", + "integrity": "sha512-/6KqX+GVUdqPuPPd2LxDDxzX6CAbjJehAAOKlNpqqUpAqPM6HeL8f+o3a+JsyGjn2lv0WY8UsTgUJjU9Ok55NA==", + "license": "MIT", + "dependencies": { + "kind-of": "^6.0.2" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/source-map-js": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz", + "integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==", + "dev": true, + "license": "BSD-3-Clause", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/tinyglobby": { + "version": "0.2.14", + "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.14.tgz", + "integrity": "sha512-tX5e7OM1HnYr2+a2C/4V0htOcSQcoSTH9KgJnVvNm5zm/cyEWKJ7j7YutsH9CxMdtOkkLFy2AHrMci9IM8IPZQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "fdir": "^6.4.4", + "picomatch": "^4.0.2" + }, + "engines": { + "node": ">=12.0.0" + }, + "funding": { + "url": "https://github.com/sponsors/SuperchupuDev" + } + }, + "node_modules/tslib": { + "version": "2.8.1", + "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz", + "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", + "license": "0BSD", + "optional": true + }, + "node_modules/typed-emitter": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/typed-emitter/-/typed-emitter-2.1.0.tgz", + "integrity": "sha512-g/KzbYKbH5C2vPkaXGu8DJlHrGKHLsM25Zg9WuC9pMGfuvT+X25tZQWo5fK1BjBm8+UrVE9LDCvaY0CQk+fXDA==", + "license": "MIT", + "optionalDependencies": { + "rxjs": "*" + } + }, + "node_modules/typescript": { + "version": "5.8.3", + "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.8.3.tgz", + "integrity": "sha512-p1diW6TqL9L07nNxvRMM7hMMw4c5XOo/1ibL4aAIGmSAt9slTE1Xgw5KWuof2uTOvCg9BY7ZRi+GaF+7sfgPeQ==", + "license": "Apache-2.0", + "bin": { + "tsc": "bin/tsc", + "tsserver": "bin/tsserver" + }, + "engines": { + "node": ">=14.17" + } + }, + "node_modules/undici-types": { + "version": "6.21.0", + "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz", + "integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/uuid": { + "version": "10.0.0", + "resolved": "https://registry.npmjs.org/uuid/-/uuid-10.0.0.tgz", + "integrity": "sha512-8XkAphELsDnEGrDxUOHB3RGvXz6TeuYSGEZBOjtTtPm2lwhGBjLgOzLHB63IUWfBpNucQjND6d3AOudO+H3RWQ==", + "funding": [ + "https://github.com/sponsors/broofa", + "https://github.com/sponsors/ctavan" + ], + "license": "MIT", + "bin": { + "uuid": "dist/bin/uuid" + } + }, + "node_modules/vite": { + "version": "6.3.5", + "resolved": "https://registry.npmjs.org/vite/-/vite-6.3.5.tgz", + "integrity": "sha512-cZn6NDFE7wdTpINgs++ZJ4N49W2vRp8LCKrn3Ob1kYNtOo21vfDoaV5GzBfLU4MovSAB8uNRm4jgzVQZ+mBzPQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "esbuild": "^0.25.0", + "fdir": "^6.4.4", + "picomatch": "^4.0.2", + "postcss": "^8.5.3", + "rollup": "^4.34.9", + "tinyglobby": "^0.2.13" + }, + "bin": { + "vite": "bin/vite.js" + }, + "engines": { + "node": "^18.0.0 || ^20.0.0 || >=22.0.0" + }, + "funding": { + "url": "https://github.com/vitejs/vite?sponsor=1" + }, + "optionalDependencies": { + "fsevents": "~2.3.3" + }, + "peerDependencies": { + "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0", + "jiti": ">=1.21.0", + "less": "*", + "lightningcss": "^1.21.0", + "sass": "*", + "sass-embedded": "*", + "stylus": "*", + "sugarss": "*", + "terser": "^5.16.0", + "tsx": "^4.8.1", + "yaml": "^2.4.2" + }, + "peerDependenciesMeta": { + "@types/node": { + "optional": true + }, + "jiti": { + "optional": true + }, + "less": { + "optional": true + }, + "lightningcss": { + "optional": true + }, + "sass": { + "optional": true + }, + "sass-embedded": { + "optional": true + }, + "stylus": { + "optional": true + }, + "sugarss": { + "optional": true + }, + "terser": { + "optional": true + }, + "tsx": { + "optional": true + }, + "yaml": { + "optional": true + } + } + }, + "node_modules/x-law": { + "version": "0.3.1", + "resolved": "https://registry.npmjs.org/x-law/-/x-law-0.3.1.tgz", + "integrity": "sha512-Nvo6OKj6UL2LuzAc08uJkwIDkK2PsTEdpLiY82NkwMptuRpAA1V7arUl7ZY12BcgRYNq8uh1pdAu7G6VeQn7Hg==", + "license": "MIT", + "engines": { + "node": ">=18" + } + } + } +} diff --git a/examples/twilio-chatbot/client/typescript/package.json b/examples/twilio-chatbot/client/typescript/package.json new file mode 100644 index 000000000..4f107e789 --- /dev/null +++ b/examples/twilio-chatbot/client/typescript/package.json @@ -0,0 +1,25 @@ +{ + "name": "client", + "version": "1.0.0", + "main": "index.js", + "scripts": { + "dev": "vite", + "build": "tsc && vite build", + "preview": "vite preview" + }, + "keywords": [], + "author": "", + "license": "ISC", + "description": "", + "devDependencies": { + "@types/node": "^22.13.1", + "@types/protobufjs": "^6.0.0", + "@vitejs/plugin-react-swc": "^3.7.2", + "typescript": "^5.7.3", + "vite": "^6.0.2" + }, + "dependencies": { + "@pipecat-ai/client-js": "^0.4.0", + "@pipecat-ai/websocket-transport": "^0.4.2" + } +} diff --git a/examples/twilio-chatbot/client/typescript/src/app.ts b/examples/twilio-chatbot/client/typescript/src/app.ts new file mode 100644 index 000000000..e52c6ebe3 --- /dev/null +++ b/examples/twilio-chatbot/client/typescript/src/app.ts @@ -0,0 +1,247 @@ +/** + * Copyright (c) 2024–2025, Daily + * + * SPDX-License-Identifier: BSD 2-Clause License + */ + +import { + RTVIClient, + RTVIClientOptions, + RTVIEvent, +} from '@pipecat-ai/client-js'; +import { + WebSocketTransport, + TwilioSerializer, +} from "@pipecat-ai/websocket-transport"; + +class WebsocketClientApp { + + private static STREAM_SID = "ws_mock_stream_sid" + private static CALL_SID = "ws_mock_call_sid" + + private rtviClient: RTVIClient | null = null; + private connectBtn: HTMLButtonElement | null = null; + private disconnectBtn: HTMLButtonElement | null = null; + private statusSpan: HTMLElement | null = null; + private debugLog: HTMLElement | null = null; + private botAudio: HTMLAudioElement; + + constructor() { + this.botAudio = document.createElement('audio'); + this.botAudio.autoplay = true; + document.body.appendChild(this.botAudio); + this.setupDOMElements(); + this.setupEventListeners(); + } + + /** + * Set up references to DOM elements and create necessary media elements + */ + private setupDOMElements(): void { + this.connectBtn = document.getElementById('connect-btn') as HTMLButtonElement; + this.disconnectBtn = document.getElementById('disconnect-btn') as HTMLButtonElement; + this.statusSpan = document.getElementById('connection-status'); + this.debugLog = document.getElementById('debug-log'); + } + + /** + * Set up event listeners for connect/disconnect buttons + */ + private setupEventListeners(): void { + this.connectBtn?.addEventListener('click', () => this.connect()); + this.disconnectBtn?.addEventListener('click', () => this.disconnect()); + } + + /** + * Add a timestamped message to the debug log + */ + private log(message: string): void { + if (!this.debugLog) return; + const entry = document.createElement('div'); + entry.textContent = `${new Date().toISOString()} - ${message}`; + if (message.startsWith('User: ')) { + entry.style.color = '#2196F3'; + } else if (message.startsWith('Bot: ')) { + entry.style.color = '#4CAF50'; + } + this.debugLog.appendChild(entry); + this.debugLog.scrollTop = this.debugLog.scrollHeight; + console.log(message); + } + + /** + * Update the connection status display + */ + private updateStatus(status: string): void { + if (this.statusSpan) { + this.statusSpan.textContent = status; + } + this.log(`Status: ${status}`); + } + + private async emulateTwilioMessages() { + const connectedMessage={"event": "connected", "protocol": "Call", "version": "1.0.0"} + + const websocketTransport = this.rtviClient?.transport as WebSocketTransport + void websocketTransport?.sendRawMessage(connectedMessage) + + const startMessage={"event": "start", "start": {"streamSid": WebsocketClientApp.STREAM_SID, "callSid": WebsocketClientApp.CALL_SID}} + void websocketTransport?.sendRawMessage(startMessage) + } + + /** + * Check for available media tracks and set them up if present + * This is called when the bot is ready or when the transport state changes to ready + */ + setupMediaTracks() { + if (!this.rtviClient) return; + const tracks = this.rtviClient.tracks(); + if (tracks.bot?.audio) { + this.setupAudioTrack(tracks.bot.audio); + } + } + + /** + * Set up listeners for track events (start/stop) + * This handles new tracks being added during the session + */ + setupTrackListeners() { + if (!this.rtviClient) return; + + // Listen for new tracks starting + this.rtviClient.on(RTVIEvent.TrackStarted, (track, participant) => { + // Only handle non-local (bot) tracks + if (!participant?.local && track.kind === 'audio') { + this.setupAudioTrack(track); + } + }); + + // Listen for tracks stopping + this.rtviClient.on(RTVIEvent.TrackStopped, (track, participant) => { + this.log(`Track stopped: ${track.kind} from ${participant?.name || 'unknown'}`); + }); + } + + /** + * Set up an audio track for playback + * Handles both initial setup and track updates + */ + private setupAudioTrack(track: MediaStreamTrack): void { + this.log('Setting up audio track'); + if (this.botAudio.srcObject && "getAudioTracks" in this.botAudio.srcObject) { + const oldTrack = this.botAudio.srcObject.getAudioTracks()[0]; + if (oldTrack?.id === track.id) return; + } + this.botAudio.srcObject = new MediaStream([track]); + } + + /** + * Initialize and connect to the bot + * This sets up the RTVI client, initializes devices, and establishes the connection + */ + public async connect(): Promise { + try { + const startTime = Date.now(); + + const transport = new WebSocketTransport({ + serializer: new TwilioSerializer(), + recorderSampleRate: 8000, + playerSampleRate: 8000 + }); + const RTVIConfig: RTVIClientOptions = { + transport, + params: { + // The baseURL and endpoint of your bot server that the client will connect to + baseUrl: 'http://localhost:8765', + endpoints: { connect: '/' }, + }, + enableMic: true, + enableCam: false, + callbacks: { + onConnected: () => { + this.emulateTwilioMessages() + this.updateStatus('Connected'); + if (this.connectBtn) this.connectBtn.disabled = true; + if (this.disconnectBtn) this.disconnectBtn.disabled = false; + }, + onDisconnected: () => { + this.updateStatus('Disconnected'); + if (this.connectBtn) this.connectBtn.disabled = false; + if (this.disconnectBtn) this.disconnectBtn.disabled = true; + this.log('Client disconnected'); + }, + onBotReady: (data) => { + this.log(`Bot ready: ${JSON.stringify(data)}`); + this.setupMediaTracks(); + }, + onUserTranscript: (data) => { + if (data.final) { + this.log(`User: ${data.text}`); + } + }, + onBotTranscript: (data) => this.log(`Bot: ${data.text}`), + onMessageError: (error) => console.error('Message error:', error), + onError: (error) => console.error('Error:', error), + }, + } + // @ts-ignore + RTVIConfig.customConnectHandler = () => Promise.resolve( + { + ws_url: "/ws", + } + ); + this.rtviClient = new RTVIClient(RTVIConfig); + this.setupTrackListeners(); + + this.log('Initializing devices...'); + await this.rtviClient.initDevices(); + + this.log('Connecting to bot...'); + await this.rtviClient.connect(); + + const timeTaken = Date.now() - startTime; + this.log(`Connection complete, timeTaken: ${timeTaken}`); + } catch (error) { + this.log(`Error connecting: ${(error as Error).message}`); + this.updateStatus('Error'); + // Clean up if there's an error + if (this.rtviClient) { + try { + await this.rtviClient.disconnect(); + } catch (disconnectError) { + this.log(`Error during disconnect: ${disconnectError}`); + } + } + } + } + + /** + * Disconnect from the bot and clean up media resources + */ + public async disconnect(): Promise { + if (this.rtviClient) { + try { + await this.rtviClient.disconnect(); + this.rtviClient = null; + if (this.botAudio.srcObject && "getAudioTracks" in this.botAudio.srcObject) { + this.botAudio.srcObject.getAudioTracks().forEach((track) => track.stop()); + this.botAudio.srcObject = null; + } + } catch (error) { + this.log(`Error disconnecting: ${(error as Error).message}`); + } + } + } + +} + +declare global { + interface Window { + WebsocketClientApp: typeof WebsocketClientApp; + } +} + +window.addEventListener('DOMContentLoaded', () => { + window.WebsocketClientApp = WebsocketClientApp; + new WebsocketClientApp(); +}); diff --git a/examples/twilio-chatbot/client/typescript/src/style.css b/examples/twilio-chatbot/client/typescript/src/style.css new file mode 100644 index 000000000..9c147266e --- /dev/null +++ b/examples/twilio-chatbot/client/typescript/src/style.css @@ -0,0 +1,98 @@ +body { + margin: 0; + padding: 20px; + font-family: Arial, sans-serif; + background-color: #f0f0f0; +} + +.container { + max-width: 1200px; + margin: 0 auto; +} + +.status-bar { + display: flex; + justify-content: space-between; + align-items: center; + padding: 10px; + background-color: #fff; + border-radius: 8px; + margin-bottom: 20px; +} + +.controls button { + padding: 8px 16px; + margin-left: 10px; + border: none; + border-radius: 4px; + cursor: pointer; +} + +#connect-btn { + background-color: #4caf50; + color: white; +} + +#disconnect-btn { + background-color: #f44336; + color: white; +} + +button:disabled { + opacity: 0.5; + cursor: not-allowed; +} + +.main-content { + background-color: #fff; + border-radius: 8px; + padding: 20px; + margin-bottom: 20px; +} + +.bot-container { + display: flex; + flex-direction: column; + align-items: center; +} + +#bot-video-container { + width: 640px; + height: 360px; + background-color: #e0e0e0; + border-radius: 8px; + margin: 20px auto; + overflow: hidden; + display: flex; + align-items: center; + justify-content: center; +} + +#bot-video-container video { + width: 100%; + height: 100%; + object-fit: cover; +} + +.debug-panel { + background-color: #fff; + border-radius: 8px; + padding: 20px; +} + +.debug-panel h3 { + margin: 0 0 10px 0; + font-size: 16px; + font-weight: bold; +} + +#debug-log { + height: 500px; + overflow-y: auto; + background-color: #f8f8f8; + padding: 10px; + border-radius: 4px; + font-family: monospace; + font-size: 12px; + line-height: 1.4; +} diff --git a/examples/twilio-chatbot/client/typescript/tsconfig.json b/examples/twilio-chatbot/client/typescript/tsconfig.json new file mode 100644 index 000000000..c9c555d96 --- /dev/null +++ b/examples/twilio-chatbot/client/typescript/tsconfig.json @@ -0,0 +1,111 @@ +{ + "compilerOptions": { + /* Visit https://aka.ms/tsconfig to read more about this file */ + + /* Projects */ + // "incremental": true, /* Save .tsbuildinfo files to allow for incremental compilation of projects. */ + // "composite": true, /* Enable constraints that allow a TypeScript project to be used with project references. */ + // "tsBuildInfoFile": "./.tsbuildinfo", /* Specify the path to .tsbuildinfo incremental compilation file. */ + // "disableSourceOfProjectReferenceRedirect": true, /* Disable preferring source files instead of declaration files when referencing composite projects. */ + // "disableSolutionSearching": true, /* Opt a project out of multi-project reference checking when editing. */ + // "disableReferencedProjectLoad": true, /* Reduce the number of projects loaded automatically by TypeScript. */ + + /* Language and Environment */ + "target": "es2016", /* Set the JavaScript language version for emitted JavaScript and include compatible library declarations. */ + // "lib": [], /* Specify a set of bundled library declaration files that describe the target runtime environment. */ + // "jsx": "preserve", /* Specify what JSX code is generated. */ + // "experimentalDecorators": true, /* Enable experimental support for legacy experimental decorators. */ + // "emitDecoratorMetadata": true, /* Emit design-type metadata for decorated declarations in source files. */ + // "jsxFactory": "", /* Specify the JSX factory function used when targeting React JSX emit, e.g. 'React.createElement' or 'h'. */ + // "jsxFragmentFactory": "", /* Specify the JSX Fragment reference used for fragments when targeting React JSX emit e.g. 'React.Fragment' or 'Fragment'. */ + // "jsxImportSource": "", /* Specify module specifier used to import the JSX factory functions when using 'jsx: react-jsx*'. */ + // "reactNamespace": "", /* Specify the object invoked for 'createElement'. This only applies when targeting 'react' JSX emit. */ + // "noLib": true, /* Disable including any library files, including the default lib.d.ts. */ + // "useDefineForClassFields": true, /* Emit ECMAScript-standard-compliant class fields. */ + // "moduleDetection": "auto", /* Control what method is used to detect module-format JS files. */ + + /* Modules */ + "module": "commonjs", /* Specify what module code is generated. */ + // "rootDir": "./", /* Specify the root folder within your source files. */ + // "moduleResolution": "node10", /* Specify how TypeScript looks up a file from a given module specifier. */ + // "baseUrl": "./", /* Specify the base directory to resolve non-relative module names. */ + // "paths": {}, /* Specify a set of entries that re-map imports to additional lookup locations. */ + // "rootDirs": [], /* Allow multiple folders to be treated as one when resolving modules. */ + // "typeRoots": [], /* Specify multiple folders that act like './node_modules/@types'. */ + // "types": [], /* Specify type package names to be included without being referenced in a source file. */ + // "allowUmdGlobalAccess": true, /* Allow accessing UMD globals from modules. */ + // "moduleSuffixes": [], /* List of file name suffixes to search when resolving a module. */ + // "allowImportingTsExtensions": true, /* Allow imports to include TypeScript file extensions. Requires '--moduleResolution bundler' and either '--noEmit' or '--emitDeclarationOnly' to be set. */ + // "rewriteRelativeImportExtensions": true, /* Rewrite '.ts', '.tsx', '.mts', and '.cts' file extensions in relative import paths to their JavaScript equivalent in output files. */ + // "resolvePackageJsonExports": true, /* Use the package.json 'exports' field when resolving package imports. */ + // "resolvePackageJsonImports": true, /* Use the package.json 'imports' field when resolving imports. */ + // "customConditions": [], /* Conditions to set in addition to the resolver-specific defaults when resolving imports. */ + // "noUncheckedSideEffectImports": true, /* Check side effect imports. */ + // "resolveJsonModule": true, /* Enable importing .json files. */ + // "allowArbitraryExtensions": true, /* Enable importing files with any extension, provided a declaration file is present. */ + // "noResolve": true, /* Disallow 'import's, 'require's or ''s from expanding the number of files TypeScript should add to a project. */ + + /* JavaScript Support */ + // "allowJs": true, /* Allow JavaScript files to be a part of your program. Use the 'checkJS' option to get errors from these files. */ + // "checkJs": true, /* Enable error reporting in type-checked JavaScript files. */ + // "maxNodeModuleJsDepth": 1, /* Specify the maximum folder depth used for checking JavaScript files from 'node_modules'. Only applicable with 'allowJs'. */ + + /* Emit */ + // "declaration": true, /* Generate .d.ts files from TypeScript and JavaScript files in your project. */ + // "declarationMap": true, /* Create sourcemaps for d.ts files. */ + // "emitDeclarationOnly": true, /* Only output d.ts files and not JavaScript files. */ + // "sourceMap": true, /* Create source map files for emitted JavaScript files. */ + // "inlineSourceMap": true, /* Include sourcemap files inside the emitted JavaScript. */ + // "noEmit": true, /* Disable emitting files from a compilation. */ + // "outFile": "./", /* Specify a file that bundles all outputs into one JavaScript file. If 'declaration' is true, also designates a file that bundles all .d.ts output. */ + // "outDir": "./", /* Specify an output folder for all emitted files. */ + // "removeComments": true, /* Disable emitting comments. */ + // "importHelpers": true, /* Allow importing helper functions from tslib once per project, instead of including them per-file. */ + // "downlevelIteration": true, /* Emit more compliant, but verbose and less performant JavaScript for iteration. */ + // "sourceRoot": "", /* Specify the root path for debuggers to find the reference source code. */ + // "mapRoot": "", /* Specify the location where debugger should locate map files instead of generated locations. */ + // "inlineSources": true, /* Include source code in the sourcemaps inside the emitted JavaScript. */ + // "emitBOM": true, /* Emit a UTF-8 Byte Order Mark (BOM) in the beginning of output files. */ + // "newLine": "crlf", /* Set the newline character for emitting files. */ + // "stripInternal": true, /* Disable emitting declarations that have '@internal' in their JSDoc comments. */ + // "noEmitHelpers": true, /* Disable generating custom helper functions like '__extends' in compiled output. */ + // "noEmitOnError": true, /* Disable emitting files if any type checking errors are reported. */ + // "preserveConstEnums": true, /* Disable erasing 'const enum' declarations in generated code. */ + // "declarationDir": "./", /* Specify the output directory for generated declaration files. */ + + /* Interop Constraints */ + // "isolatedModules": true, /* Ensure that each file can be safely transpiled without relying on other imports. */ + // "verbatimModuleSyntax": true, /* Do not transform or elide any imports or exports not marked as type-only, ensuring they are written in the output file's format based on the 'module' setting. */ + // "isolatedDeclarations": true, /* Require sufficient annotation on exports so other tools can trivially generate declaration files. */ + // "allowSyntheticDefaultImports": true, /* Allow 'import x from y' when a module doesn't have a default export. */ + "esModuleInterop": true, /* Emit additional JavaScript to ease support for importing CommonJS modules. This enables 'allowSyntheticDefaultImports' for type compatibility. */ + // "preserveSymlinks": true, /* Disable resolving symlinks to their realpath. This correlates to the same flag in node. */ + "forceConsistentCasingInFileNames": true, /* Ensure that casing is correct in imports. */ + + /* Type Checking */ + "strict": true, /* Enable all strict type-checking options. */ + // "noImplicitAny": true, /* Enable error reporting for expressions and declarations with an implied 'any' type. */ + // "strictNullChecks": true, /* When type checking, take into account 'null' and 'undefined'. */ + // "strictFunctionTypes": true, /* When assigning functions, check to ensure parameters and the return values are subtype-compatible. */ + // "strictBindCallApply": true, /* Check that the arguments for 'bind', 'call', and 'apply' methods match the original function. */ + // "strictPropertyInitialization": true, /* Check for class properties that are declared but not set in the constructor. */ + // "strictBuiltinIteratorReturn": true, /* Built-in iterators are instantiated with a 'TReturn' type of 'undefined' instead of 'any'. */ + // "noImplicitThis": true, /* Enable error reporting when 'this' is given the type 'any'. */ + // "useUnknownInCatchVariables": true, /* Default catch clause variables as 'unknown' instead of 'any'. */ + // "alwaysStrict": true, /* Ensure 'use strict' is always emitted. */ + // "noUnusedLocals": true, /* Enable error reporting when local variables aren't read. */ + // "noUnusedParameters": true, /* Raise an error when a function parameter isn't read. */ + // "exactOptionalPropertyTypes": true, /* Interpret optional property types as written, rather than adding 'undefined'. */ + // "noImplicitReturns": true, /* Enable error reporting for codepaths that do not explicitly return in a function. */ + // "noFallthroughCasesInSwitch": true, /* Enable error reporting for fallthrough cases in switch statements. */ + // "noUncheckedIndexedAccess": true, /* Add 'undefined' to a type when accessed using an index. */ + // "noImplicitOverride": true, /* Ensure overriding members in derived classes are marked with an override modifier. */ + // "noPropertyAccessFromIndexSignature": true, /* Enforces using indexed accessors for keys declared using an indexed type. */ + // "allowUnusedLabels": true, /* Disable error reporting for unused labels. */ + // "allowUnreachableCode": true, /* Disable error reporting for unreachable code. */ + + /* Completeness */ + // "skipDefaultLibCheck": true, /* Skip type checking .d.ts files that are included with TypeScript. */ + "skipLibCheck": true /* Skip type checking all .d.ts files. */ + } +} diff --git a/examples/twilio-chatbot/client/typescript/vite.config.js b/examples/twilio-chatbot/client/typescript/vite.config.js new file mode 100644 index 000000000..6bcaa3bc8 --- /dev/null +++ b/examples/twilio-chatbot/client/typescript/vite.config.js @@ -0,0 +1,15 @@ +import { defineConfig } from 'vite'; +import react from '@vitejs/plugin-react-swc'; + +export default defineConfig({ + base: "./", //Use relative paths so it works at any mount path + plugins: [react()], + server: { + proxy: { + '/ws': { + target: 'ws://0.0.0.0:8765', // Replace with your backend URL + changeOrigin: true, + }, + }, + }, +}); diff --git a/examples/websocket-server/Dockerfile b/examples/websocket-server/Dockerfile deleted file mode 100644 index 0610ab7f8..000000000 --- a/examples/websocket-server/Dockerfile +++ /dev/null @@ -1,15 +0,0 @@ -FROM python:3.10-bullseye - -RUN mkdir /app - -COPY *.py /app/ -COPY requirements.txt /app/ -COPY .env /app/ - -WORKDIR /app - -RUN pip3 install -r requirements.txt - -EXPOSE 7860 - -CMD ["python3", "bot.py"] diff --git a/examples/websocket-server/README.md b/examples/websocket-server/README.md deleted file mode 100644 index a8f2f1aac..000000000 --- a/examples/websocket-server/README.md +++ /dev/null @@ -1,28 +0,0 @@ -# Websocket Server - -This is an example that shows how to use `WebsocketServerTransport` to communicate with a web client. - -## Get started - -```python -python3 -m venv venv -source venv/bin/activate -pip install -r requirements.txt -cp env.example .env # and add your credentials -``` - -## Run the bot - -```bash -python bot.py -``` - -## Run the HTTP server - -This will host the static web client: - -```bash -python -m http.server -``` - -Then, visit `http://localhost:8000` in your browser to start a session. diff --git a/examples/websocket-server/bot.py b/examples/websocket-server/bot.py deleted file mode 100644 index 816d7540b..000000000 --- a/examples/websocket-server/bot.py +++ /dev/null @@ -1,153 +0,0 @@ -# -# Copyright (c) 2024–2025, Daily -# -# SPDX-License-Identifier: BSD 2-Clause License -# - -import asyncio -import os -import sys - -from dotenv import load_dotenv -from loguru import logger - -from pipecat.audio.vad.silero import SileroVADAnalyzer -from pipecat.frames.frames import BotInterruptionFrame, EndFrame -from pipecat.pipeline.pipeline import Pipeline -from pipecat.pipeline.runner import PipelineRunner -from pipecat.pipeline.task import PipelineParams, PipelineTask -from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext -from pipecat.serializers.protobuf import ProtobufFrameSerializer -from pipecat.services.cartesia.tts import CartesiaTTSService -from pipecat.services.deepgram.stt import DeepgramSTTService -from pipecat.services.openai.llm import OpenAILLMService -from pipecat.transports.network.websocket_server import ( - WebsocketServerParams, - WebsocketServerTransport, -) - -load_dotenv(override=True) - -logger.remove(0) -logger.add(sys.stderr, level="DEBUG") - - -class SessionTimeoutHandler: - """Handles actions to be performed when a session times out. - Inputs: - - task: Pipeline task (used to queue frames). - - tts: TTS service (used to generate speech output). - """ - - def __init__(self, task, tts): - self.task = task - self.tts = tts - self.background_tasks = set() - - async def handle_timeout(self, client_address): - """Handles the timeout event for a session.""" - try: - logger.info(f"Connection timeout for {client_address}") - - # Queue a BotInterruptionFrame to notify the user - await self.task.queue_frames([BotInterruptionFrame()]) - - # Send the TTS message to inform the user about the timeout - await self.tts.say( - "I'm sorry, we are ending the call now. Please feel free to reach out again if you need assistance." - ) - - # Start the process to gracefully end the call in the background - end_call_task = asyncio.create_task(self._end_call()) - self.background_tasks.add(end_call_task) - end_call_task.add_done_callback(self.background_tasks.discard) - except Exception as e: - logger.error(f"Error during session timeout handling: {e}") - - async def _end_call(self): - """Completes the session termination process after the TTS message.""" - try: - # Wait for a duration to ensure TTS has completed - await asyncio.sleep(15) - - # Queue both BotInterruptionFrame and EndFrame to conclude the session - await self.task.queue_frames([BotInterruptionFrame(), EndFrame()]) - - logger.info("TTS completed and EndFrame pushed successfully.") - except Exception as e: - logger.error(f"Error during call termination: {e}") - - -async def main(): - transport = WebsocketServerTransport( - params=WebsocketServerParams( - serializer=ProtobufFrameSerializer(), - audio_in_enabled=True, - audio_out_enabled=True, - add_wav_header=True, - vad_analyzer=SileroVADAnalyzer(), - session_timeout=60 * 3, # 3 minutes - ) - ) - - llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY")) - - stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY")) - - tts = CartesiaTTSService( - api_key=os.getenv("CARTESIA_API_KEY"), - voice_id="71a7ad14-091c-4e8e-a314-022ece01c121", # British Reading Lady - ) - - messages = [ - { - "role": "system", - "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.", - }, - ] - - context = OpenAILLMContext(messages) - context_aggregator = llm.create_context_aggregator(context) - - pipeline = Pipeline( - [ - transport.input(), # Websocket input from client - stt, # Speech-To-Text - context_aggregator.user(), - llm, # LLM - tts, # Text-To-Speech - transport.output(), # Websocket output to client - context_aggregator.assistant(), - ] - ) - - task = PipelineTask( - pipeline, - params=PipelineParams( - audio_in_sample_rate=16000, - audio_out_sample_rate=16000, - allow_interruptions=True, - ), - ) - - @transport.event_handler("on_client_connected") - async def on_client_connected(transport, client): - # Kick off the conversation. - messages.append({"role": "system", "content": "Please introduce yourself to the user."}) - await task.queue_frames([context_aggregator.user().get_context_frame()]) - - @transport.event_handler("on_session_timeout") - async def on_session_timeout(transport, client): - logger.info(f"Entering in timeout for {client.remote_address}") - - timeout_handler = SessionTimeoutHandler(task, tts) - - await timeout_handler.handle_timeout(client) - - runner = PipelineRunner() - - await runner.run(task) - - -if __name__ == "__main__": - asyncio.run(main()) diff --git a/examples/websocket-server/env.example b/examples/websocket-server/env.example deleted file mode 100644 index c3359ada2..000000000 --- a/examples/websocket-server/env.example +++ /dev/null @@ -1,8 +0,0 @@ -# OpenAI API Key -OPENAI_API_KEY=your_openai_api_key_here - -# Deepgram API Key -DEEPGRAM_API_KEY=your_deepgram_api_key_here - -# Cartesia API Key -CARTESIA_API_KEY=your_cartesia_api_key_here diff --git a/examples/websocket-server/frames.proto b/examples/websocket-server/frames.proto deleted file mode 100644 index 98dc014db..000000000 --- a/examples/websocket-server/frames.proto +++ /dev/null @@ -1,44 +0,0 @@ -// -// Copyright (c) 2024–2025, Daily -// -// SPDX-License-Identifier: BSD 2-Clause License -// - -// Generate frames_pb2.py with: -// -// python -m grpc_tools.protoc --proto_path=./ --python_out=./protobufs frames.proto - -syntax = "proto3"; - -package pipecat; - -message TextFrame { - uint64 id = 1; - string name = 2; - string text = 3; -} - -message AudioRawFrame { - uint64 id = 1; - string name = 2; - bytes audio = 3; - uint32 sample_rate = 4; - uint32 num_channels = 5; - optional uint64 pts = 6; -} - -message TranscriptionFrame { - uint64 id = 1; - string name = 2; - string text = 3; - string user_id = 4; - string timestamp = 5; -} - -message Frame { - oneof frame { - TextFrame text = 1; - AudioRawFrame audio = 2; - TranscriptionFrame transcription = 3; - } -} diff --git a/examples/websocket-server/index.html b/examples/websocket-server/index.html deleted file mode 100644 index 6f0bd1ada..000000000 --- a/examples/websocket-server/index.html +++ /dev/null @@ -1,211 +0,0 @@ - - - - - - - - Pipecat WebSocket Client Example - - - -

Pipecat WebSocket Client Example

-

Loading, wait...

- - - - - - diff --git a/examples/websocket-server/requirements.txt b/examples/websocket-server/requirements.txt deleted file mode 100644 index ed2130a79..000000000 --- a/examples/websocket-server/requirements.txt +++ /dev/null @@ -1,2 +0,0 @@ -python-dotenv -pipecat-ai[cartesia,openai,silero,websocket,deepgram] diff --git a/examples/websocket/README.md b/examples/websocket/README.md new file mode 100644 index 000000000..38c23ddaa --- /dev/null +++ b/examples/websocket/README.md @@ -0,0 +1,54 @@ +# Voice Agent + +A Pipecat example demonstrating the simplest way to create a voice agent using `WebsocketTransport`. + +## 🚀 Quick Start + +### 1️⃣ Start the Bot Server + +#### 🔧 Set Up the Environment +1. Create and activate a virtual environment: + ```bash + python3 -m venv venv + source venv/bin/activate # On Windows: venv\Scripts\activate + ``` + +2. Install dependencies: + ```bash + pip install -r requirements.txt + ``` + +3. Configure environment variables: + - Copy `env.example` to `.env` + ```bash + cp env.example .env + ``` + - Add your API keys + - Choose what do you wish to use, 'fast_api' or 'websocket_server' + +#### ▶️ Run the Server +```bash +python server/server.py +``` + +### 3️⃣ Connect Using a Custom Client App + +For client-side setup, refer to the: +- [Typescript Guide](client/README.md). + +## ⚠️ Important Note +Ensure the bot server is running before using any client implementations. + +## 📌 Requirements + +- Python **3.10+** +- Node.js **16+** (for JavaScript components) +- Google API Key + +--- + +### 💡 Notes +- Ensure all dependencies are installed before running the server. +- Check the `.env` file for missing configurations. + +Happy coding! 🎉 \ No newline at end of file diff --git a/examples/websocket/client/README.md b/examples/websocket/client/README.md new file mode 100644 index 000000000..753c6d563 --- /dev/null +++ b/examples/websocket/client/README.md @@ -0,0 +1,27 @@ +# JavaScript Implementation + +Basic implementation using the [Pipecat JavaScript SDK](https://docs.pipecat.ai/client/js/introduction). + +## Setup + +1. Run the bot server. See the [server README](../README). + +2. Navigate to the `client/javascript` directory: + +```bash +cd client/javascript +``` + +3. Install dependencies: + +```bash +npm install +``` + +4. Run the client app: + +``` +npm run dev +``` + +5. Visit http://localhost:5173 in your browser. diff --git a/examples/websocket/client/index.html b/examples/websocket/client/index.html new file mode 100644 index 000000000..83c24031a --- /dev/null +++ b/examples/websocket/client/index.html @@ -0,0 +1,34 @@ + + + + + + + AI Chatbot + + + +
+
+
+ Transport: Disconnected +
+
+ + +
+
+ + + +
+

Debug Info

+
+
+
+ + + + + + diff --git a/examples/websocket/client/package-lock.json b/examples/websocket/client/package-lock.json new file mode 100644 index 000000000..bccbf27b4 --- /dev/null +++ b/examples/websocket/client/package-lock.json @@ -0,0 +1,1780 @@ +{ + "name": "client", + "version": "1.0.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "client", + "version": "1.0.0", + "license": "ISC", + "dependencies": { + "@pipecat-ai/client-js": "^0.4.0", + "@pipecat-ai/websocket-transport": "^0.4.2", + "protobufjs": "^7.4.0" + }, + "devDependencies": { + "@types/node": "^22.15.30", + "@types/protobufjs": "^6.0.0", + "@vitejs/plugin-react-swc": "^3.10.1", + "typescript": "^5.8.3", + "vite": "^6.3.5" + } + }, + "node_modules/@babel/runtime": { + "version": "7.27.6", + "resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.27.6.tgz", + "integrity": "sha512-vbavdySgbTTrmFE+EsiqUTzlOr5bzlnJtUv9PynGCAKvfQqjIXbvFdumPM/GxMDfyuGMJaJAU6TO4zc1Jf1i8Q==", + "license": "MIT", + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@bufbuild/protobuf": { + "version": "2.5.2", + "resolved": "https://registry.npmjs.org/@bufbuild/protobuf/-/protobuf-2.5.2.tgz", + "integrity": "sha512-foZ7qr0IsUBjzWIq+SuBLfdQCpJ1j8cTuNNT4owngTHoN5KsJb8L9t65fzz7SCeSWzescoOil/0ldqiL041ABg==", + "license": "(Apache-2.0 AND BSD-3-Clause)" + }, + "node_modules/@bufbuild/protoplugin": { + "version": "2.5.2", + "resolved": "https://registry.npmjs.org/@bufbuild/protoplugin/-/protoplugin-2.5.2.tgz", + "integrity": "sha512-7d/NUae/ugs/qgHEYOwkVWGDE3Bf/xjuGviVFs38+MLRdwiHNTiuvzPVwuIPo/1wuZCZn3Nax1cg1owLuY72xw==", + "license": "Apache-2.0", + "dependencies": { + "@bufbuild/protobuf": "2.5.2", + "@typescript/vfs": "^1.5.2", + "typescript": "5.4.5" + } + }, + "node_modules/@bufbuild/protoplugin/node_modules/typescript": { + "version": "5.4.5", + "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.4.5.tgz", + "integrity": "sha512-vcI4UpRgg81oIRUFwR0WSIHKt11nJ7SAVlYNIu+QpqeyXP+gpQJy/Z4+F0aGxSE4MqwjyXvW/TzgkLAx2AGHwQ==", + "license": "Apache-2.0", + "bin": { + "tsc": "bin/tsc", + "tsserver": "bin/tsserver" + }, + "engines": { + "node": ">=14.17" + } + }, + "node_modules/@daily-co/daily-js": { + "version": "0.79.0", + "resolved": "https://registry.npmjs.org/@daily-co/daily-js/-/daily-js-0.79.0.tgz", + "integrity": "sha512-Ii/Zi6cfTl2EZBpX8msRPNkkCHcajA+ErXpbN2Xe2KySd1Nb4IzC/QWJlSl9VA9pIlYPQicRTDoZnoym/0uEAw==", + "license": "BSD-2-Clause", + "dependencies": { + "@babel/runtime": "^7.12.5", + "@sentry/browser": "^8.33.1", + "bowser": "^2.8.1", + "dequal": "^2.0.3", + "events": "^3.1.0" + }, + "engines": { + "node": ">=10.0.0" + } + }, + "node_modules/@esbuild/aix-ppc64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.25.5.tgz", + "integrity": "sha512-9o3TMmpmftaCMepOdA5k/yDw8SfInyzWWTjYTFCX3kPSDJMROQTb8jg+h9Cnwnmm1vOzvxN7gIfB5V2ewpjtGA==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "aix" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-arm": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.25.5.tgz", + "integrity": "sha512-AdJKSPeEHgi7/ZhuIPtcQKr5RQdo6OO2IL87JkianiMYMPbCtot9fxPbrMiBADOWWm3T2si9stAiVsGbTQFkbA==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-arm64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.25.5.tgz", + "integrity": "sha512-VGzGhj4lJO+TVGV1v8ntCZWJktV7SGCs3Pn1GRWI1SBFtRALoomm8k5E9Pmwg3HOAal2VDc2F9+PM/rEY6oIDg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/android-x64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.25.5.tgz", + "integrity": "sha512-D2GyJT1kjvO//drbRT3Hib9XPwQeWd9vZoBJn+bu/lVsOZ13cqNdDeqIF/xQ5/VmWvMduP6AmXvylO/PIc2isw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/darwin-arm64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.25.5.tgz", + "integrity": "sha512-GtaBgammVvdF7aPIgH2jxMDdivezgFu6iKpmT+48+F8Hhg5J/sfnDieg0aeG/jfSvkYQU2/pceFPDKlqZzwnfQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/darwin-x64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.25.5.tgz", + "integrity": "sha512-1iT4FVL0dJ76/q1wd7XDsXrSW+oLoquptvh4CLR4kITDtqi2e/xwXwdCVH8hVHU43wgJdsq7Gxuzcs6Iq/7bxQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/freebsd-arm64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.25.5.tgz", + "integrity": "sha512-nk4tGP3JThz4La38Uy/gzyXtpkPW8zSAmoUhK9xKKXdBCzKODMc2adkB2+8om9BDYugz+uGV7sLmpTYzvmz6Sw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/freebsd-x64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.25.5.tgz", + "integrity": "sha512-PrikaNjiXdR2laW6OIjlbeuCPrPaAl0IwPIaRv+SMV8CiM8i2LqVUHFC1+8eORgWyY7yhQY+2U2fA55mBzReaw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-arm": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.25.5.tgz", + "integrity": "sha512-cPzojwW2okgh7ZlRpcBEtsX7WBuqbLrNXqLU89GxWbNt6uIg78ET82qifUy3W6OVww6ZWobWub5oqZOVtwolfw==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-arm64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.25.5.tgz", + "integrity": "sha512-Z9kfb1v6ZlGbWj8EJk9T6czVEjjq2ntSYLY2cw6pAZl4oKtfgQuS4HOq41M/BcoLPzrUbNd+R4BXFyH//nHxVg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-ia32": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.25.5.tgz", + "integrity": "sha512-sQ7l00M8bSv36GLV95BVAdhJ2QsIbCuCjh/uYrWiMQSUuV+LpXwIqhgJDcvMTj+VsQmqAHL2yYaasENvJ7CDKA==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-loong64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.25.5.tgz", + "integrity": "sha512-0ur7ae16hDUC4OL5iEnDb0tZHDxYmuQyhKhsPBV8f99f6Z9KQM02g33f93rNH5A30agMS46u2HP6qTdEt6Q1kg==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-mips64el": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.25.5.tgz", + "integrity": "sha512-kB/66P1OsHO5zLz0i6X0RxlQ+3cu0mkxS3TKFvkb5lin6uwZ/ttOkP3Z8lfR9mJOBk14ZwZ9182SIIWFGNmqmg==", + "cpu": [ + "mips64el" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-ppc64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.25.5.tgz", + "integrity": "sha512-UZCmJ7r9X2fe2D6jBmkLBMQetXPXIsZjQJCjgwpVDz+YMcS6oFR27alkgGv3Oqkv07bxdvw7fyB71/olceJhkQ==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-riscv64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.25.5.tgz", + "integrity": "sha512-kTxwu4mLyeOlsVIFPfQo+fQJAV9mh24xL+y+Bm6ej067sYANjyEw1dNHmvoqxJUCMnkBdKpvOn0Ahql6+4VyeA==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-s390x": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.25.5.tgz", + "integrity": "sha512-K2dSKTKfmdh78uJ3NcWFiqyRrimfdinS5ErLSn3vluHNeHVnBAFWC8a4X5N+7FgVE1EjXS1QDZbpqZBjfrqMTQ==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/linux-x64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.25.5.tgz", + "integrity": "sha512-uhj8N2obKTE6pSZ+aMUbqq+1nXxNjZIIjCjGLfsWvVpy7gKCOL6rsY1MhRh9zLtUtAI7vpgLMK6DxjO8Qm9lJw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/netbsd-arm64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.25.5.tgz", + "integrity": "sha512-pwHtMP9viAy1oHPvgxtOv+OkduK5ugofNTVDilIzBLpoWAM16r7b/mxBvfpuQDpRQFMfuVr5aLcn4yveGvBZvw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/netbsd-x64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.25.5.tgz", + "integrity": "sha512-WOb5fKrvVTRMfWFNCroYWWklbnXH0Q5rZppjq0vQIdlsQKuw6mdSihwSo4RV/YdQ5UCKKvBy7/0ZZYLBZKIbwQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/openbsd-arm64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.25.5.tgz", + "integrity": "sha512-7A208+uQKgTxHd0G0uqZO8UjK2R0DDb4fDmERtARjSHWxqMTye4Erz4zZafx7Di9Cv+lNHYuncAkiGFySoD+Mw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/openbsd-x64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.25.5.tgz", + "integrity": "sha512-G4hE405ErTWraiZ8UiSoesH8DaCsMm0Cay4fsFWOOUcz8b8rC6uCvnagr+gnioEjWn0wC+o1/TAHt+It+MpIMg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/sunos-x64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.25.5.tgz", + "integrity": "sha512-l+azKShMy7FxzY0Rj4RCt5VD/q8mG/e+mDivgspo+yL8zW7qEwctQ6YqKX34DTEleFAvCIUviCFX1SDZRSyMQA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "sunos" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-arm64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.25.5.tgz", + "integrity": "sha512-O2S7SNZzdcFG7eFKgvwUEZ2VG9D/sn/eIiz8XRZ1Q/DO5a3s76Xv0mdBzVM5j5R639lXQmPmSo0iRpHqUUrsxw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-ia32": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.25.5.tgz", + "integrity": "sha512-onOJ02pqs9h1iMJ1PQphR+VZv8qBMQ77Klcsqv9CNW2w6yLqoURLcgERAIurY6QE63bbLuqgP9ATqajFLK5AMQ==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@esbuild/win32-x64": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.25.5.tgz", + "integrity": "sha512-TXv6YnJ8ZMVdX+SXWVBo/0p8LTcrUYngpWjvm91TMjjBQii7Oz11Lw5lbDV5Y0TzuhSJHwiH4hEtC1I42mMS0g==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=18" + } + }, + "node_modules/@pipecat-ai/client-js": { + "version": "0.4.1", + "resolved": "https://registry.npmjs.org/@pipecat-ai/client-js/-/client-js-0.4.1.tgz", + "integrity": "sha512-3jLKRzeryqLxtkqvr4Bvxe2OxoI7mdOFecm6iolZizXnk/BE480SEg2oAKyov3b5oT6+jmPlT+1HRBlTzEtL7A==", + "license": "BSD-2-Clause", + "dependencies": { + "@types/events": "^3.0.3", + "clone-deep": "^4.0.1", + "events": "^3.3.0", + "typed-emitter": "^2.1.0", + "uuid": "^10.0.0" + } + }, + "node_modules/@pipecat-ai/websocket-transport": { + "version": "0.4.2", + "resolved": "https://registry.npmjs.org/@pipecat-ai/websocket-transport/-/websocket-transport-0.4.2.tgz", + "integrity": "sha512-mOYnw9n60usODrE35D+uhFbJXl0DqXV32pAqSHu1of049s128mex6Qv+W49DBMVr8h5W6pLGrXhm+XDAtN5leg==", + "license": "BSD-2-Clause", + "dependencies": { + "@daily-co/daily-js": "^0.79.0", + "@protobuf-ts/plugin": "^2.11.0", + "@protobuf-ts/runtime": "^2.11.0", + "x-law": "^0.3.1" + }, + "peerDependencies": { + "@pipecat-ai/client-js": "~0.4.0" + } + }, + "node_modules/@protobuf-ts/plugin": { + "version": "2.11.0", + "resolved": "https://registry.npmjs.org/@protobuf-ts/plugin/-/plugin-2.11.0.tgz", + "integrity": "sha512-Y+p4Axrk3thxws4BVSIO+x4CKWH2c8k3K+QPrp6Oq8agdsXPL/uwsMTIdpTdXIzTaUEZFASJL9LU56pob5GTHg==", + "license": "Apache-2.0", + "dependencies": { + "@bufbuild/protobuf": "^2.4.0", + "@bufbuild/protoplugin": "^2.4.0", + "@protobuf-ts/protoc": "^2.11.0", + "@protobuf-ts/runtime": "^2.11.0", + "@protobuf-ts/runtime-rpc": "^2.11.0", + "typescript": "^3.9" + }, + "bin": { + "protoc-gen-dump": "bin/protoc-gen-dump", + "protoc-gen-ts": "bin/protoc-gen-ts" + } + }, + "node_modules/@protobuf-ts/plugin/node_modules/typescript": { + "version": "3.9.10", + "resolved": "https://registry.npmjs.org/typescript/-/typescript-3.9.10.tgz", + "integrity": "sha512-w6fIxVE/H1PkLKcCPsFqKE7Kv7QUwhU8qQY2MueZXWx5cPZdwFupLgKK3vntcK98BtNHZtAF4LA/yl2a7k8R6Q==", + "license": "Apache-2.0", + "bin": { + "tsc": "bin/tsc", + "tsserver": "bin/tsserver" + }, + "engines": { + "node": ">=4.2.0" + } + }, + "node_modules/@protobuf-ts/protoc": { + "version": "2.11.0", + "resolved": "https://registry.npmjs.org/@protobuf-ts/protoc/-/protoc-2.11.0.tgz", + "integrity": "sha512-GYfmv1rjZ/7MWzUqMszhdXiuoa4Js/j6zCbcxFmeThBBUhbrXdPU42vY+QVCHL9PvAMXO+wEhUfPWYdd1YgnlA==", + "license": "Apache-2.0", + "bin": { + "protoc": "protoc.js" + } + }, + "node_modules/@protobuf-ts/runtime": { + "version": "2.11.0", + "resolved": "https://registry.npmjs.org/@protobuf-ts/runtime/-/runtime-2.11.0.tgz", + "integrity": "sha512-DfpRpUiNvPC3Kj48CmlU4HaIEY1Myh++PIumMmohBAk8/k0d2CkxYxJfPyUAxfuUfl97F4AvuCu1gXmfOG7OJQ==", + "license": "(Apache-2.0 AND BSD-3-Clause)" + }, + "node_modules/@protobuf-ts/runtime-rpc": { + "version": "2.11.0", + "resolved": "https://registry.npmjs.org/@protobuf-ts/runtime-rpc/-/runtime-rpc-2.11.0.tgz", + "integrity": "sha512-g/oMPym5LjVyCc3nlQc6cHer0R3CyleBos4p7CjRNzdKuH/FlRXzfQYo6EN5uv8vLtn7zEK9Cy4YBKvHStIaag==", + "license": "Apache-2.0", + "dependencies": { + "@protobuf-ts/runtime": "^2.11.0" + } + }, + "node_modules/@protobufjs/aspromise": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/aspromise/-/aspromise-1.1.2.tgz", + "integrity": "sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/base64": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/base64/-/base64-1.1.2.tgz", + "integrity": "sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/codegen": { + "version": "2.0.4", + "resolved": "https://registry.npmjs.org/@protobufjs/codegen/-/codegen-2.0.4.tgz", + "integrity": "sha512-YyFaikqM5sH0ziFZCN3xDC7zeGaB/d0IUb9CATugHWbd1FRFwWwt4ld4OYMPWu5a3Xe01mGAULCdqhMlPl29Jg==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/eventemitter": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@protobufjs/eventemitter/-/eventemitter-1.1.0.tgz", + "integrity": "sha512-j9ednRT81vYJ9OfVuXG6ERSTdEL1xVsNgqpkxMsbIabzSo3goCjDIveeGv5d03om39ML71RdmrGNjG5SReBP/Q==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/fetch": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@protobufjs/fetch/-/fetch-1.1.0.tgz", + "integrity": "sha512-lljVXpqXebpsijW71PZaCYeIcE5on1w5DlQy5WH6GLbFryLUrBD4932W/E2BSpfRJWseIL4v/KPgBFxDOIdKpQ==", + "license": "BSD-3-Clause", + "dependencies": { + "@protobufjs/aspromise": "^1.1.1", + "@protobufjs/inquire": "^1.1.0" + } + }, + "node_modules/@protobufjs/float": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/@protobufjs/float/-/float-1.0.2.tgz", + "integrity": "sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/inquire": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@protobufjs/inquire/-/inquire-1.1.0.tgz", + "integrity": "sha512-kdSefcPdruJiFMVSbn801t4vFK7KB/5gd2fYvrxhuJYg8ILrmn9SKSX2tZdV6V+ksulWqS7aXjBcRXl3wHoD9Q==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/path": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/path/-/path-1.1.2.tgz", + "integrity": "sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/pool": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@protobufjs/pool/-/pool-1.1.0.tgz", + "integrity": "sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/utf8": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@protobufjs/utf8/-/utf8-1.1.0.tgz", + "integrity": "sha512-Vvn3zZrhQZkkBE8LSuW3em98c0FwgO4nxzv6OdSxPKJIEKY2bGbHn+mhGIPerzI4twdxaP8/0+06HBpwf345Lw==", + "license": "BSD-3-Clause" + }, + "node_modules/@rolldown/pluginutils": { + "version": "1.0.0-beta.11", + "resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0-beta.11.tgz", + "integrity": "sha512-L/gAA/hyCSuzTF1ftlzUSI/IKr2POHsv1Dd78GfqkR83KMNuswWD61JxGV2L7nRwBBBSDr6R1gCkdTmoN7W4ag==", + "dev": true, + "license": "MIT" + }, + "node_modules/@rollup/rollup-android-arm-eabi": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm-eabi/-/rollup-android-arm-eabi-4.43.0.tgz", + "integrity": "sha512-Krjy9awJl6rKbruhQDgivNbD1WuLb8xAclM4IR4cN5pHGAs2oIMMQJEiC3IC/9TZJ+QZkmZhlMO/6MBGxPidpw==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ] + }, + "node_modules/@rollup/rollup-android-arm64": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm64/-/rollup-android-arm64-4.43.0.tgz", + "integrity": "sha512-ss4YJwRt5I63454Rpj+mXCXicakdFmKnUNxr1dLK+5rv5FJgAxnN7s31a5VchRYxCFWdmnDWKd0wbAdTr0J5EA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ] + }, + "node_modules/@rollup/rollup-darwin-arm64": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-arm64/-/rollup-darwin-arm64-4.43.0.tgz", + "integrity": "sha512-eKoL8ykZ7zz8MjgBenEF2OoTNFAPFz1/lyJ5UmmFSz5jW+7XbH1+MAgCVHy72aG59rbuQLcJeiMrP8qP5d/N0A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@rollup/rollup-darwin-x64": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-x64/-/rollup-darwin-x64-4.43.0.tgz", + "integrity": "sha512-SYwXJgaBYW33Wi/q4ubN+ldWC4DzQY62S4Ll2dgfr/dbPoF50dlQwEaEHSKrQdSjC6oIe1WgzosoaNoHCdNuMg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@rollup/rollup-freebsd-arm64": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-arm64/-/rollup-freebsd-arm64-4.43.0.tgz", + "integrity": "sha512-SV+U5sSo0yujrjzBF7/YidieK2iF6E7MdF6EbYxNz94lA+R0wKl3SiixGyG/9Klab6uNBIqsN7j4Y/Fya7wAjQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ] + }, + "node_modules/@rollup/rollup-freebsd-x64": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-x64/-/rollup-freebsd-x64-4.43.0.tgz", + "integrity": "sha512-J7uCsiV13L/VOeHJBo5SjasKiGxJ0g+nQTrBkAsmQBIdil3KhPnSE9GnRon4ejX1XDdsmK/l30IYLiAaQEO0Cg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ] + }, + "node_modules/@rollup/rollup-linux-arm-gnueabihf": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-gnueabihf/-/rollup-linux-arm-gnueabihf-4.43.0.tgz", + "integrity": "sha512-gTJ/JnnjCMc15uwB10TTATBEhK9meBIY+gXP4s0sHD1zHOaIh4Dmy1X9wup18IiY9tTNk5gJc4yx9ctj/fjrIw==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm-musleabihf": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-musleabihf/-/rollup-linux-arm-musleabihf-4.43.0.tgz", + "integrity": "sha512-ZJ3gZynL1LDSIvRfz0qXtTNs56n5DI2Mq+WACWZ7yGHFUEirHBRt7fyIk0NsCKhmRhn7WAcjgSkSVVxKlPNFFw==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm64-gnu": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-gnu/-/rollup-linux-arm64-gnu-4.43.0.tgz", + "integrity": "sha512-8FnkipasmOOSSlfucGYEu58U8cxEdhziKjPD2FIa0ONVMxvl/hmONtX/7y4vGjdUhjcTHlKlDhw3H9t98fPvyA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm64-musl": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-musl/-/rollup-linux-arm64-musl-4.43.0.tgz", + "integrity": "sha512-KPPyAdlcIZ6S9C3S2cndXDkV0Bb1OSMsX0Eelr2Bay4EsF9yi9u9uzc9RniK3mcUGCLhWY9oLr6er80P5DE6XA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-loongarch64-gnu": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-loongarch64-gnu/-/rollup-linux-loongarch64-gnu-4.43.0.tgz", + "integrity": "sha512-HPGDIH0/ZzAZjvtlXj6g+KDQ9ZMHfSP553za7o2Odegb/BEfwJcR0Sw0RLNpQ9nC6Gy8s+3mSS9xjZ0n3rhcYg==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-powerpc64le-gnu": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-powerpc64le-gnu/-/rollup-linux-powerpc64le-gnu-4.43.0.tgz", + "integrity": "sha512-gEmwbOws4U4GLAJDhhtSPWPXUzDfMRedT3hFMyRAvM9Mrnj+dJIFIeL7otsv2WF3D7GrV0GIewW0y28dOYWkmw==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-riscv64-gnu": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-gnu/-/rollup-linux-riscv64-gnu-4.43.0.tgz", + "integrity": "sha512-XXKvo2e+wFtXZF/9xoWohHg+MuRnvO29TI5Hqe9xwN5uN8NKUYy7tXUG3EZAlfchufNCTHNGjEx7uN78KsBo0g==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-riscv64-musl": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-musl/-/rollup-linux-riscv64-musl-4.43.0.tgz", + "integrity": "sha512-ruf3hPWhjw6uDFsOAzmbNIvlXFXlBQ4nk57Sec8E8rUxs/AI4HD6xmiiasOOx/3QxS2f5eQMKTAwk7KHwpzr/Q==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-s390x-gnu": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-s390x-gnu/-/rollup-linux-s390x-gnu-4.43.0.tgz", + "integrity": "sha512-QmNIAqDiEMEvFV15rsSnjoSmO0+eJLoKRD9EAa9rrYNwO/XRCtOGM3A5A0X+wmG+XRrw9Fxdsw+LnyYiZWWcVw==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-x64-gnu": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-gnu/-/rollup-linux-x64-gnu-4.43.0.tgz", + "integrity": "sha512-jAHr/S0iiBtFyzjhOkAics/2SrXE092qyqEg96e90L3t9Op8OTzS6+IX0Fy5wCt2+KqeHAkti+eitV0wvblEoQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-x64-musl": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-musl/-/rollup-linux-x64-musl-4.43.0.tgz", + "integrity": "sha512-3yATWgdeXyuHtBhrLt98w+5fKurdqvs8B53LaoKD7P7H7FKOONLsBVMNl9ghPQZQuYcceV5CDyPfyfGpMWD9mQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-win32-arm64-msvc": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-arm64-msvc/-/rollup-win32-arm64-msvc-4.43.0.tgz", + "integrity": "sha512-wVzXp2qDSCOpcBCT5WRWLmpJRIzv23valvcTwMHEobkjippNf+C3ys/+wf07poPkeNix0paTNemB2XrHr2TnGw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-ia32-msvc": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-ia32-msvc/-/rollup-win32-ia32-msvc-4.43.0.tgz", + "integrity": "sha512-fYCTEyzf8d+7diCw8b+asvWDCLMjsCEA8alvtAutqJOJp/wL5hs1rWSqJ1vkjgW0L2NB4bsYJrpKkiIPRR9dvw==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-x64-msvc": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-msvc/-/rollup-win32-x64-msvc-4.43.0.tgz", + "integrity": "sha512-SnGhLiE5rlK0ofq8kzuDkM0g7FN1s5VYY+YSMTibP7CqShxCQvqtNxTARS4xX4PFJfHjG0ZQYX9iGzI3FQh5Aw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@sentry-internal/browser-utils": { + "version": "8.55.0", + "resolved": "https://registry.npmjs.org/@sentry-internal/browser-utils/-/browser-utils-8.55.0.tgz", + "integrity": "sha512-ROgqtQfpH/82AQIpESPqPQe0UyWywKJsmVIqi3c5Fh+zkds5LUxnssTj3yNd1x+kxaPDVB023jAP+3ibNgeNDw==", + "license": "MIT", + "dependencies": { + "@sentry/core": "8.55.0" + }, + "engines": { + "node": ">=14.18" + } + }, + "node_modules/@sentry-internal/feedback": { + "version": "8.55.0", + "resolved": "https://registry.npmjs.org/@sentry-internal/feedback/-/feedback-8.55.0.tgz", + "integrity": "sha512-cP3BD/Q6pquVQ+YL+rwCnorKuTXiS9KXW8HNKu4nmmBAyf7urjs+F6Hr1k9MXP5yQ8W3yK7jRWd09Yu6DHWOiw==", + "license": "MIT", + "dependencies": { + "@sentry/core": "8.55.0" + }, + "engines": { + "node": ">=14.18" + } + }, + "node_modules/@sentry-internal/replay": { + "version": "8.55.0", + "resolved": "https://registry.npmjs.org/@sentry-internal/replay/-/replay-8.55.0.tgz", + "integrity": "sha512-roCDEGkORwolxBn8xAKedybY+Jlefq3xYmgN2fr3BTnsXjSYOPC7D1/mYqINBat99nDtvgFvNfRcZPiwwZ1hSw==", + "license": "MIT", + "dependencies": { + "@sentry-internal/browser-utils": "8.55.0", + "@sentry/core": "8.55.0" + }, + "engines": { + "node": ">=14.18" + } + }, + "node_modules/@sentry-internal/replay-canvas": { + "version": "8.55.0", + "resolved": "https://registry.npmjs.org/@sentry-internal/replay-canvas/-/replay-canvas-8.55.0.tgz", + "integrity": "sha512-nIkfgRWk1091zHdu4NbocQsxZF1rv1f7bbp3tTIlZYbrH62XVZosx5iHAuZG0Zc48AETLE7K4AX9VGjvQj8i9w==", + "license": "MIT", + "dependencies": { + "@sentry-internal/replay": "8.55.0", + "@sentry/core": "8.55.0" + }, + "engines": { + "node": ">=14.18" + } + }, + "node_modules/@sentry/browser": { + "version": "8.55.0", + "resolved": "https://registry.npmjs.org/@sentry/browser/-/browser-8.55.0.tgz", + "integrity": "sha512-1A31mCEWCjaMxJt6qGUK+aDnLDcK6AwLAZnqpSchNysGni1pSn1RWSmk9TBF8qyTds5FH8B31H480uxMPUJ7Cw==", + "license": "MIT", + "dependencies": { + "@sentry-internal/browser-utils": "8.55.0", + "@sentry-internal/feedback": "8.55.0", + "@sentry-internal/replay": "8.55.0", + "@sentry-internal/replay-canvas": "8.55.0", + "@sentry/core": "8.55.0" + }, + "engines": { + "node": ">=14.18" + } + }, + "node_modules/@sentry/core": { + "version": "8.55.0", + "resolved": "https://registry.npmjs.org/@sentry/core/-/core-8.55.0.tgz", + "integrity": "sha512-6g7jpbefjHYs821Z+EBJ8r4Z7LT5h80YSWRJaylGS4nW5W5Z2KXzpdnyFarv37O7QjauzVC2E+PABmpkw5/JGA==", + "license": "MIT", + "engines": { + "node": ">=14.18" + } + }, + "node_modules/@swc/core": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core/-/core-1.12.0.tgz", + "integrity": "sha512-/C0kiMHPY/HnLfqXYGMGxGck3A5Y3mqwxfv+EwHTPHGjAVRfHpWAEEBTSTF5C88vVY6CvwBEkhR2TX7t8Mahcw==", + "dev": true, + "hasInstallScript": true, + "license": "Apache-2.0", + "dependencies": { + "@swc/counter": "^0.1.3", + "@swc/types": "^0.1.22" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/swc" + }, + "optionalDependencies": { + "@swc/core-darwin-arm64": "1.12.0", + "@swc/core-darwin-x64": "1.12.0", + "@swc/core-linux-arm-gnueabihf": "1.12.0", + "@swc/core-linux-arm64-gnu": "1.12.0", + "@swc/core-linux-arm64-musl": "1.12.0", + "@swc/core-linux-x64-gnu": "1.12.0", + "@swc/core-linux-x64-musl": "1.12.0", + "@swc/core-win32-arm64-msvc": "1.12.0", + "@swc/core-win32-ia32-msvc": "1.12.0", + "@swc/core-win32-x64-msvc": "1.12.0" + }, + "peerDependencies": { + "@swc/helpers": ">=0.5.17" + }, + "peerDependenciesMeta": { + "@swc/helpers": { + "optional": true + } + } + }, + "node_modules/@swc/core-darwin-arm64": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core-darwin-arm64/-/core-darwin-arm64-1.12.0.tgz", + "integrity": "sha512-usLr8kC80GDv3pwH2zoEaS279kxtWY0MY3blbMFw7zA8fAjqxa8IDxm3WcgyNLNWckWn4asFfguEwz/Weem3nA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0 AND MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=10" + } + }, + "node_modules/@swc/core-darwin-x64": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core-darwin-x64/-/core-darwin-x64-1.12.0.tgz", + "integrity": "sha512-Cvv4sqDcTY7QF2Dh1vn2Xbt/1ENYQcpmrGHzITJrXzxA2aBopsz/n4yQDiyRxTR0t802m4xu0CzMoZIHvVruWQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0 AND MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=10" + } + }, + "node_modules/@swc/core-linux-arm-gnueabihf": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core-linux-arm-gnueabihf/-/core-linux-arm-gnueabihf-1.12.0.tgz", + "integrity": "sha512-seM4/XMJMOupkzfLfHl8sRa3NdhsVZp+XgwA/vVeYZYJE4wuWUxVzhCYzwmNftVY32eF2IiRaWnhG6ho6jusnQ==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=10" + } + }, + "node_modules/@swc/core-linux-arm64-gnu": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core-linux-arm64-gnu/-/core-linux-arm64-gnu-1.12.0.tgz", + "integrity": "sha512-Al0x33gUVxNY5tutEYpSyv7mze6qQS1ONa0HEwoRxcK9WXsX0NHLTiOSGZoCUS1SsXM37ONlbA6/Bsp1MQyP+g==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0 AND MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=10" + } + }, + "node_modules/@swc/core-linux-arm64-musl": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core-linux-arm64-musl/-/core-linux-arm64-musl-1.12.0.tgz", + "integrity": "sha512-OeFHz/5Hl9v75J9TYA5jQxNIYAZMqaiPpd9dYSTK2Xyqa/ZGgTtNyPhIwVfxx+9mHBf6+9c1mTlXUtACMtHmaQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0 AND MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=10" + } + }, + "node_modules/@swc/core-linux-x64-gnu": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core-linux-x64-gnu/-/core-linux-x64-gnu-1.12.0.tgz", + "integrity": "sha512-ltIvqNi7H0c5pRawyqjeYSKEIfZP4vv/datT3mwT6BW7muJtd1+KIDCPFLMIQ4wm/h76YQwPocsin3fzmnFdNA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0 AND MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=10" + } + }, + "node_modules/@swc/core-linux-x64-musl": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core-linux-x64-musl/-/core-linux-x64-musl-1.12.0.tgz", + "integrity": "sha512-Z/DhpjehaTK0uf+MhNB7mV9SuewpGs3P/q9/8+UsJeYoFr7yuOoPbAvrD6AqZkf6Bh7MRZ5OtG+KQgG5L+goiA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0 AND MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=10" + } + }, + "node_modules/@swc/core-win32-arm64-msvc": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core-win32-arm64-msvc/-/core-win32-arm64-msvc-1.12.0.tgz", + "integrity": "sha512-wHnvbfHIh2gfSbvuFT7qP97YCMUDh+fuiso+pcC6ug8IsMxuViNapHET4o0ZdFNWHhXJ7/s0e6w7mkOalsqQiQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0 AND MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=10" + } + }, + "node_modules/@swc/core-win32-ia32-msvc": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core-win32-ia32-msvc/-/core-win32-ia32-msvc-1.12.0.tgz", + "integrity": "sha512-88umlXwK+7J2p4DjfWHXQpmlZgCf1ayt6Ssj+PYlAfMCR0aBiJoAMwHWrvDXEozyOrsyP1j2X6WxbmA861vL5Q==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "Apache-2.0 AND MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=10" + } + }, + "node_modules/@swc/core-win32-x64-msvc": { + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@swc/core-win32-x64-msvc/-/core-win32-x64-msvc-1.12.0.tgz", + "integrity": "sha512-KR9TSRp+FEVOhbgTU6c94p/AYpsyBk7dIvlKQiDp8oKScUoyHG5yjmMBFN/BqUyTq4kj6zlgsY2rFE4R8/yqWg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0 AND MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=10" + } + }, + "node_modules/@swc/counter": { + "version": "0.1.3", + "resolved": "https://registry.npmjs.org/@swc/counter/-/counter-0.1.3.tgz", + "integrity": "sha512-e2BR4lsJkkRlKZ/qCHPw9ZaSxc0MVUd7gtbtaB7aMvHeJVYe8sOB8DBZkP2DtISHGSku9sCK6T6cnY0CtXrOCQ==", + "dev": true, + "license": "Apache-2.0" + }, + "node_modules/@swc/types": { + "version": "0.1.23", + "resolved": "https://registry.npmjs.org/@swc/types/-/types-0.1.23.tgz", + "integrity": "sha512-u1iIVZV9Q0jxY+yM2vw/hZGDNudsN85bBpTqzAQ9rzkxW9D+e3aEM4Han+ow518gSewkXgjmEK0BD79ZcNVgPw==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "@swc/counter": "^0.1.3" + } + }, + "node_modules/@types/estree": { + "version": "1.0.7", + "resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.7.tgz", + "integrity": "sha512-w28IoSUCJpidD/TGviZwwMJckNESJZXFu7NBZ5YJ4mEUnNraUn9Pm8HSZm/jDF1pDWYKspWE7oVphigUPRakIQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/@types/events": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/@types/events/-/events-3.0.3.tgz", + "integrity": "sha512-trOc4AAUThEz9hapPtSd7wf5tiQKvTtu5b371UxXdTuqzIh0ArcRspRP0i0Viu+LXstIQ1z96t1nsPxT9ol01g==", + "license": "MIT" + }, + "node_modules/@types/node": { + "version": "22.15.31", + "resolved": "https://registry.npmjs.org/@types/node/-/node-22.15.31.tgz", + "integrity": "sha512-jnVe5ULKl6tijxUhvQeNbQG/84fHfg+yMak02cT8QVhBx/F05rAVxCGBYYTh2EKz22D6JF5ktXuNwdx7b9iEGw==", + "license": "MIT", + "dependencies": { + "undici-types": "~6.21.0" + } + }, + "node_modules/@types/protobufjs": { + "version": "6.0.0", + "resolved": "https://registry.npmjs.org/@types/protobufjs/-/protobufjs-6.0.0.tgz", + "integrity": "sha512-A27RDExpAf3rdDjIrHKiJK6x8kqqJ4CmoChwtipfhVAn1p7+wviQFFP7dppn8FslSbHtQeVPvi8wNKkDjSYjHw==", + "deprecated": "This is a stub types definition for protobufjs (https://github.com/dcodeIO/ProtoBuf.js). protobufjs provides its own type definitions, so you don't need @types/protobufjs installed!", + "dev": true, + "license": "MIT", + "dependencies": { + "protobufjs": "*" + } + }, + "node_modules/@typescript/vfs": { + "version": "1.6.1", + "resolved": "https://registry.npmjs.org/@typescript/vfs/-/vfs-1.6.1.tgz", + "integrity": "sha512-JwoxboBh7Oz1v38tPbkrZ62ZXNHAk9bJ7c9x0eI5zBfBnBYGhURdbnh7Z4smN/MV48Y5OCcZb58n972UtbazsA==", + "license": "MIT", + "dependencies": { + "debug": "^4.1.1" + }, + "peerDependencies": { + "typescript": "*" + } + }, + "node_modules/@vitejs/plugin-react-swc": { + "version": "3.10.2", + "resolved": "https://registry.npmjs.org/@vitejs/plugin-react-swc/-/plugin-react-swc-3.10.2.tgz", + "integrity": "sha512-xD3Rdvrt5LgANug7WekBn1KhcvLn1H3jNBfJRL3reeOIua/WnZOEV5qi5qIBq5T8R0jUDmRtxuvk4bPhzGHDWw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@rolldown/pluginutils": "1.0.0-beta.11", + "@swc/core": "^1.11.31" + }, + "peerDependencies": { + "vite": "^4 || ^5 || ^6 || ^7.0.0-beta.0" + } + }, + "node_modules/bowser": { + "version": "2.11.0", + "resolved": "https://registry.npmjs.org/bowser/-/bowser-2.11.0.tgz", + "integrity": "sha512-AlcaJBi/pqqJBIQ8U9Mcpc9i8Aqxn88Skv5d+xBX006BY5u8N3mGLHa5Lgppa7L/HfwgwLgZ6NYs+Ag6uUmJRA==", + "license": "MIT" + }, + "node_modules/clone-deep": { + "version": "4.0.1", + "resolved": "https://registry.npmjs.org/clone-deep/-/clone-deep-4.0.1.tgz", + "integrity": "sha512-neHB9xuzh/wk0dIHweyAXv2aPGZIVk3pLMe+/RNzINf17fe0OG96QroktYAUm7SM1PBnzTabaLboqqxDyMU+SQ==", + "license": "MIT", + "dependencies": { + "is-plain-object": "^2.0.4", + "kind-of": "^6.0.2", + "shallow-clone": "^3.0.0" + }, + "engines": { + "node": ">=6" + } + }, + "node_modules/debug": { + "version": "4.4.1", + "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.1.tgz", + "integrity": "sha512-KcKCqiftBJcZr++7ykoDIEwSa3XWowTfNPo92BYxjXiyYEVrUQh2aLyhxBCwww+heortUFxEJYcRzosstTEBYQ==", + "license": "MIT", + "dependencies": { + "ms": "^2.1.3" + }, + "engines": { + "node": ">=6.0" + }, + "peerDependenciesMeta": { + "supports-color": { + "optional": true + } + } + }, + "node_modules/dequal": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/dequal/-/dequal-2.0.3.tgz", + "integrity": "sha512-0je+qPKHEMohvfRTCEo3CrPG6cAzAYgmzKyxRiYSSDkS6eGJdyVJm7WaYA5ECaAD9wLB2T4EEeymA5aFVcYXCA==", + "license": "MIT", + "engines": { + "node": ">=6" + } + }, + "node_modules/esbuild": { + "version": "0.25.5", + "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.25.5.tgz", + "integrity": "sha512-P8OtKZRv/5J5hhz0cUAdu/cLuPIKXpQl1R9pZtvmHWQvrAUVd0UNIPT4IB4W3rNOqVO0rlqHmCIbSwxh/c9yUQ==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "bin": { + "esbuild": "bin/esbuild" + }, + "engines": { + "node": ">=18" + }, + "optionalDependencies": { + "@esbuild/aix-ppc64": "0.25.5", + "@esbuild/android-arm": "0.25.5", + "@esbuild/android-arm64": "0.25.5", + "@esbuild/android-x64": "0.25.5", + "@esbuild/darwin-arm64": "0.25.5", + "@esbuild/darwin-x64": "0.25.5", + "@esbuild/freebsd-arm64": "0.25.5", + "@esbuild/freebsd-x64": "0.25.5", + "@esbuild/linux-arm": "0.25.5", + "@esbuild/linux-arm64": "0.25.5", + "@esbuild/linux-ia32": "0.25.5", + "@esbuild/linux-loong64": "0.25.5", + "@esbuild/linux-mips64el": "0.25.5", + "@esbuild/linux-ppc64": "0.25.5", + "@esbuild/linux-riscv64": "0.25.5", + "@esbuild/linux-s390x": "0.25.5", + "@esbuild/linux-x64": "0.25.5", + "@esbuild/netbsd-arm64": "0.25.5", + "@esbuild/netbsd-x64": "0.25.5", + "@esbuild/openbsd-arm64": "0.25.5", + "@esbuild/openbsd-x64": "0.25.5", + "@esbuild/sunos-x64": "0.25.5", + "@esbuild/win32-arm64": "0.25.5", + "@esbuild/win32-ia32": "0.25.5", + "@esbuild/win32-x64": "0.25.5" + } + }, + "node_modules/events": { + "version": "3.3.0", + "resolved": "https://registry.npmjs.org/events/-/events-3.3.0.tgz", + "integrity": "sha512-mQw+2fkQbALzQ7V0MY0IqdnXNOeTtP4r0lN9z7AAawCXgqea7bDii20AYrIBrFd/Hx0M2Ocz6S111CaFkUcb0Q==", + "license": "MIT", + "engines": { + "node": ">=0.8.x" + } + }, + "node_modules/fdir": { + "version": "6.4.6", + "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.4.6.tgz", + "integrity": "sha512-hiFoqpyZcfNm1yc4u8oWCf9A2c4D3QjCrks3zmoVKVxpQRzmPNar1hUJcBG2RQHvEVGDN+Jm81ZheVLAQMK6+w==", + "dev": true, + "license": "MIT", + "peerDependencies": { + "picomatch": "^3 || ^4" + }, + "peerDependenciesMeta": { + "picomatch": { + "optional": true + } + } + }, + "node_modules/fsevents": { + "version": "2.3.3", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", + "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, + "node_modules/is-plain-object": { + "version": "2.0.4", + "resolved": "https://registry.npmjs.org/is-plain-object/-/is-plain-object-2.0.4.tgz", + "integrity": "sha512-h5PpgXkWitc38BBMYawTYMWJHFZJVnBquFE57xFpjB8pJFiF6gZ+bU+WyI/yqXiFR5mdLsgYNaPe8uao6Uv9Og==", + "license": "MIT", + "dependencies": { + "isobject": "^3.0.1" + }, + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/isobject": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/isobject/-/isobject-3.0.1.tgz", + "integrity": "sha512-WhB9zCku7EGTj/HQQRz5aUQEUeoQZH2bWcltRErOpymJ4boYE6wL9Tbr23krRPSZ+C5zqNSrSw+Cc7sZZ4b7vg==", + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/kind-of": { + "version": "6.0.3", + "resolved": "https://registry.npmjs.org/kind-of/-/kind-of-6.0.3.tgz", + "integrity": "sha512-dcS1ul+9tmeD95T+x28/ehLgd9mENa3LsvDTtzm3vyBEO7RPptvAD+t44WVXaUjTBRcrpFeFlC8WCruUR456hw==", + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/long": { + "version": "5.3.2", + "resolved": "https://registry.npmjs.org/long/-/long-5.3.2.tgz", + "integrity": "sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA==", + "license": "Apache-2.0" + }, + "node_modules/ms": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", + "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", + "license": "MIT" + }, + "node_modules/nanoid": { + "version": "3.3.11", + "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.11.tgz", + "integrity": "sha512-N8SpfPUnUp1bK+PMYW8qSWdl9U+wwNWI4QKxOYDy9JAro3WMX7p2OeVRF9v+347pnakNevPmiHhNmZ2HbFA76w==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "bin": { + "nanoid": "bin/nanoid.cjs" + }, + "engines": { + "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1" + } + }, + "node_modules/picocolors": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", + "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==", + "dev": true, + "license": "ISC" + }, + "node_modules/picomatch": { + "version": "4.0.2", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.2.tgz", + "integrity": "sha512-M7BAV6Rlcy5u+m6oPhAPFgJTzAioX/6B0DxyvDlo9l8+T3nLKbrczg2WLUyzd45L8RqfUMyGPzekbMvX2Ldkwg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/jonschlinkert" + } + }, + "node_modules/postcss": { + "version": "8.5.5", + "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.5.tgz", + "integrity": "sha512-d/jtm+rdNT8tpXuHY5MMtcbJFBkhXE6593XVR9UoGCH8jSFGci7jGvMGH5RYd5PBJW+00NZQt6gf7CbagJCrhg==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/postcss/" + }, + { + "type": "tidelift", + "url": "https://tidelift.com/funding/github/npm/postcss" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "dependencies": { + "nanoid": "^3.3.11", + "picocolors": "^1.1.1", + "source-map-js": "^1.2.1" + }, + "engines": { + "node": "^10 || ^12 || >=14" + } + }, + "node_modules/protobufjs": { + "version": "7.5.3", + "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-7.5.3.tgz", + "integrity": "sha512-sildjKwVqOI2kmFDiXQ6aEB0fjYTafpEvIBs8tOR8qI4spuL9OPROLVu2qZqi/xgCfsHIwVqlaF8JBjWFHnKbw==", + "hasInstallScript": true, + "license": "BSD-3-Clause", + "dependencies": { + "@protobufjs/aspromise": "^1.1.2", + "@protobufjs/base64": "^1.1.2", + "@protobufjs/codegen": "^2.0.4", + "@protobufjs/eventemitter": "^1.1.0", + "@protobufjs/fetch": "^1.1.0", + "@protobufjs/float": "^1.0.2", + "@protobufjs/inquire": "^1.1.0", + "@protobufjs/path": "^1.1.2", + "@protobufjs/pool": "^1.1.0", + "@protobufjs/utf8": "^1.1.0", + "@types/node": ">=13.7.0", + "long": "^5.0.0" + }, + "engines": { + "node": ">=12.0.0" + } + }, + "node_modules/rollup": { + "version": "4.43.0", + "resolved": "https://registry.npmjs.org/rollup/-/rollup-4.43.0.tgz", + "integrity": "sha512-wdN2Kd3Twh8MAEOEJZsuxuLKCsBEo4PVNLK6tQWAn10VhsVewQLzcucMgLolRlhFybGxfclbPeEYBaP6RvUFGg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/estree": "1.0.7" + }, + "bin": { + "rollup": "dist/bin/rollup" + }, + "engines": { + "node": ">=18.0.0", + "npm": ">=8.0.0" + }, + "optionalDependencies": { + "@rollup/rollup-android-arm-eabi": "4.43.0", + "@rollup/rollup-android-arm64": "4.43.0", + "@rollup/rollup-darwin-arm64": "4.43.0", + "@rollup/rollup-darwin-x64": "4.43.0", + "@rollup/rollup-freebsd-arm64": "4.43.0", + "@rollup/rollup-freebsd-x64": "4.43.0", + "@rollup/rollup-linux-arm-gnueabihf": "4.43.0", + "@rollup/rollup-linux-arm-musleabihf": "4.43.0", + "@rollup/rollup-linux-arm64-gnu": "4.43.0", + "@rollup/rollup-linux-arm64-musl": "4.43.0", + "@rollup/rollup-linux-loongarch64-gnu": "4.43.0", + "@rollup/rollup-linux-powerpc64le-gnu": "4.43.0", + "@rollup/rollup-linux-riscv64-gnu": "4.43.0", + "@rollup/rollup-linux-riscv64-musl": "4.43.0", + "@rollup/rollup-linux-s390x-gnu": "4.43.0", + "@rollup/rollup-linux-x64-gnu": "4.43.0", + "@rollup/rollup-linux-x64-musl": "4.43.0", + "@rollup/rollup-win32-arm64-msvc": "4.43.0", + "@rollup/rollup-win32-ia32-msvc": "4.43.0", + "@rollup/rollup-win32-x64-msvc": "4.43.0", + "fsevents": "~2.3.2" + } + }, + "node_modules/rxjs": { + "version": "7.8.2", + "resolved": "https://registry.npmjs.org/rxjs/-/rxjs-7.8.2.tgz", + "integrity": "sha512-dhKf903U/PQZY6boNNtAGdWbG85WAbjT/1xYoZIC7FAY0yWapOBQVsVrDl58W86//e1VpMNBtRV4MaXfdMySFA==", + "license": "Apache-2.0", + "optional": true, + "dependencies": { + "tslib": "^2.1.0" + } + }, + "node_modules/shallow-clone": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/shallow-clone/-/shallow-clone-3.0.1.tgz", + "integrity": "sha512-/6KqX+GVUdqPuPPd2LxDDxzX6CAbjJehAAOKlNpqqUpAqPM6HeL8f+o3a+JsyGjn2lv0WY8UsTgUJjU9Ok55NA==", + "license": "MIT", + "dependencies": { + "kind-of": "^6.0.2" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/source-map-js": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz", + "integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==", + "dev": true, + "license": "BSD-3-Clause", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/tinyglobby": { + "version": "0.2.14", + "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.14.tgz", + "integrity": "sha512-tX5e7OM1HnYr2+a2C/4V0htOcSQcoSTH9KgJnVvNm5zm/cyEWKJ7j7YutsH9CxMdtOkkLFy2AHrMci9IM8IPZQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "fdir": "^6.4.4", + "picomatch": "^4.0.2" + }, + "engines": { + "node": ">=12.0.0" + }, + "funding": { + "url": "https://github.com/sponsors/SuperchupuDev" + } + }, + "node_modules/tslib": { + "version": "2.8.1", + "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz", + "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", + "license": "0BSD", + "optional": true + }, + "node_modules/typed-emitter": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/typed-emitter/-/typed-emitter-2.1.0.tgz", + "integrity": "sha512-g/KzbYKbH5C2vPkaXGu8DJlHrGKHLsM25Zg9WuC9pMGfuvT+X25tZQWo5fK1BjBm8+UrVE9LDCvaY0CQk+fXDA==", + "license": "MIT", + "optionalDependencies": { + "rxjs": "*" + } + }, + "node_modules/typescript": { + "version": "5.8.3", + "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.8.3.tgz", + "integrity": "sha512-p1diW6TqL9L07nNxvRMM7hMMw4c5XOo/1ibL4aAIGmSAt9slTE1Xgw5KWuof2uTOvCg9BY7ZRi+GaF+7sfgPeQ==", + "license": "Apache-2.0", + "bin": { + "tsc": "bin/tsc", + "tsserver": "bin/tsserver" + }, + "engines": { + "node": ">=14.17" + } + }, + "node_modules/undici-types": { + "version": "6.21.0", + "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz", + "integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==", + "license": "MIT" + }, + "node_modules/uuid": { + "version": "10.0.0", + "resolved": "https://registry.npmjs.org/uuid/-/uuid-10.0.0.tgz", + "integrity": "sha512-8XkAphELsDnEGrDxUOHB3RGvXz6TeuYSGEZBOjtTtPm2lwhGBjLgOzLHB63IUWfBpNucQjND6d3AOudO+H3RWQ==", + "funding": [ + "https://github.com/sponsors/broofa", + "https://github.com/sponsors/ctavan" + ], + "license": "MIT", + "bin": { + "uuid": "dist/bin/uuid" + } + }, + "node_modules/vite": { + "version": "6.3.5", + "resolved": "https://registry.npmjs.org/vite/-/vite-6.3.5.tgz", + "integrity": "sha512-cZn6NDFE7wdTpINgs++ZJ4N49W2vRp8LCKrn3Ob1kYNtOo21vfDoaV5GzBfLU4MovSAB8uNRm4jgzVQZ+mBzPQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "esbuild": "^0.25.0", + "fdir": "^6.4.4", + "picomatch": "^4.0.2", + "postcss": "^8.5.3", + "rollup": "^4.34.9", + "tinyglobby": "^0.2.13" + }, + "bin": { + "vite": "bin/vite.js" + }, + "engines": { + "node": "^18.0.0 || ^20.0.0 || >=22.0.0" + }, + "funding": { + "url": "https://github.com/vitejs/vite?sponsor=1" + }, + "optionalDependencies": { + "fsevents": "~2.3.3" + }, + "peerDependencies": { + "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0", + "jiti": ">=1.21.0", + "less": "*", + "lightningcss": "^1.21.0", + "sass": "*", + "sass-embedded": "*", + "stylus": "*", + "sugarss": "*", + "terser": "^5.16.0", + "tsx": "^4.8.1", + "yaml": "^2.4.2" + }, + "peerDependenciesMeta": { + "@types/node": { + "optional": true + }, + "jiti": { + "optional": true + }, + "less": { + "optional": true + }, + "lightningcss": { + "optional": true + }, + "sass": { + "optional": true + }, + "sass-embedded": { + "optional": true + }, + "stylus": { + "optional": true + }, + "sugarss": { + "optional": true + }, + "terser": { + "optional": true + }, + "tsx": { + "optional": true + }, + "yaml": { + "optional": true + } + } + }, + "node_modules/x-law": { + "version": "0.3.1", + "resolved": "https://registry.npmjs.org/x-law/-/x-law-0.3.1.tgz", + "integrity": "sha512-Nvo6OKj6UL2LuzAc08uJkwIDkK2PsTEdpLiY82NkwMptuRpAA1V7arUl7ZY12BcgRYNq8uh1pdAu7G6VeQn7Hg==", + "license": "MIT", + "engines": { + "node": ">=18" + } + } + } +} diff --git a/examples/websocket/client/package.json b/examples/websocket/client/package.json new file mode 100644 index 000000000..21a8dc95c --- /dev/null +++ b/examples/websocket/client/package.json @@ -0,0 +1,26 @@ +{ + "name": "client", + "version": "1.0.0", + "main": "index.js", + "scripts": { + "dev": "vite", + "build": "tsc && vite build", + "preview": "vite preview" + }, + "keywords": [], + "author": "", + "license": "ISC", + "description": "", + "devDependencies": { + "@types/node": "^22.15.30", + "@types/protobufjs": "^6.0.0", + "@vitejs/plugin-react-swc": "^3.10.1", + "typescript": "^5.8.3", + "vite": "^6.3.5" + }, + "dependencies": { + "@pipecat-ai/client-js": "^0.4.0", + "@pipecat-ai/websocket-transport": "^0.4.2", + "protobufjs": "^7.4.0" + } +} diff --git a/examples/websocket/client/src/app.ts b/examples/websocket/client/src/app.ts new file mode 100644 index 000000000..f583d353c --- /dev/null +++ b/examples/websocket/client/src/app.ts @@ -0,0 +1,234 @@ +/** + * Copyright (c) 2024–2025, Daily + * + * SPDX-License-Identifier: BSD 2-Clause License + */ + +/** + * RTVI Client Implementation + * + * This client connects to an RTVI-compatible bot server using WebSocket. + * + * Requirements: + * - A running RTVI bot server (defaults to http://localhost:7860) + */ + +import { + RTVIClient, + RTVIClientOptions, + RTVIEvent, +} from '@pipecat-ai/client-js'; +import { + WebSocketTransport +} from "@pipecat-ai/websocket-transport"; + +class WebsocketClientApp { + private rtviClient: RTVIClient | null = null; + private connectBtn: HTMLButtonElement | null = null; + private disconnectBtn: HTMLButtonElement | null = null; + private statusSpan: HTMLElement | null = null; + private debugLog: HTMLElement | null = null; + private botAudio: HTMLAudioElement; + + constructor() { + console.log("WebsocketClientApp"); + this.botAudio = document.createElement('audio'); + this.botAudio.autoplay = true; + //this.botAudio.playsInline = true; + document.body.appendChild(this.botAudio); + + this.setupDOMElements(); + this.setupEventListeners(); + } + + /** + * Set up references to DOM elements and create necessary media elements + */ + private setupDOMElements(): void { + this.connectBtn = document.getElementById('connect-btn') as HTMLButtonElement; + this.disconnectBtn = document.getElementById('disconnect-btn') as HTMLButtonElement; + this.statusSpan = document.getElementById('connection-status'); + this.debugLog = document.getElementById('debug-log'); + } + + /** + * Set up event listeners for connect/disconnect buttons + */ + private setupEventListeners(): void { + this.connectBtn?.addEventListener('click', () => this.connect()); + this.disconnectBtn?.addEventListener('click', () => this.disconnect()); + } + + /** + * Add a timestamped message to the debug log + */ + private log(message: string): void { + if (!this.debugLog) return; + const entry = document.createElement('div'); + entry.textContent = `${new Date().toISOString()} - ${message}`; + if (message.startsWith('User: ')) { + entry.style.color = '#2196F3'; + } else if (message.startsWith('Bot: ')) { + entry.style.color = '#4CAF50'; + } + this.debugLog.appendChild(entry); + this.debugLog.scrollTop = this.debugLog.scrollHeight; + console.log(message); + } + + /** + * Update the connection status display + */ + private updateStatus(status: string): void { + if (this.statusSpan) { + this.statusSpan.textContent = status; + } + this.log(`Status: ${status}`); + } + + /** + * Check for available media tracks and set them up if present + * This is called when the bot is ready or when the transport state changes to ready + */ + setupMediaTracks() { + if (!this.rtviClient) return; + const tracks = this.rtviClient.tracks(); + if (tracks.bot?.audio) { + this.setupAudioTrack(tracks.bot.audio); + } + } + + /** + * Set up listeners for track events (start/stop) + * This handles new tracks being added during the session + */ + setupTrackListeners() { + if (!this.rtviClient) return; + + // Listen for new tracks starting + this.rtviClient.on(RTVIEvent.TrackStarted, (track, participant) => { + // Only handle non-local (bot) tracks + if (!participant?.local && track.kind === 'audio') { + this.setupAudioTrack(track); + } + }); + + // Listen for tracks stopping + this.rtviClient.on(RTVIEvent.TrackStopped, (track, participant) => { + this.log(`Track stopped: ${track.kind} from ${participant?.name || 'unknown'}`); + }); + } + + /** + * Set up an audio track for playback + * Handles both initial setup and track updates + */ + private setupAudioTrack(track: MediaStreamTrack): void { + this.log('Setting up audio track'); + if (this.botAudio.srcObject && "getAudioTracks" in this.botAudio.srcObject) { + const oldTrack = this.botAudio.srcObject.getAudioTracks()[0]; + if (oldTrack?.id === track.id) return; + } + this.botAudio.srcObject = new MediaStream([track]); + } + + /** + * Initialize and connect to the bot + * This sets up the RTVI client, initializes devices, and establishes the connection + */ + public async connect(): Promise { + try { + const startTime = Date.now(); + + //const transport = new DailyTransport(); + const transport = new WebSocketTransport(); + const RTVIConfig: RTVIClientOptions = { + transport, + params: { + // The baseURL and endpoint of your bot server that the client will connect to + baseUrl: 'http://localhost:7860', + endpoints: { connect: '/connect' }, + }, + enableMic: true, + enableCam: false, + callbacks: { + onConnected: () => { + this.updateStatus('Connected'); + if (this.connectBtn) this.connectBtn.disabled = true; + if (this.disconnectBtn) this.disconnectBtn.disabled = false; + }, + onDisconnected: () => { + this.updateStatus('Disconnected'); + if (this.connectBtn) this.connectBtn.disabled = false; + if (this.disconnectBtn) this.disconnectBtn.disabled = true; + this.log('Client disconnected'); + }, + onBotReady: (data) => { + this.log(`Bot ready: ${JSON.stringify(data)}`); + this.setupMediaTracks(); + }, + onUserTranscript: (data) => { + if (data.final) { + this.log(`User: ${data.text}`); + } + }, + onBotTranscript: (data) => this.log(`Bot: ${data.text}`), + onMessageError: (error) => console.error('Message error:', error), + onError: (error) => console.error('Error:', error), + }, + } + this.rtviClient = new RTVIClient(RTVIConfig); + this.setupTrackListeners(); + + this.log('Initializing devices...'); + await this.rtviClient.initDevices(); + + this.log('Connecting to bot...'); + await this.rtviClient.connect(); + + const timeTaken = Date.now() - startTime; + this.log(`Connection complete, timeTaken: ${timeTaken}`); + } catch (error) { + this.log(`Error connecting: ${(error as Error).message}`); + this.updateStatus('Error'); + // Clean up if there's an error + if (this.rtviClient) { + try { + await this.rtviClient.disconnect(); + } catch (disconnectError) { + this.log(`Error during disconnect: ${disconnectError}`); + } + } + } + } + + /** + * Disconnect from the bot and clean up media resources + */ + public async disconnect(): Promise { + if (this.rtviClient) { + try { + await this.rtviClient.disconnect(); + this.rtviClient = null; + if (this.botAudio.srcObject && "getAudioTracks" in this.botAudio.srcObject) { + this.botAudio.srcObject.getAudioTracks().forEach((track) => track.stop()); + this.botAudio.srcObject = null; + } + } catch (error) { + this.log(`Error disconnecting: ${(error as Error).message}`); + } + } + } + +} + +declare global { + interface Window { + WebsocketClientApp: typeof WebsocketClientApp; + } +} + +window.addEventListener('DOMContentLoaded', () => { + window.WebsocketClientApp = WebsocketClientApp; + new WebsocketClientApp(); +}); diff --git a/examples/websocket/client/src/style.css b/examples/websocket/client/src/style.css new file mode 100644 index 000000000..9c147266e --- /dev/null +++ b/examples/websocket/client/src/style.css @@ -0,0 +1,98 @@ +body { + margin: 0; + padding: 20px; + font-family: Arial, sans-serif; + background-color: #f0f0f0; +} + +.container { + max-width: 1200px; + margin: 0 auto; +} + +.status-bar { + display: flex; + justify-content: space-between; + align-items: center; + padding: 10px; + background-color: #fff; + border-radius: 8px; + margin-bottom: 20px; +} + +.controls button { + padding: 8px 16px; + margin-left: 10px; + border: none; + border-radius: 4px; + cursor: pointer; +} + +#connect-btn { + background-color: #4caf50; + color: white; +} + +#disconnect-btn { + background-color: #f44336; + color: white; +} + +button:disabled { + opacity: 0.5; + cursor: not-allowed; +} + +.main-content { + background-color: #fff; + border-radius: 8px; + padding: 20px; + margin-bottom: 20px; +} + +.bot-container { + display: flex; + flex-direction: column; + align-items: center; +} + +#bot-video-container { + width: 640px; + height: 360px; + background-color: #e0e0e0; + border-radius: 8px; + margin: 20px auto; + overflow: hidden; + display: flex; + align-items: center; + justify-content: center; +} + +#bot-video-container video { + width: 100%; + height: 100%; + object-fit: cover; +} + +.debug-panel { + background-color: #fff; + border-radius: 8px; + padding: 20px; +} + +.debug-panel h3 { + margin: 0 0 10px 0; + font-size: 16px; + font-weight: bold; +} + +#debug-log { + height: 500px; + overflow-y: auto; + background-color: #f8f8f8; + padding: 10px; + border-radius: 4px; + font-family: monospace; + font-size: 12px; + line-height: 1.4; +} diff --git a/examples/websocket/client/tsconfig.json b/examples/websocket/client/tsconfig.json new file mode 100644 index 000000000..c9c555d96 --- /dev/null +++ b/examples/websocket/client/tsconfig.json @@ -0,0 +1,111 @@ +{ + "compilerOptions": { + /* Visit https://aka.ms/tsconfig to read more about this file */ + + /* Projects */ + // "incremental": true, /* Save .tsbuildinfo files to allow for incremental compilation of projects. */ + // "composite": true, /* Enable constraints that allow a TypeScript project to be used with project references. */ + // "tsBuildInfoFile": "./.tsbuildinfo", /* Specify the path to .tsbuildinfo incremental compilation file. */ + // "disableSourceOfProjectReferenceRedirect": true, /* Disable preferring source files instead of declaration files when referencing composite projects. */ + // "disableSolutionSearching": true, /* Opt a project out of multi-project reference checking when editing. */ + // "disableReferencedProjectLoad": true, /* Reduce the number of projects loaded automatically by TypeScript. */ + + /* Language and Environment */ + "target": "es2016", /* Set the JavaScript language version for emitted JavaScript and include compatible library declarations. */ + // "lib": [], /* Specify a set of bundled library declaration files that describe the target runtime environment. */ + // "jsx": "preserve", /* Specify what JSX code is generated. */ + // "experimentalDecorators": true, /* Enable experimental support for legacy experimental decorators. */ + // "emitDecoratorMetadata": true, /* Emit design-type metadata for decorated declarations in source files. */ + // "jsxFactory": "", /* Specify the JSX factory function used when targeting React JSX emit, e.g. 'React.createElement' or 'h'. */ + // "jsxFragmentFactory": "", /* Specify the JSX Fragment reference used for fragments when targeting React JSX emit e.g. 'React.Fragment' or 'Fragment'. */ + // "jsxImportSource": "", /* Specify module specifier used to import the JSX factory functions when using 'jsx: react-jsx*'. */ + // "reactNamespace": "", /* Specify the object invoked for 'createElement'. This only applies when targeting 'react' JSX emit. */ + // "noLib": true, /* Disable including any library files, including the default lib.d.ts. */ + // "useDefineForClassFields": true, /* Emit ECMAScript-standard-compliant class fields. */ + // "moduleDetection": "auto", /* Control what method is used to detect module-format JS files. */ + + /* Modules */ + "module": "commonjs", /* Specify what module code is generated. */ + // "rootDir": "./", /* Specify the root folder within your source files. */ + // "moduleResolution": "node10", /* Specify how TypeScript looks up a file from a given module specifier. */ + // "baseUrl": "./", /* Specify the base directory to resolve non-relative module names. */ + // "paths": {}, /* Specify a set of entries that re-map imports to additional lookup locations. */ + // "rootDirs": [], /* Allow multiple folders to be treated as one when resolving modules. */ + // "typeRoots": [], /* Specify multiple folders that act like './node_modules/@types'. */ + // "types": [], /* Specify type package names to be included without being referenced in a source file. */ + // "allowUmdGlobalAccess": true, /* Allow accessing UMD globals from modules. */ + // "moduleSuffixes": [], /* List of file name suffixes to search when resolving a module. */ + // "allowImportingTsExtensions": true, /* Allow imports to include TypeScript file extensions. Requires '--moduleResolution bundler' and either '--noEmit' or '--emitDeclarationOnly' to be set. */ + // "rewriteRelativeImportExtensions": true, /* Rewrite '.ts', '.tsx', '.mts', and '.cts' file extensions in relative import paths to their JavaScript equivalent in output files. */ + // "resolvePackageJsonExports": true, /* Use the package.json 'exports' field when resolving package imports. */ + // "resolvePackageJsonImports": true, /* Use the package.json 'imports' field when resolving imports. */ + // "customConditions": [], /* Conditions to set in addition to the resolver-specific defaults when resolving imports. */ + // "noUncheckedSideEffectImports": true, /* Check side effect imports. */ + // "resolveJsonModule": true, /* Enable importing .json files. */ + // "allowArbitraryExtensions": true, /* Enable importing files with any extension, provided a declaration file is present. */ + // "noResolve": true, /* Disallow 'import's, 'require's or ''s from expanding the number of files TypeScript should add to a project. */ + + /* JavaScript Support */ + // "allowJs": true, /* Allow JavaScript files to be a part of your program. Use the 'checkJS' option to get errors from these files. */ + // "checkJs": true, /* Enable error reporting in type-checked JavaScript files. */ + // "maxNodeModuleJsDepth": 1, /* Specify the maximum folder depth used for checking JavaScript files from 'node_modules'. Only applicable with 'allowJs'. */ + + /* Emit */ + // "declaration": true, /* Generate .d.ts files from TypeScript and JavaScript files in your project. */ + // "declarationMap": true, /* Create sourcemaps for d.ts files. */ + // "emitDeclarationOnly": true, /* Only output d.ts files and not JavaScript files. */ + // "sourceMap": true, /* Create source map files for emitted JavaScript files. */ + // "inlineSourceMap": true, /* Include sourcemap files inside the emitted JavaScript. */ + // "noEmit": true, /* Disable emitting files from a compilation. */ + // "outFile": "./", /* Specify a file that bundles all outputs into one JavaScript file. If 'declaration' is true, also designates a file that bundles all .d.ts output. */ + // "outDir": "./", /* Specify an output folder for all emitted files. */ + // "removeComments": true, /* Disable emitting comments. */ + // "importHelpers": true, /* Allow importing helper functions from tslib once per project, instead of including them per-file. */ + // "downlevelIteration": true, /* Emit more compliant, but verbose and less performant JavaScript for iteration. */ + // "sourceRoot": "", /* Specify the root path for debuggers to find the reference source code. */ + // "mapRoot": "", /* Specify the location where debugger should locate map files instead of generated locations. */ + // "inlineSources": true, /* Include source code in the sourcemaps inside the emitted JavaScript. */ + // "emitBOM": true, /* Emit a UTF-8 Byte Order Mark (BOM) in the beginning of output files. */ + // "newLine": "crlf", /* Set the newline character for emitting files. */ + // "stripInternal": true, /* Disable emitting declarations that have '@internal' in their JSDoc comments. */ + // "noEmitHelpers": true, /* Disable generating custom helper functions like '__extends' in compiled output. */ + // "noEmitOnError": true, /* Disable emitting files if any type checking errors are reported. */ + // "preserveConstEnums": true, /* Disable erasing 'const enum' declarations in generated code. */ + // "declarationDir": "./", /* Specify the output directory for generated declaration files. */ + + /* Interop Constraints */ + // "isolatedModules": true, /* Ensure that each file can be safely transpiled without relying on other imports. */ + // "verbatimModuleSyntax": true, /* Do not transform or elide any imports or exports not marked as type-only, ensuring they are written in the output file's format based on the 'module' setting. */ + // "isolatedDeclarations": true, /* Require sufficient annotation on exports so other tools can trivially generate declaration files. */ + // "allowSyntheticDefaultImports": true, /* Allow 'import x from y' when a module doesn't have a default export. */ + "esModuleInterop": true, /* Emit additional JavaScript to ease support for importing CommonJS modules. This enables 'allowSyntheticDefaultImports' for type compatibility. */ + // "preserveSymlinks": true, /* Disable resolving symlinks to their realpath. This correlates to the same flag in node. */ + "forceConsistentCasingInFileNames": true, /* Ensure that casing is correct in imports. */ + + /* Type Checking */ + "strict": true, /* Enable all strict type-checking options. */ + // "noImplicitAny": true, /* Enable error reporting for expressions and declarations with an implied 'any' type. */ + // "strictNullChecks": true, /* When type checking, take into account 'null' and 'undefined'. */ + // "strictFunctionTypes": true, /* When assigning functions, check to ensure parameters and the return values are subtype-compatible. */ + // "strictBindCallApply": true, /* Check that the arguments for 'bind', 'call', and 'apply' methods match the original function. */ + // "strictPropertyInitialization": true, /* Check for class properties that are declared but not set in the constructor. */ + // "strictBuiltinIteratorReturn": true, /* Built-in iterators are instantiated with a 'TReturn' type of 'undefined' instead of 'any'. */ + // "noImplicitThis": true, /* Enable error reporting when 'this' is given the type 'any'. */ + // "useUnknownInCatchVariables": true, /* Default catch clause variables as 'unknown' instead of 'any'. */ + // "alwaysStrict": true, /* Ensure 'use strict' is always emitted. */ + // "noUnusedLocals": true, /* Enable error reporting when local variables aren't read. */ + // "noUnusedParameters": true, /* Raise an error when a function parameter isn't read. */ + // "exactOptionalPropertyTypes": true, /* Interpret optional property types as written, rather than adding 'undefined'. */ + // "noImplicitReturns": true, /* Enable error reporting for codepaths that do not explicitly return in a function. */ + // "noFallthroughCasesInSwitch": true, /* Enable error reporting for fallthrough cases in switch statements. */ + // "noUncheckedIndexedAccess": true, /* Add 'undefined' to a type when accessed using an index. */ + // "noImplicitOverride": true, /* Ensure overriding members in derived classes are marked with an override modifier. */ + // "noPropertyAccessFromIndexSignature": true, /* Enforces using indexed accessors for keys declared using an indexed type. */ + // "allowUnusedLabels": true, /* Disable error reporting for unused labels. */ + // "allowUnreachableCode": true, /* Disable error reporting for unreachable code. */ + + /* Completeness */ + // "skipDefaultLibCheck": true, /* Skip type checking .d.ts files that are included with TypeScript. */ + "skipLibCheck": true /* Skip type checking all .d.ts files. */ + } +} diff --git a/examples/websocket/client/vite.config.js b/examples/websocket/client/vite.config.js new file mode 100644 index 000000000..daf85167d --- /dev/null +++ b/examples/websocket/client/vite.config.js @@ -0,0 +1,15 @@ +import { defineConfig } from 'vite'; +import react from '@vitejs/plugin-react-swc'; + +export default defineConfig({ + plugins: [react()], + server: { + proxy: { + // Proxy /api requests to the backend server + '/connect': { + target: 'http://0.0.0.0:7860', // Replace with your backend URL + changeOrigin: true, + }, + }, + }, +}); diff --git a/examples/websocket/server/bot_fast_api.py b/examples/websocket/server/bot_fast_api.py new file mode 100644 index 000000000..421fe8dbf --- /dev/null +++ b/examples/websocket/server/bot_fast_api.py @@ -0,0 +1,111 @@ +# +# Copyright (c) 2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# +import os +import sys + +from dotenv import load_dotenv +from loguru import logger + +from pipecat.audio.vad.silero import SileroVADAnalyzer +from pipecat.pipeline.pipeline import Pipeline +from pipecat.pipeline.runner import PipelineRunner +from pipecat.pipeline.task import PipelineParams, PipelineTask +from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext +from pipecat.processors.frameworks.rtvi import RTVIConfig, RTVIObserver, RTVIProcessor +from pipecat.serializers.protobuf import ProtobufFrameSerializer +from pipecat.services.gemini_multimodal_live import GeminiMultimodalLiveLLMService +from pipecat.transports.network.fastapi_websocket import ( + FastAPIWebsocketParams, + FastAPIWebsocketTransport, +) + +load_dotenv(override=True) + +logger.remove(0) +logger.add(sys.stderr, level="DEBUG") + + +SYSTEM_INSTRUCTION = f""" +"You are Gemini Chatbot, a friendly, helpful robot. + +Your goal is to demonstrate your capabilities in a succinct way. + +Your output will be converted to audio so don't include special characters in your answers. + +Respond to what the user said in a creative and helpful way. Keep your responses brief. One or two sentences at most. +""" + + +async def run_bot(websocket_client): + ws_transport = FastAPIWebsocketTransport( + websocket=websocket_client, + params=FastAPIWebsocketParams( + audio_in_enabled=True, + audio_out_enabled=True, + add_wav_header=False, + vad_analyzer=SileroVADAnalyzer(), + serializer=ProtobufFrameSerializer(), + ), + ) + + llm = GeminiMultimodalLiveLLMService( + api_key=os.getenv("GOOGLE_API_KEY"), + voice_id="Puck", # Aoede, Charon, Fenrir, Kore, Puck + transcribe_model_audio=True, + system_instruction=SYSTEM_INSTRUCTION, + ) + + context = OpenAILLMContext( + [ + { + "role": "user", + "content": "Start by greeting the user warmly and introducing yourself.", + } + ], + ) + context_aggregator = llm.create_context_aggregator(context) + + # RTVI events for Pipecat client UI + rtvi = RTVIProcessor(config=RTVIConfig(config=[])) + + pipeline = Pipeline( + [ + ws_transport.input(), + context_aggregator.user(), + rtvi, + llm, # LLM + ws_transport.output(), + context_aggregator.assistant(), + ] + ) + + task = PipelineTask( + pipeline, + params=PipelineParams( + allow_interruptions=True, + ), + observers=[RTVIObserver(rtvi)], + ) + + @rtvi.event_handler("on_client_ready") + async def on_client_ready(rtvi): + logger.info("Pipecat client ready.") + await rtvi.set_bot_ready() + # Kick off the conversation. + await task.queue_frames([context_aggregator.user().get_context_frame()]) + + @ws_transport.event_handler("on_client_connected") + async def on_client_connected(transport, client): + logger.info("Pipecat Client connected") + + @ws_transport.event_handler("on_client_disconnected") + async def on_client_disconnected(transport, client): + logger.info("Pipecat Client disconnected") + await task.cancel() + + runner = PipelineRunner(handle_sigint=False) + + await runner.run(task) diff --git a/examples/websocket/server/bot_websocket_server.py b/examples/websocket/server/bot_websocket_server.py new file mode 100644 index 000000000..5274c11f6 --- /dev/null +++ b/examples/websocket/server/bot_websocket_server.py @@ -0,0 +1,109 @@ +# +# Copyright (c) 2024–2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +import os + +from loguru import logger + +from pipecat.audio.vad.silero import SileroVADAnalyzer +from pipecat.pipeline.pipeline import Pipeline +from pipecat.pipeline.runner import PipelineRunner +from pipecat.pipeline.task import PipelineParams, PipelineTask +from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext +from pipecat.processors.frameworks.rtvi import RTVIConfig, RTVIObserver, RTVIProcessor +from pipecat.serializers.protobuf import ProtobufFrameSerializer +from pipecat.services.gemini_multimodal_live import GeminiMultimodalLiveLLMService +from pipecat.transports.network.websocket_server import ( + WebsocketServerParams, + WebsocketServerTransport, +) + +SYSTEM_INSTRUCTION = f""" +"You are Gemini Chatbot, a friendly, helpful robot. + +Your goal is to demonstrate your capabilities in a succinct way. + +Your output will be converted to audio so don't include special characters in your answers. + +Respond to what the user said in a creative and helpful way. Keep your responses brief. One or two sentences at most. +""" + + +async def run_bot_websocket_server(): + ws_transport = WebsocketServerTransport( + params=WebsocketServerParams( + serializer=ProtobufFrameSerializer(), + audio_in_enabled=True, + audio_out_enabled=True, + add_wav_header=False, + vad_analyzer=SileroVADAnalyzer(), + session_timeout=60 * 3, # 3 minutes + ) + ) + + llm = GeminiMultimodalLiveLLMService( + api_key=os.getenv("GOOGLE_API_KEY"), + voice_id="Puck", # Aoede, Charon, Fenrir, Kore, Puck + transcribe_model_audio=True, + system_instruction=SYSTEM_INSTRUCTION, + ) + + context = OpenAILLMContext( + [ + { + "role": "user", + "content": "Start by greeting the user warmly and introducing yourself.", + } + ], + ) + context_aggregator = llm.create_context_aggregator(context) + + # RTVI events for Pipecat client UI + rtvi = RTVIProcessor(config=RTVIConfig(config=[])) + + pipeline = Pipeline( + [ + ws_transport.input(), + context_aggregator.user(), + rtvi, + llm, # LLM + ws_transport.output(), + context_aggregator.assistant(), + ] + ) + + task = PipelineTask( + pipeline, + params=PipelineParams( + allow_interruptions=True, + ), + observers=[RTVIObserver(rtvi)], + ) + + @rtvi.event_handler("on_client_ready") + async def on_client_ready(rtvi): + logger.info("Pipecat client ready.") + await rtvi.set_bot_ready() + # Kick off the conversation. + await task.queue_frames([context_aggregator.user().get_context_frame()]) + + @ws_transport.event_handler("on_client_connected") + async def on_client_connected(transport, client): + logger.info("Pipecat Client connected") + + @ws_transport.event_handler("on_client_disconnected") + async def on_client_disconnected(transport, client): + logger.info("Pipecat Client disconnected") + await task.cancel() + + @ws_transport.event_handler("on_session_timeout") + async def on_session_timeout(transport, client): + logger.info(f"Entering in timeout for {client.remote_address}") + await task.cancel() + + runner = PipelineRunner() + + await runner.run(task) diff --git a/examples/websocket/server/env.example b/examples/websocket/server/env.example new file mode 100644 index 000000000..ceb211161 --- /dev/null +++ b/examples/websocket/server/env.example @@ -0,0 +1,2 @@ +GOOGLE_API_KEY= +WEBSOCKET_SERVER= # Options: 'fast_api' or 'websocket_server' \ No newline at end of file diff --git a/examples/websocket/server/requirements.txt b/examples/websocket/server/requirements.txt new file mode 100644 index 000000000..db4eb0e5b --- /dev/null +++ b/examples/websocket/server/requirements.txt @@ -0,0 +1,4 @@ +python-dotenv +fastapi[all] +uvicorn +pipecat-ai[silero,websocket,google] diff --git a/examples/websocket/server/server.py b/examples/websocket/server/server.py new file mode 100644 index 000000000..7a528f480 --- /dev/null +++ b/examples/websocket/server/server.py @@ -0,0 +1,79 @@ +# +# Copyright (c) 2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# +import asyncio +import os +from contextlib import asynccontextmanager +from typing import Any, Dict + +import uvicorn +from dotenv import load_dotenv +from fastapi import FastAPI, Request, WebSocket +from fastapi.middleware.cors import CORSMiddleware + +# Load environment variables +load_dotenv(override=True) + +from bot_fast_api import run_bot +from bot_websocket_server import run_bot_websocket_server + + +@asynccontextmanager +async def lifespan(app: FastAPI): + """Handles FastAPI startup and shutdown.""" + yield # Run app + + +# Initialize FastAPI app with lifespan manager +app = FastAPI(lifespan=lifespan) + +# Configure CORS to allow requests from any origin +app.add_middleware( + CORSMiddleware, + allow_origins=["*"], + allow_credentials=True, + allow_methods=["*"], + allow_headers=["*"], +) + + +@app.websocket("/ws") +async def websocket_endpoint(websocket: WebSocket): + await websocket.accept() + print("WebSocket connection accepted") + try: + await run_bot(websocket) + except Exception as e: + print(f"Exception in run_bot: {e}") + + +@app.post("/connect") +async def bot_connect(request: Request) -> Dict[Any, Any]: + server_mode = os.getenv("WEBSOCKET_SERVER", "fast_api") + if server_mode == "websocket_server": + ws_url = "ws://localhost:8765" + else: + ws_url = "ws://localhost:7860/ws" + return {"ws_url": ws_url} + + +async def main(): + server_mode = os.getenv("WEBSOCKET_SERVER", "fast_api") + tasks = [] + try: + if server_mode == "websocket_server": + tasks.append(run_bot_websocket_server()) + + config = uvicorn.Config(app, host="0.0.0.0", port=7860) + server = uvicorn.Server(config) + tasks.append(server.serve()) + + await asyncio.gather(*tasks) + except asyncio.CancelledError: + print("Tasks cancelled (probably due to shutdown).") + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/pyproject.toml b/pyproject.toml index 8f9ff9f9d..b5c874919 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -31,8 +31,7 @@ dependencies = [ "pyloudnorm~=0.1.1", "resampy~=0.4.3", "soxr~=0.5.0", - "openai~=1.70.0", - "uvloop~=0.21.0; sys_platform!='win32'" + "openai~=1.70.0" ] [project.urls] @@ -48,7 +47,7 @@ azure = [ "azure-cognitiveservices-speech~=1.42.0"] cartesia = [ "cartesia~=2.0.3", "websockets~=13.1" ] cerebras = [] deepseek = [] -daily = [ "daily-python~=0.19.1" ] +daily = [ "daily-python~=0.19.3" ] deepgram = [ "deepgram-sdk~=4.1.0" ] elevenlabs = [ "websockets~=13.1" ] fal = [ "fal-client~=0.5.9" ] @@ -59,7 +58,7 @@ google = [ "google-cloud-speech~=2.32.0", "google-cloud-texttospeech~=2.26.0", " grok = [] groq = [ "groq~=0.23.0" ] gstreamer = [ "pygobject~=3.50.0" ] -krisp = [ "pipecat-ai-krisp~=0.3.0" ] +krisp = [ "pipecat-ai-krisp~=0.4.0" ] koala = [ "pvkoala~=2.0.3" ] langchain = [ "langchain~=0.3.20", "langchain-community~=0.3.20", "langchain-openai~=0.3.9" ] livekit = [ "livekit~=0.22.0", "livekit-api~=0.8.2", "tenacity~=9.0.0" ] diff --git a/scripts/evals/README.md b/scripts/evals/README.md new file mode 100644 index 000000000..ad94e4605 --- /dev/null +++ b/scripts/evals/README.md @@ -0,0 +1,55 @@ +# Pipecat Evals + +This directory contains a set of utilities to help test Pipecat, specifically +its examples. + +## Release Evals + +Before any Pipecat release, we make sure that all (or most) of the examples work +flawlessly. We have 100+ examples, and checking each one manually was very +time-consuming (and painful!), especially because we aim to release often. + +To make this process easier, we designed these "release evals," which do the +following: + +- Start one of the foundational examples (the user bot) +- Start an eval bot + +The user bot (i.e. the example) introduces itself, and the eval bot then asks a +question. The user bot replies, and the eval bot verifies the response. + +For example, the eval bot might ask: + +"What's 2 plus 2?" + +The user bot replies: + +"2 plus 2 is 4." + +The eval bot (powered by an LLM) evaluates the response and emits a result. It +also explains why it thinks the answer is valid or invalid. + +To run the release evals: + +```sh +python run-release-evals.py -a -v +``` + +This runs all the evals and stores logs and audio (`-a`) for each test. + +You can also specify which tests to run. For example, to run all `07` series +tests: + +```sh +python run-release-evals.py -p 07 -a -v +``` + +## Script Evals + +You can also run evals for a single example (not part of the release set): + +```sh +python run-eval.py YOUR_EXAMPLE_SCRIPT -a -v +``` + +Your script needs to follow any of the foundation examples pattern. diff --git a/scripts/release/eval.py b/scripts/evals/eval.py similarity index 92% rename from scripts/release/eval.py rename to scripts/evals/eval.py index 5ad2bb8f3..1ad165a4a 100644 --- a/scripts/release/eval.py +++ b/scripts/evals/eval.py @@ -9,7 +9,6 @@ import asyncio import io import os import re -import sys import time import wave from datetime import datetime @@ -17,6 +16,7 @@ from pathlib import Path from typing import List, Optional import aiofiles +from deepgram import LiveOptions from loguru import logger from utils import ( EvalResult, @@ -45,12 +45,6 @@ from pipecat.transports.services.daily import DailyParams, DailyTransport SCRIPT_DIR = Path(__file__).resolve().parent -FOUNDATIONAL_DIR = SCRIPT_DIR.parent.parent / "examples" / "foundational" - -sys.path.insert(0, os.path.abspath(FOUNDATIONAL_DIR)) - -EVAL_PROMPT = "" - PIPELINE_IDLE_TIMEOUT_SECS = 30 @@ -58,11 +52,13 @@ class EvalRunner: def __init__( self, *, + examples_dir: Path, pattern: str = "", record_audio: bool = False, name: Optional[str] = None, log_level: str = "DEBUG", ): + self._examples_dir = examples_dir self._pattern = f".*{pattern}.*" if pattern else "" self._record_audio = record_audio self._log_level = log_level @@ -79,9 +75,10 @@ class EvalRunner: os.makedirs(self._recordings_dir, exist_ok=True) async def assert_eval(self, params: FunctionCallParams): + result = params.arguments["result"] reasoning = params.arguments["reasoning"] - logger.debug(f"🧠 EVAL REASONING: {reasoning}") - await self._queue.put(params.arguments["result"]) + logger.debug(f"🧠 EVAL REASONING(result: {result}): {reasoning}") + await self._queue.put(result) await params.result_callback(None) await params.llm.push_frame(EndTaskFrame(), FrameDirection.UPSTREAM) @@ -98,12 +95,14 @@ class EvalRunner: print_begin_test(example_file) + script_path = self._examples_dir / example_file + start_time = time.time() try: await asyncio.wait( [ - asyncio.create_task(run_example_pipeline(example_file)), + asyncio.create_task(run_example_pipeline(script_path)), asyncio.create_task(run_eval_pipeline(self, example_file, prompt, eval)), ], timeout=90, @@ -160,11 +159,9 @@ class EvalRunner: return os.path.join(self._recordings_dir, f"{base_name}.wav") -async def run_example_pipeline(example_file: str): +async def run_example_pipeline(script_path: Path): room_url = os.getenv("DAILY_SAMPLE_ROOM_URL") - script_path = FOUNDATIONAL_DIR / example_file - module = load_module_from_path(script_path) transport = DailyTransport( @@ -199,7 +196,12 @@ async def run_eval_pipeline( ), ) - stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY")) + # We disable smart formatting because some times if the user says "3 + 2 is + # 5" (in audio) this can be converted to "32 is 5". + stt = DeepgramSTTService( + api_key=os.getenv("DEEPGRAM_API_KEY"), + live_options=LiveOptions(smart_format=False), + ) tts = CartesiaTTSService( api_key=os.getenv("CARTESIA_API_KEY"), diff --git a/scripts/evals/run-eval.py b/scripts/evals/run-eval.py new file mode 100644 index 000000000..4ea4aa56b --- /dev/null +++ b/scripts/evals/run-eval.py @@ -0,0 +1,50 @@ +# +# Copyright (c) 2024-2025 Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +import argparse +import asyncio +import os +import sys +from pathlib import Path + +from dotenv import load_dotenv +from eval import EvalRunner +from loguru import logger +from utils import check_env_variables + +load_dotenv(override=True) + + +async def main(args: argparse.Namespace): + if not check_env_variables(): + return + + # Log level + logger.remove(0) + log_level = "TRACE" if args.verbose >= 2 else "DEBUG" + if args.verbose: + logger.add(sys.stderr, level=log_level) + + script_path = Path(os.path.dirname(os.path.abspath(args.script))) + script_file = os.path.basename(args.script) + + runner = EvalRunner(examples_dir=script_path, record_audio=args.audio, log_level=log_level) + + await runner.run_eval(script_file, args.prompt, args.eval) + + runner.print_results() + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="Pipecat Eval Runner") + parser.add_argument("--audio", "-a", action="store_true", help="Record audio for each test") + parser.add_argument("--prompt", "-p", required=True, help="Prompt for this eval") + parser.add_argument("--eval", "-e", required=False, help="Eval verification") + parser.add_argument("--verbose", "-v", action="count", default=0) + parser.add_argument("script", help="Script to run") + args = parser.parse_args() + + asyncio.run(main(args)) diff --git a/scripts/release/run-release-evals.py b/scripts/evals/run-release-evals.py similarity index 97% rename from scripts/release/run-release-evals.py rename to scripts/evals/run-release-evals.py index 33511fdeb..242dd6fc5 100644 --- a/scripts/release/run-release-evals.py +++ b/scripts/evals/run-release-evals.py @@ -8,6 +8,7 @@ import argparse import asyncio import sys from datetime import datetime, timezone +from pathlib import Path from dotenv import load_dotenv from eval import EvalRunner @@ -16,6 +17,11 @@ from utils import check_env_variables load_dotenv(override=True) +SCRIPT_DIR = Path(__file__).resolve().parent + +FOUNDATIONAL_DIR = SCRIPT_DIR.parent.parent / "examples" / "foundational" + + # Math PROMPT_SIMPLE_MATH = "A simple math addition." @@ -124,6 +130,7 @@ async def main(args: argparse.Namespace): logger.add(sys.stderr, level=log_level) runner = EvalRunner( + examples_dir=FOUNDATIONAL_DIR, name=args.name, pattern=args.pattern, record_audio=args.audio, diff --git a/scripts/release/utils.py b/scripts/evals/utils.py similarity index 100% rename from scripts/release/utils.py rename to scripts/evals/utils.py diff --git a/src/pipecat/__init__.py b/src/pipecat/__init__.py index fe1bc421e..de89130cc 100644 --- a/src/pipecat/__init__.py +++ b/src/pipecat/__init__.py @@ -4,7 +4,6 @@ # SPDX-License-Identifier: BSD 2-Clause License # -import asyncio import sys from importlib.metadata import version @@ -12,18 +11,4 @@ from loguru import logger __version__ = version("pipecat-ai") -event_loop = "asyncio" - -if sys.platform in ("linux", "darwin"): - try: - import asyncio - - import uvloop - - event_loop = f"uvloop {uvloop.__version__}" - asyncio.set_event_loop_policy(uvloop.EventLoopPolicy()) - except ImportError: - logger.debug(f"Couldn't find `uvloop`") - pass - -logger.info(f"ᓚᘏᗢ Pipecat {__version__} ({event_loop}; Python {sys.version}) ᓚᘏᗢ") +logger.info(f"ᓚᘏᗢ Pipecat {__version__} (Python {sys.version}) ᓚᘏᗢ") diff --git a/src/pipecat/audio/interruptions/__init__.py b/src/pipecat/audio/interruptions/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/src/pipecat/audio/interruptions/base_interruption_strategy.py b/src/pipecat/audio/interruptions/base_interruption_strategy.py new file mode 100644 index 000000000..7811e8418 --- /dev/null +++ b/src/pipecat/audio/interruptions/base_interruption_strategy.py @@ -0,0 +1,38 @@ +# +# Copyright (c) 2024–2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +from abc import ABC, abstractmethod + + +class BaseInterruptionStrategy(ABC): + """This is a base class for interruption strategies. Interruption strategies + decide when the user can interrupt the bot while the bot is speaking. For + example, there could be strategies based on audio volume or strategies based + on the number of words the user spoke. + + """ + + async def append_audio(self, audio: bytes, sample_rate: int): + """Appends audio to the strategy. Not all strategies handle audio.""" + pass + + async def append_text(self, text: str): + """Appends text to the strategy. Not all strategies handle text.""" + pass + + @abstractmethod + async def should_interrupt(self) -> bool: + """This is called when the user stops speaking and it's time to decide + whether the user should interrupt the bot. The decision will be based on + the aggregated audio and/or text. + + """ + pass + + @abstractmethod + async def reset(self): + """Reset the current accumulated text and/or audio.""" + pass diff --git a/src/pipecat/audio/interruptions/min_words_interruption_strategy.py b/src/pipecat/audio/interruptions/min_words_interruption_strategy.py new file mode 100644 index 000000000..f9f7595ab --- /dev/null +++ b/src/pipecat/audio/interruptions/min_words_interruption_strategy.py @@ -0,0 +1,40 @@ +# +# Copyright (c) 2024–2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +from loguru import logger + +from pipecat.audio.interruptions.base_interruption_strategy import BaseInterruptionStrategy + + +class MinWordsInterruptionStrategy(BaseInterruptionStrategy): + """This is an interruption strategy based on a minimum number of words said + by the user. That is, the strategy will be true if the user has said at + least that amount of words. + + """ + + def __init__(self, *, min_words: int): + super().__init__() + self._min_words = min_words + self._text = "" + + async def append_text(self, text: str): + """Appends text for later analysis. Not all strategies need to handle + text. + + """ + self._text += text + + async def should_interrupt(self) -> bool: + word_count = len(self._text.split()) + interrupt = word_count >= self._min_words + logger.debug( + f"should_interrupt={interrupt} num_spoken_words={word_count} min_words={self._min_words}" + ) + return interrupt + + async def reset(self): + self._text = "" diff --git a/src/pipecat/examples/run.py b/src/pipecat/examples/run.py index d999a5b09..24f673e06 100644 --- a/src/pipecat/examples/run.py +++ b/src/pipecat/examples/run.py @@ -64,7 +64,7 @@ async def maybe_capture_participant_screen( def run_example_daily( run_example: Callable, args: argparse.Namespace, - params: DailyParams, + transport_params: Mapping[str, Callable] = {}, ): logger.info("Running example with DailyTransport...") @@ -75,6 +75,7 @@ def run_example_daily( (room_url, token) = await configure(session) # Run example function with DailyTransport transport arguments. + params: DailyParams = transport_params[args.transport]() transport = DailyTransport(room_url, token, "Pipecat", params=params) await run_example(transport, args, True) @@ -84,7 +85,7 @@ def run_example_daily( def run_example_webrtc( run_example: Callable, args: argparse.Namespace, - params: TransportParams, + transport_params: Mapping[str, Callable] = {}, ): logger.info("Running example with SmallWebRTCTransport...") @@ -130,6 +131,7 @@ def run_example_webrtc( pcs_map.pop(webrtc_connection.pc_id, None) # Run example function with SmallWebRTC transport arguments. + params: TransportParams = transport_params[args.transport]() transport = SmallWebRTCTransport(params=params, webrtc_connection=pipecat_connection) background_tasks.add_task(run_example, transport, args, False) @@ -142,7 +144,7 @@ def run_example_webrtc( @asynccontextmanager async def lifespan(app: FastAPI): yield # Run app - coros = [pc.close() for pc in pcs_map.values()] + coros = [pc.disconnect() for pc in pcs_map.values()] await asyncio.gather(*coros) pcs_map.clear() @@ -152,7 +154,7 @@ def run_example_webrtc( def run_example_twilio( run_example: Callable, args: argparse.Namespace, - params: FastAPIWebsocketParams, + transport_params: Mapping[str, Callable] = {}, ): logger.info("Running example with FastAPIWebsocketTransport (Twilio)...") @@ -195,6 +197,7 @@ def run_example_twilio( call_sid = call_data["start"]["callSid"] # Create websocket transport and update params. + params: FastAPIWebsocketParams = transport_params[args.transport]() params.add_wav_header = False params.serializer = TwilioFrameSerializer( stream_sid=stream_sid, @@ -217,14 +220,13 @@ def run_main( logger.error(f"Transport '{args.transport}' not supported by this example") return - params = transport_params[args.transport]() match args.transport: case "daily": - run_example_daily(run_example, args, params) + run_example_daily(run_example, args, transport_params) case "webrtc": - run_example_webrtc(run_example, args, params) + run_example_webrtc(run_example, args, transport_params) case "twilio": - run_example_twilio(run_example, args, params) + run_example_twilio(run_example, args, transport_params) def main( diff --git a/src/pipecat/frames/frames.py b/src/pipecat/frames/frames.py index b24ff7b19..ec368789d 100644 --- a/src/pipecat/frames/frames.py +++ b/src/pipecat/frames/frames.py @@ -15,9 +15,11 @@ from typing import ( Literal, Mapping, Optional, + Sequence, Tuple, ) +from pipecat.audio.interruptions.base_interruption_strategy import BaseInterruptionStrategy from pipecat.audio.vad.vad_analyzer import VADParams from pipecat.metrics.metrics import MetricsData from pipecat.transcriptions.language import Language @@ -448,6 +450,7 @@ class StartFrame(SystemFrame): enable_metrics: bool = False enable_usage_metrics: bool = False report_only_initial_ttfb: bool = False + interruption_strategies: List[BaseInterruptionStrategy] = field(default_factory=list) @dataclass @@ -524,6 +527,29 @@ class StopTaskFrame(SystemFrame): pass +@dataclass +class FrameProcessorPauseUrgentFrame(SystemFrame): + """This processor is used to pause frame processing for the given processor + as fast as possible. Pausing frame processing will keep frames in the + internal queue which will then be processed when frame processing is resumed + with `FrameProcessorResumeFrame`. + + """ + + processor: str + + +@dataclass +class FrameProcessorResumeUrgentFrame(SystemFrame): + """This processor is used to resume frame processing for the given processor + if it was previously paused as fast as possible. After resuming frame + processing all queued frames will be processed in the order received. + + """ + + processor: str + + @dataclass class StartInterruptionFrame(SystemFrame): """Emitted by VAD to indicate that a user has started speaking (i.e. is @@ -643,6 +669,32 @@ class MetricsFrame(SystemFrame): data: List[MetricsData] +@dataclass +class FunctionCallFromLLM: + """Represents a function call returned by the LLM to be registered for execution. + + Attributes: + function_name (str): The name of the function. + tool_call_id (str): A unique identifier for the function call. + arguments (Mapping[str, Any]): The arguments for the function. + context (OpenAILLMContext): The LLM context. + + """ + + function_name: str + tool_call_id: str + arguments: Mapping[str, Any] + context: Any + + +@dataclass +class FunctionCallsStartedFrame(SystemFrame): + """A frame signaling that one or more function call execution is going to + start.""" + + function_calls: Sequence[FunctionCallFromLLM] + + @dataclass class FunctionCallInProgressFrame(SystemFrame): """A frame signaling that a function call is in progress.""" @@ -677,6 +729,7 @@ class FunctionCallResultFrame(SystemFrame): tool_call_id: str arguments: Any result: Any + run_llm: Optional[bool] = None properties: Optional[FunctionCallResultProperties] = None @@ -824,6 +877,27 @@ class StopFrame(ControlFrame): pass +@dataclass +class FrameProcessorPauseFrame(ControlFrame): + """This processor is used to pause frame processing for the given + processor. Pausing frame processing will keep frames in the internal queue + which will then be processed when frame processing is resumed with + `FrameProcessorResumeFrame`.""" + + processor: str + + +@dataclass +class FrameProcessorResumeFrame(ControlFrame): + """This processor is used to resume frame processing for the given processor + if it was previously paused. After resuming frame processing all queued + frames will be processed in the order received. + + """ + + processor: str + + @dataclass class LLMFullResponseStartFrame(ControlFrame): """Used to indicate the beginning of an LLM response. Following by one or diff --git a/src/pipecat/pipeline/runner.py b/src/pipecat/pipeline/runner.py index 7ac07064f..23c43c06e 100644 --- a/src/pipecat/pipeline/runner.py +++ b/src/pipecat/pipeline/runner.py @@ -59,7 +59,7 @@ class PipelineRunner(BaseObject): await asyncio.gather(*[t.stop_when_done() for t in self._tasks.values()]) async def cancel(self): - logger.debug(f"Canceling runner {self}") + logger.debug(f"Cancelling runner {self}") await asyncio.gather(*[t.cancel() for t in self._tasks.values()]) def _setup_sigint(self): @@ -72,7 +72,7 @@ class PipelineRunner(BaseObject): self._sig_task = asyncio.create_task(self._sig_cancel()) async def _sig_cancel(self): - logger.warning(f"Interruption detected. Canceling runner {self}") + logger.warning(f"Interruption detected. Cancelling runner {self}") await self.cancel() def _gc_collect(self): diff --git a/src/pipecat/pipeline/task.py b/src/pipecat/pipeline/task.py index 0fe330655..736f98244 100644 --- a/src/pipecat/pipeline/task.py +++ b/src/pipecat/pipeline/task.py @@ -6,11 +6,12 @@ import asyncio import time -from typing import Any, AsyncIterable, Dict, Iterable, List, Optional, Tuple, Type +from typing import Any, AsyncIterable, Dict, Iterable, List, Optional, Sequence, Tuple, Type from loguru import logger from pydantic import BaseModel, ConfigDict, Field +from pipecat.audio.interruptions.base_interruption_strategy import BaseInterruptionStrategy from pipecat.clocks.base_clock import BaseClock from pipecat.clocks.system_clock import SystemClock from pipecat.frames.frames import ( @@ -54,10 +55,11 @@ class PipelineParams(BaseModel): enable_metrics: Whether to enable metrics collection. enable_usage_metrics: Whether to enable usage metrics. heartbeats_period_secs: Period between heartbeats in seconds. - observers: List of observers for monitoring pipeline execution. + observers: [deprecated] Use `observers` arg in `PipelineTask` class. report_only_initial_ttfb: Whether to report only initial time to first byte. send_initial_empty_metrics: Whether to send initial empty metrics. start_metadata: Additional metadata for pipeline start. + interruption_strategies: Strategies for bot interruption behavior. """ model_config = ConfigDict(arbitrary_types_allowed=True) @@ -73,6 +75,7 @@ class PipelineParams(BaseModel): report_only_initial_ttfb: bool = False send_initial_empty_metrics: bool = True start_metadata: Dict[str, Any] = Field(default_factory=dict) + interruption_strategies: List[BaseInterruptionStrategy] = Field(default_factory=list) class PipelineTaskSource(FrameProcessor): @@ -181,7 +184,9 @@ class PipelineTask(BaseTask): the idle timeout is reached. enable_turn_tracking: Whether to enable turn tracking. enable_turn_tracing: Whether to enable turn tracing. - + conversation_id: Optional custom ID for the conversation. + additional_span_attributes: Optional dictionary of attributes to propagate as + OpenTelemetry conversation span attributes. """ def __init__( @@ -202,6 +207,7 @@ class PipelineTask(BaseTask): enable_turn_tracking: bool = True, enable_tracing: bool = False, conversation_id: Optional[str] = None, + additional_span_attributes: Optional[dict] = None, ): super().__init__() self._pipeline = pipeline @@ -214,6 +220,7 @@ class PipelineTask(BaseTask): self._enable_turn_tracking = enable_turn_tracking self._enable_tracing = enable_tracing and is_tracing_available() self._conversation_id = conversation_id + self._additional_span_attributes = additional_span_attributes or {} if self._params.observers: import warnings @@ -232,7 +239,9 @@ class PipelineTask(BaseTask): observers.append(self._turn_tracking_observer) if self._enable_tracing and self._turn_tracking_observer: self._turn_trace_observer = TurnTraceObserver( - self._turn_tracking_observer, conversation_id=self._conversation_id + self._turn_tracking_observer, + conversation_id=self._conversation_id, + additional_span_attributes=self._additional_span_attributes, ) observers.append(self._turn_trace_observer) self._finished = False @@ -307,8 +316,8 @@ class PipelineTask(BaseTask): """Return the turn trace observer if enabled.""" return self._turn_trace_observer - async def add_observer(self, observer: BaseObserver): - await self._observer.add_observer(observer) + def add_observer(self, observer: BaseObserver): + self._observer.add_observer(observer) async def remove_observer(self, observer: BaseObserver): await self._observer.remove_observer(observer) @@ -518,6 +527,7 @@ class PipelineTask(BaseTask): enable_metrics=self._params.enable_metrics, enable_usage_metrics=self._params.enable_usage_metrics, report_only_initial_ttfb=self._params.report_only_initial_ttfb, + interruption_strategies=self._params.interruption_strategies, ) start_frame.metadata = self._params.start_metadata await self._source.queue_frame(start_frame, FrameDirection.DOWNSTREAM) diff --git a/src/pipecat/pipeline/task_observer.py b/src/pipecat/pipeline/task_observer.py index b7a54f58d..950ddc43b 100644 --- a/src/pipecat/pipeline/task_observer.py +++ b/src/pipecat/pipeline/task_observer.py @@ -49,21 +49,31 @@ class TaskObserver(BaseObserver): super().__init__(**kwargs) self._observers = observers or [] self._task_manager = task_manager - self._proxies: Dict[BaseObserver, Proxy] = {} + self._proxies: Optional[Dict[BaseObserver, Proxy]] = ( + None # Becomes a dict after start() is called + ) - async def add_observer(self, observer: BaseObserver): - proxy = self._create_proxy(observer) - self._proxies[observer] = proxy + def add_observer(self, observer: BaseObserver): + # Add the observer to the list. self._observers.append(observer) + # If we already started, create a new proxy for the observer. + # Otherwise, it will be created in start(). + if self._started(): + proxy = self._create_proxy(observer) + self._proxies[observer] = proxy + async def remove_observer(self, observer: BaseObserver): + # If the observer has a proxy, remove it. if observer in self._proxies: proxy = self._proxies[observer] # Remove the proxy so it doesn't get called anymore. del self._proxies[observer] # Cancel the proxy task right away. await self._task_manager.cancel_task(proxy.task) - # Remove the observer. + + # Remove the observer from the list. + if observer in self._observers: self._observers.remove(observer) async def start(self): @@ -79,6 +89,9 @@ class TaskObserver(BaseObserver): for proxy in self._proxies.values(): await proxy.queue.put(data) + def _started(self) -> bool: + return self._proxies is not None + def _create_proxy(self, observer: BaseObserver) -> Proxy: queue = asyncio.Queue() task = self._task_manager.create_task( diff --git a/src/pipecat/processors/aggregators/llm_response.py b/src/pipecat/processors/aggregators/llm_response.py index 55ac0e2d5..e9aebb2a0 100644 --- a/src/pipecat/processors/aggregators/llm_response.py +++ b/src/pipecat/processors/aggregators/llm_response.py @@ -11,7 +11,9 @@ from typing import Dict, List, Literal, Optional, Set from loguru import logger +from pipecat.audio.interruptions.base_interruption_strategy import BaseInterruptionStrategy from pipecat.frames.frames import ( + BotInterruptionFrame, BotStartedSpeakingFrame, BotStoppedSpeakingFrame, CancelFrame, @@ -22,6 +24,8 @@ from pipecat.frames.frames import ( FunctionCallCancelFrame, FunctionCallInProgressFrame, FunctionCallResultFrame, + FunctionCallsStartedFrame, + InputAudioRawFrame, InterimTranscriptionFrame, LLMFullResponseEndFrame, LLMFullResponseStartFrame, @@ -159,7 +163,7 @@ class BaseLLMResponseAggregator(FrameProcessor): pass @abstractmethod - def reset(self): + async def reset(self): """Reset the internals of this aggregator. This should not modify the internal messages. """ @@ -193,7 +197,7 @@ class LLMContextResponseAggregator(BaseLLMResponseAggregator): self._context = context self._role = role - self._aggregation = "" + self._aggregation: str = "" @property def messages(self) -> List[dict]: @@ -226,7 +230,7 @@ class LLMContextResponseAggregator(BaseLLMResponseAggregator): def set_tool_choice(self, tool_choice: Literal["none", "auto", "required"] | dict): self._context.set_tool_choice(tool_choice) - def reset(self): + async def reset(self): self._aggregation = "" @@ -269,10 +273,11 @@ class LLMUserContextAggregator(LLMContextResponseAggregator): self._aggregation_event = asyncio.Event() self._aggregation_task = None - def reset(self): - super().reset() + async def reset(self): + await super().reset() self._seen_interim_results = False self._waiting_for_aggregation = False + [await s.reset() for s in self._interruption_strategies] async def handle_aggregation(self, aggregation: str): self._context.add_message({"role": self.role, "content": aggregation}) @@ -293,6 +298,9 @@ class LLMUserContextAggregator(LLMContextResponseAggregator): elif isinstance(frame, CancelFrame): await self._cancel(frame) await self.push_frame(frame, direction) + elif isinstance(frame, InputAudioRawFrame): + await self._handle_input_audio(frame) + await self.push_frame(frame, direction) elif isinstance(frame, UserStartedSpeakingFrame): await self._handle_user_started_speaking(frame) await self.push_frame(frame, direction) @@ -320,18 +328,42 @@ class LLMUserContextAggregator(LLMContextResponseAggregator): else: await self.push_frame(frame, direction) + async def _process_aggregation(self): + """Process the current aggregation and push it downstream.""" + aggregation = self._aggregation + await self.reset() + await self.handle_aggregation(aggregation) + frame = OpenAILLMContextFrame(self._context) + await self.push_frame(frame) + async def push_aggregation(self): + """Pushes the current aggregation based on interruption strategies and conditions.""" if len(self._aggregation) > 0: - aggregation = self._aggregation + if self.interruption_strategies and self._bot_speaking: + should_interrupt = await self._should_interrupt_based_on_strategies() - # Reset the aggregation. Reset it before pushing it down, otherwise - # if the tasks gets cancelled we won't be able to clear things up. - self.reset() + if should_interrupt: + logger.debug( + "Interruption conditions met - pushing BotInterruptionFrame and aggregation" + ) + await self.push_frame(BotInterruptionFrame(), FrameDirection.UPSTREAM) + await self._process_aggregation() + else: + logger.debug("Interruption conditions not met - not pushing aggregation") + # Don't process aggregation, just reset it + await self.reset() + else: + # No interruption config - normal behavior (always push aggregation) + await self._process_aggregation() - await self.handle_aggregation(aggregation) + async def _should_interrupt_based_on_strategies(self) -> bool: + """Check if interruption should occur based on configured strategies.""" - frame = OpenAILLMContextFrame(self._context) - await self.push_frame(frame) + async def should_interrupt(strategy: BaseInterruptionStrategy): + await strategy.append_text(self._aggregation) + return await strategy.should_interrupt() + + return any([await should_interrupt(s) for s in self._interruption_strategies]) async def _start(self, frame: StartFrame): self._create_aggregation_task() @@ -342,6 +374,10 @@ class LLMUserContextAggregator(LLMContextResponseAggregator): async def _cancel(self, frame: CancelFrame): await self._cancel_aggregation_task() + async def _handle_input_audio(self, frame: InputAudioRawFrame): + for s in self.interruption_strategies: + await s.append_audio(frame.audio, frame.sample_rate) + async def _handle_user_started_speaking(self, frame: UserStartedSpeakingFrame): self._user_speaking = True self._waiting_for_aggregation = True @@ -427,7 +463,7 @@ class LLMUserContextAggregator(LLMContextResponseAggregator): # If we reached this case and the bot is speaking, let's ignore # what the user said. logger.debug("Ignoring user speaking emulation, bot is speaking.") - self.reset() + await self.reset() else: # The bot is not speaking so, let's trigger user speaking # emulation. @@ -465,9 +501,18 @@ class LLMAssistantContextAggregator(LLMContextResponseAggregator): self._params.expect_stripped_words = kwargs["expect_stripped_words"] self._started = 0 - self._function_calls_in_progress: Dict[str, FunctionCallInProgressFrame] = {} + self._function_calls_in_progress: Dict[str, Optional[FunctionCallInProgressFrame]] = {} self._context_updated_tasks: Set[asyncio.Task] = set() + @property + def has_function_calls_in_progress(self) -> bool: + """Check if there are any function calls currently in progress. + + Returns: + bool: True if function calls are in progress, False otherwise + """ + return bool(self._function_calls_in_progress) + async def handle_aggregation(self, aggregation: str): self._context.add_message({"role": "assistant", "content": aggregation}) @@ -503,6 +548,8 @@ class LLMAssistantContextAggregator(LLMContextResponseAggregator): self.set_tools(frame.tools) elif isinstance(frame, LLMSetToolChoiceFrame): self.set_tool_choice(frame.tool_choice) + elif isinstance(frame, FunctionCallsStartedFrame): + await self._handle_function_calls_started(frame) elif isinstance(frame, FunctionCallInProgressFrame): await self._handle_function_call_in_progress(frame) elif isinstance(frame, FunctionCallResultFrame): @@ -522,7 +569,7 @@ class LLMAssistantContextAggregator(LLMContextResponseAggregator): return aggregation = self._aggregation.strip() - self.reset() + await self.reset() if aggregation: await self.handle_aggregation(aggregation) @@ -537,7 +584,13 @@ class LLMAssistantContextAggregator(LLMContextResponseAggregator): async def _handle_interruptions(self, frame: StartInterruptionFrame): await self.push_aggregation() self._started = 0 - self.reset() + await self.reset() + + async def _handle_function_calls_started(self, frame: FunctionCallsStartedFrame): + function_names = [f"{f.function_name}:{f.tool_call_id}" for f in frame.function_calls] + logger.debug(f"{self} FunctionCallsStartedFrame: {function_names}") + for function_call in frame.function_calls: + self._function_calls_in_progress[function_call.tool_call_id] = None async def _handle_function_call_in_progress(self, frame: FunctionCallInProgressFrame): logger.debug( @@ -562,19 +615,22 @@ class LLMAssistantContextAggregator(LLMContextResponseAggregator): await self.handle_function_call_result(frame) + run_llm = False + # Run inference if the function call result requires it. if frame.result: - run_llm = False - if properties and properties.run_llm is not None: - # If the tool call result has a run_llm property, use it + # If the tool call result has a run_llm property, use it. run_llm = properties.run_llm + elif frame.run_llm is not None: + # If the frame is indicating we should run the LLM, do it. + run_llm = frame.run_llm else: - # Default behavior is to run the LLM if there are no function calls in progress + # If this is the last function call in progress, run the LLM. run_llm = not bool(self._function_calls_in_progress) - if run_llm: - await self.push_context_frame(FrameDirection.UPSTREAM) + if run_llm: + await self.push_context_frame(FrameDirection.UPSTREAM) # Call the `on_context_updated` callback once the function call result # is added to the context. Also, run this in a separate task to make @@ -653,7 +709,7 @@ class LLMUserResponseAggregator(LLMUserContextAggregator): # Reset the aggregation. Reset it before pushing it down, otherwise # if the tasks gets cancelled we won't be able to clear things up. - self.reset() + await self.reset() frame = LLMMessagesFrame(self._context.messages) await self.push_frame(frame) @@ -675,7 +731,7 @@ class LLMAssistantResponseAggregator(LLMAssistantContextAggregator): # Reset the aggregation. Reset it before pushing it down, otherwise # if the tasks gets cancelled we won't be able to clear things up. - self.reset() + await self.reset() frame = LLMMessagesFrame(self._context.messages) await self.push_frame(frame) diff --git a/src/pipecat/processors/aggregators/openai_llm_context.py b/src/pipecat/processors/aggregators/openai_llm_context.py index 948e3e101..806741c4c 100644 --- a/src/pipecat/processors/aggregators/openai_llm_context.py +++ b/src/pipecat/processors/aggregators/openai_llm_context.py @@ -106,7 +106,7 @@ class OpenAILLMContext: if "mime_type" in msg and msg["mime_type"].startswith("image/"): msg["data"] = "..." msgs.append(msg) - return json.dumps(msgs) + return json.dumps(msgs, ensure_ascii=False) def from_standard_message(self, message): """Convert from OpenAI message format to OpenAI message format (passthrough). diff --git a/src/pipecat/processors/aggregators/user_response.py b/src/pipecat/processors/aggregators/user_response.py index 6998fe200..8831f7d10 100644 --- a/src/pipecat/processors/aggregators/user_response.py +++ b/src/pipecat/processors/aggregators/user_response.py @@ -23,4 +23,4 @@ class UserResponseAggregator(LLMUserResponseAggregator): await self.push_frame(frame) # Reset our accumulator state. - self.reset() + await self.reset() diff --git a/src/pipecat/processors/frame_processor.py b/src/pipecat/processors/frame_processor.py index b444b4c58..1d2f066ed 100644 --- a/src/pipecat/processors/frame_processor.py +++ b/src/pipecat/processors/frame_processor.py @@ -7,15 +7,20 @@ import asyncio from dataclasses import dataclass from enum import Enum -from typing import Awaitable, Callable, Coroutine, Optional +from typing import Awaitable, Callable, Coroutine, List, Optional, Sequence from loguru import logger +from pipecat.audio.interruptions.base_interruption_strategy import BaseInterruptionStrategy from pipecat.clocks.base_clock import BaseClock from pipecat.frames.frames import ( CancelFrame, ErrorFrame, Frame, + FrameProcessorPauseFrame, + FrameProcessorPauseUrgentFrame, + FrameProcessorResumeFrame, + FrameProcessorResumeUrgentFrame, StartFrame, StartInterruptionFrame, StopInterruptionFrame, @@ -67,6 +72,7 @@ class FrameProcessor(BaseObject): self._enable_metrics = False self._enable_usage_metrics = False self._report_only_initial_ttfb = False + self._interruption_strategies: List[BaseInterruptionStrategy] = [] # Indicates whether we have received the StartFrame. self.__started = False @@ -119,6 +125,10 @@ class FrameProcessor(BaseObject): def report_only_initial_ttfb(self): return self._report_only_initial_ttfb + @property + def interruption_strategies(self) -> Sequence[BaseInterruptionStrategy]: + return self._interruption_strategies + def can_generate_metrics(self) -> bool: return False @@ -253,6 +263,10 @@ class FrameProcessor(BaseObject): self._should_report_ttfb = True elif isinstance(frame, CancelFrame): await self.__cancel(frame) + elif isinstance(frame, (FrameProcessorPauseFrame, FrameProcessorPauseUrgentFrame)): + await self.__pause(frame) + elif isinstance(frame, (FrameProcessorResumeFrame, FrameProcessorResumeUrgentFrame)): + await self.__resume(frame) async def push_error(self, error: ErrorFrame): await self.push_frame(error, FrameDirection.UPSTREAM) @@ -272,6 +286,7 @@ class FrameProcessor(BaseObject): self._enable_metrics = frame.enable_metrics self._enable_usage_metrics = frame.enable_usage_metrics self._report_only_initial_ttfb = frame.report_only_initial_ttfb + self._interruption_strategies = frame.interruption_strategies self.__create_input_task() self.__create_push_task() @@ -280,6 +295,14 @@ class FrameProcessor(BaseObject): await self.__cancel_input_task() await self.__cancel_push_task() + async def __pause(self, frame: FrameProcessorPauseFrame | FrameProcessorPauseUrgentFrame): + if frame.name == self.name: + await self.pause_processing_frames() + + async def __resume(self, frame: FrameProcessorResumeFrame | FrameProcessorResumeUrgentFrame): + if frame.name == self.name: + await self.resume_processing_frames() + # # Handle interruptions # diff --git a/src/pipecat/serializers/exotel.py b/src/pipecat/serializers/exotel.py new file mode 100644 index 000000000..4e546e442 --- /dev/null +++ b/src/pipecat/serializers/exotel.py @@ -0,0 +1,161 @@ +# +# Copyright (c) 2024–2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +import base64 +import json +from typing import Optional + +from loguru import logger +from pydantic import BaseModel + +from pipecat.audio.utils import create_default_resampler +from pipecat.frames.frames import ( + AudioRawFrame, + Frame, + InputAudioRawFrame, + InputDTMFFrame, + KeypadEntry, + StartFrame, + StartInterruptionFrame, + TransportMessageFrame, + TransportMessageUrgentFrame, +) +from pipecat.serializers.base_serializer import FrameSerializer, FrameSerializerType + + +class ExotelFrameSerializer(FrameSerializer): + """Serializer for Exotel Media Streams WebSocket protocol. + + This serializer handles converting between Pipecat frames and Exotel's WebSocket + media streams protocol. It supports audio conversion, DTMF events, and automatic + call termination. + + Ref Doc for events - https://support.exotel.com/support/solutions/articles/3000108630-working-with-the-stream-and-voicebot-applet + """ + + class InputParams(BaseModel): + """Configuration parameters for ExotelFrameSerializer. + + Attributes: + exotel_sample_rate: Sample rate used by Exotel, defaults to 8000 Hz. + sample_rate: Optional override for pipeline input sample rate. + """ + + exotel_sample_rate: int = 8000 + sample_rate: Optional[int] = None + + def __init__( + self, stream_sid: str, call_sid: Optional[str] = None, params: Optional[InputParams] = None + ): + """Initialize the ExotelFrameSerializer. + + Args: + stream_sid: The Exotel Media Stream SID. + call_sid: The associated Exotel Call SID (optional, not used in this implementation). + params: Configuration parameters. + """ + self._stream_sid = stream_sid + self._call_sid = call_sid + self._params = params or ExotelFrameSerializer.InputParams() + + self._exotel_sample_rate = self._params.exotel_sample_rate + self._sample_rate = 0 # Pipeline input rate + + self._resampler = create_default_resampler() + + @property + def type(self) -> FrameSerializerType: + """Gets the serializer type. + + Returns: + The serializer type, either TEXT or BINARY. + """ + return FrameSerializerType.TEXT + + async def setup(self, frame: StartFrame): + """Sets up the serializer with pipeline configuration. + + Args: + frame: The StartFrame containing pipeline configuration. + """ + self._sample_rate = self._params.sample_rate or frame.audio_in_sample_rate + + async def serialize(self, frame: Frame) -> str | bytes | None: + """Serializes a Pipecat frame to Exotel WebSocket format. + + Handles conversion of various frame types to Exotel WebSocket messages. + + Args: + frame: The Pipecat frame to serialize. + + Returns: + Serialized data as string or bytes, or None if the frame isn't handled. + """ + if isinstance(frame, StartInterruptionFrame): + answer = {"event": "clear", "streamSid": self._stream_sid} + return json.dumps(answer) + elif isinstance(frame, AudioRawFrame): + data = frame.audio + + # Output: Exotel outputs PCM audio, but we need to resample to match requested sample_rate + serialized_data = await self._resampler.resample( + data, frame.sample_rate, self._exotel_sample_rate + ) + payload = base64.b64encode(serialized_data).decode("ascii") + + answer = { + "event": "media", + "streamSid": self._stream_sid, + "media": {"payload": payload}, + } + + return json.dumps(answer) + elif isinstance(frame, (TransportMessageFrame, TransportMessageUrgentFrame)): + return json.dumps(frame.message) + + return None + + async def deserialize(self, data: str | bytes) -> Frame | None: + """Deserializes Exotel WebSocket data to Pipecat frames. + + Handles conversion of Exotel media events to appropriate Pipecat frames. + + Args: + data: The raw WebSocket data from Exotel. + + Returns: + A Pipecat frame corresponding to the Exotel event, or None if unhandled. + """ + message = json.loads(data) + + if message["event"] == "media": + payload_base64 = message["media"]["payload"] + payload = base64.b64decode(payload_base64) + + deserialized_data = await self._resampler.resample( + payload, + self._exotel_sample_rate, + self._sample_rate, + ) + + # Input: Exotel takes PCM data, so just resample to match sample_rate + audio_frame = InputAudioRawFrame( + audio=deserialized_data, + num_channels=1, # Assuming mono audio from Exotel + sample_rate=self._sample_rate, # Use the configured pipeline input rate + ) + return audio_frame + elif message["event"] == "dtmf": + digit = message.get("dtmf", {}).get("digit") + + try: + return InputDTMFFrame(KeypadEntry(digit)) + except ValueError: + # Handle case where string doesn't match any enum value + logger.info(f"Invalid DTMF digit: {digit}") + return None + + return None diff --git a/src/pipecat/serializers/protobuf.py b/src/pipecat/serializers/protobuf.py index c3b6d86af..c91c6661e 100644 --- a/src/pipecat/serializers/protobuf.py +++ b/src/pipecat/serializers/protobuf.py @@ -41,6 +41,7 @@ class ProtobufFrameSerializer(FrameSerializer): TextFrame: "text", InputAudioRawFrame: "audio", TranscriptionFrame: "transcription", + MessageFrame: "message", } DESERIALIZABLE_FIELDS = {v: k for k, v in DESERIALIZABLE_TYPES.items()} @@ -97,8 +98,18 @@ class ProtobufFrameSerializer(FrameSerializer): if "pts" in args_dict: del args_dict["pts"] - # Create the instance - instance = class_name(**args_dict) + # Special handling for MessageFrame -> TransportMessageUrgentFrame + if class_name == MessageFrame: + try: + msg = json.loads(args_dict["data"]) + instance = TransportMessageUrgentFrame(message=msg) + logger.debug(f"ProtobufFrameSerializer: Transport message {instance}") + except Exception as e: + logger.error(f"Error parsing MessageFrame data: {e}") + return None + else: + # Normal deserialization, create the instance + instance = class_name(**args_dict) # Set special fields if id: diff --git a/src/pipecat/services/anthropic/llm.py b/src/pipecat/services/anthropic/llm.py index 2c4e9078d..236f269fa 100644 --- a/src/pipecat/services/anthropic/llm.py +++ b/src/pipecat/services/anthropic/llm.py @@ -45,7 +45,7 @@ from pipecat.processors.aggregators.openai_llm_context import ( OpenAILLMContextFrame, ) from pipecat.processors.frame_processor import FrameDirection -from pipecat.services.llm_service import LLMService +from pipecat.services.llm_service import FunctionCallFromLLM, LLMService from pipecat.utils.tracing.service_decorators import traced_llm try: @@ -202,9 +202,8 @@ class AnthropicLLMService(LLMService): tool_use_block = None json_accumulator = "" + function_calls = [] async for event in response: - # logger.debug(f"Anthropic LLM event: {event}") - # Aggregate streaming content, create frames, trigger events if event.type == "content_block_delta": @@ -226,11 +225,14 @@ class AnthropicLLMService(LLMService): and event.delta.stop_reason == "tool_use" ): if tool_use_block: - await self.call_function( - context=context, - tool_call_id=tool_use_block.id, - function_name=tool_use_block.name, - arguments=json.loads(json_accumulator) if json_accumulator else dict(), + args = json.loads(json_accumulator) if json_accumulator else {} + function_calls.append( + FunctionCallFromLLM( + context=context, + tool_call_id=tool_use_block.id, + function_name=tool_use_block.name, + arguments=args, + ) ) # Calculate usage. Do this here in its own if statement, because there may be usage @@ -277,6 +279,8 @@ class AnthropicLLMService(LLMService): if total_input_tokens >= 1024: context.turns_above_cache_threshold += 1 + await self.run_function_calls(function_calls) + except asyncio.CancelledError: # If we're interrupted, we won't get a complete usage report. So set our flag to use the # token estimate. The reraise the exception so all the processors running in this task diff --git a/src/pipecat/services/assemblyai/stt.py b/src/pipecat/services/assemblyai/stt.py index c7e1d9e48..14d9fb397 100644 --- a/src/pipecat/services/assemblyai/stt.py +++ b/src/pipecat/services/assemblyai/stt.py @@ -98,7 +98,7 @@ class AssemblyAISTTService(STTService): self._audio_buffer = self._audio_buffer[self._chunk_size_bytes :] await self._websocket.send(chunk) - yield Frame() + yield None async def process_frame(self, frame: Frame, direction: FrameDirection): await super().process_frame(frame, direction) diff --git a/src/pipecat/services/aws/llm.py b/src/pipecat/services/aws/llm.py index a90620b20..dcec91463 100644 --- a/src/pipecat/services/aws/llm.py +++ b/src/pipecat/services/aws/llm.py @@ -21,6 +21,7 @@ from pipecat.adapters.services.bedrock_adapter import AWSBedrockLLMAdapter from pipecat.frames.frames import ( Frame, FunctionCallCancelFrame, + FunctionCallFromLLM, FunctionCallInProgressFrame, FunctionCallResultFrame, LLMFullResponseEndFrame, @@ -708,6 +709,7 @@ class AWSBedrockLLMService(LLMService): tool_use_block = None json_accumulator = "" + function_calls = [] for event in response["stream"]: # Handle text content if "contentBlockDelta" in event: @@ -740,11 +742,13 @@ class AWSBedrockLLMService(LLMService): # Only call function if it's not the no_operation tool if not using_noop_tool: - await self.call_function( - context=context, - tool_call_id=tool_use_block["id"], - function_name=tool_use_block["name"], - arguments=arguments, + function_calls.append( + FunctionCallFromLLM( + context=context, + tool_call_id=tool_use_block["id"], + function_name=tool_use_block["name"], + arguments=arguments, + ) ) else: logger.debug("Ignoring no_operation tool call") @@ -758,7 +762,7 @@ class AWSBedrockLLMService(LLMService): completion_tokens += usage.get("outputTokens", 0) cache_read_input_tokens += usage.get("cacheReadInputTokens", 0) cache_creation_input_tokens += usage.get("cacheWriteInputTokens", 0) - + await self.run_function_calls(function_calls) except asyncio.CancelledError: # If we're interrupted, we won't get a complete usage report. So set our flag to use the # token estimate. The reraise the exception so all the processors running in this task diff --git a/src/pipecat/services/aws/stt.py b/src/pipecat/services/aws/stt.py index 4490f3795..d6aa1fa5d 100644 --- a/src/pipecat/services/aws/stt.py +++ b/src/pipecat/services/aws/stt.py @@ -266,6 +266,7 @@ class AWSTranscribeSTTService(STTService): Language.JA: "ja-JP", Language.KO: "ko-KR", Language.ZH: "zh-CN", + Language.PL: "pl-PL", } return language_map.get(language) diff --git a/src/pipecat/services/aws/tts.py b/src/pipecat/services/aws/tts.py index a0719f227..0159096d1 100644 --- a/src/pipecat/services/aws/tts.py +++ b/src/pipecat/services/aws/tts.py @@ -253,7 +253,8 @@ class AWSPollyTTSService(TTSService): yield TTSStartedFrame() - CHUNK_SIZE = 1024 + CHUNK_SIZE = self.chunk_size + for i in range(0, len(audio_data), CHUNK_SIZE): chunk = audio_data[i : i + CHUNK_SIZE] if len(chunk) > 0: diff --git a/src/pipecat/services/cartesia/__init__.py b/src/pipecat/services/cartesia/__init__.py index 56c789743..efa771163 100644 --- a/src/pipecat/services/cartesia/__init__.py +++ b/src/pipecat/services/cartesia/__init__.py @@ -8,6 +8,7 @@ import sys from pipecat.services import DeprecatedModuleProxy +from .stt import * from .tts import * -sys.modules[__name__] = DeprecatedModuleProxy(globals(), "cartesia", "cartesia.tts") +sys.modules[__name__] = DeprecatedModuleProxy(globals(), "cartesia", "cartesia.[stt,tts]") diff --git a/src/pipecat/services/cartesia/stt.py b/src/pipecat/services/cartesia/stt.py new file mode 100644 index 000000000..104e4b2c5 --- /dev/null +++ b/src/pipecat/services/cartesia/stt.py @@ -0,0 +1,239 @@ +# +# Copyright (c) 2024–2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +import asyncio +import json +import urllib.parse +from typing import AsyncGenerator, Optional + +import websockets +from loguru import logger + +from pipecat.frames.frames import ( + CancelFrame, + EndFrame, + Frame, + InterimTranscriptionFrame, + StartFrame, + TranscriptionFrame, + UserStartedSpeakingFrame, + UserStoppedSpeakingFrame, +) +from pipecat.processors.frame_processor import FrameDirection +from pipecat.services.stt_service import STTService +from pipecat.transcriptions.language import Language +from pipecat.utils.time import time_now_iso8601 +from pipecat.utils.tracing.service_decorators import traced_stt + + +class CartesiaLiveOptions: + def __init__( + self, + *, + model: str = "ink-whisper", + language: str = Language.EN.value, + encoding: str = "pcm_s16le", + sample_rate: int = 16000, + **kwargs, + ): + self.model = model + self.language = language + self.encoding = encoding + self.sample_rate = sample_rate + self.additional_params = kwargs + + def to_dict(self): + params = { + "model": self.model, + "language": self.language if isinstance(self.language, str) else self.language.value, + "encoding": self.encoding, + "sample_rate": str(self.sample_rate), + } + + return params + + def items(self): + return self.to_dict().items() + + def get(self, key, default=None): + if hasattr(self, key): + return getattr(self, key) + return self.additional_params.get(key, default) + + @classmethod + def from_json(cls, json_str: str) -> "CartesiaLiveOptions": + return cls(**json.loads(json_str)) + + +class CartesiaSTTService(STTService): + def __init__( + self, + *, + api_key: str, + base_url: str = "", + sample_rate: int = 16000, + live_options: Optional[CartesiaLiveOptions] = None, + **kwargs, + ): + sample_rate = sample_rate or (live_options.sample_rate if live_options else None) + super().__init__(sample_rate=sample_rate, **kwargs) + + default_options = CartesiaLiveOptions( + model="ink-whisper", + language=Language.EN.value, + encoding="pcm_s16le", + sample_rate=sample_rate, + ) + + merged_options = default_options + if live_options: + merged_options_dict = default_options.to_dict() + merged_options_dict.update(live_options.to_dict()) + merged_options = CartesiaLiveOptions( + **{ + k: v + for k, v in merged_options_dict.items() + if not isinstance(v, str) or v != "None" + } + ) + + self._settings = merged_options + self.set_model_name(merged_options.model) + self._api_key = api_key + self._base_url = base_url or "api.cartesia.ai" + self._connection = None + self._receiver_task = None + + def can_generate_metrics(self) -> bool: + return True + + async def start(self, frame: StartFrame): + await super().start(frame) + await self._connect() + + async def stop(self, frame: EndFrame): + await super().stop(frame) + await self._disconnect() + + async def cancel(self, frame: CancelFrame): + await super().cancel(frame) + await self._disconnect() + + async def run_stt(self, audio: bytes) -> AsyncGenerator[Frame, None]: + # If the connection is closed, due to timeout, we need to reconnect when the user starts speaking again + if not self._connection or self._connection.closed: + await self._connect() + + await self._connection.send(audio) + yield None + + async def _connect(self): + params = self._settings.to_dict() + ws_url = f"wss://{self._base_url}/stt/websocket?{urllib.parse.urlencode(params)}" + logger.debug(f"Connecting to Cartesia: {ws_url}") + headers = {"Cartesia-Version": "2025-04-16", "X-API-Key": self._api_key} + + try: + self._connection = await websockets.connect(ws_url, extra_headers=headers) + # Setup the receiver task to handle the incoming messages from the Cartesia server + if self._receiver_task is None or self._receiver_task.done(): + self._receiver_task = asyncio.create_task(self._receive_messages()) + logger.debug(f"Connected to Cartesia") + except Exception as e: + logger.error(f"{self}: unable to connect to Cartesia: {e}") + + async def _receive_messages(self): + try: + while True: + if not self._connection or self._connection.closed: + break + + message = await self._connection.recv() + try: + data = json.loads(message) + await self._process_response(data) + except json.JSONDecodeError: + logger.warning(f"Received non-JSON message: {message}") + except asyncio.CancelledError: + pass + except websockets.exceptions.ConnectionClosed as e: + logger.debug(f"WebSocket connection closed: {e}") + except Exception as e: + logger.error(f"Error in message receiver: {e}") + + async def _process_response(self, data): + if "type" in data: + if data["type"] == "transcript": + await self._on_transcript(data) + + elif data["type"] == "error": + logger.error(f"Cartesia error: {data.get('message', 'Unknown error')}") + + @traced_stt + async def _handle_transcription( + self, transcript: str, is_final: bool, language: Optional[Language] = None + ): + """Handle a transcription result with tracing.""" + pass + + async def _on_transcript(self, data): + if "text" not in data: + return + + transcript = data.get("text", "") + is_final = data.get("is_final", False) + language = None + + if "language" in data: + try: + language = Language(data["language"]) + except (ValueError, KeyError): + pass + + if len(transcript) > 0: + await self.stop_ttfb_metrics() + if is_final: + await self.push_frame( + TranscriptionFrame(transcript, "", time_now_iso8601(), language) + ) + await self._handle_transcription(transcript, is_final, language) + await self.stop_processing_metrics() + else: + # For interim transcriptions, just push the frame without tracing + await self.push_frame( + InterimTranscriptionFrame(transcript, "", time_now_iso8601(), language) + ) + + async def _disconnect(self): + if self._receiver_task: + self._receiver_task.cancel() + try: + await self._receiver_task + except asyncio.CancelledError: + pass + except Exception as e: + logger.exception(f"Unexpected exception while cancelling task: {e}") + self._receiver_task = None + + if self._connection and self._connection.open: + logger.debug("Disconnecting from Cartesia") + + await self._connection.close() + self._connection = None + + async def start_metrics(self): + await self.start_ttfb_metrics() + await self.start_processing_metrics() + + async def process_frame(self, frame: Frame, direction: FrameDirection): + await super().process_frame(frame, direction) + + if isinstance(frame, UserStartedSpeakingFrame): + await self.start_metrics() + elif isinstance(frame, UserStoppedSpeakingFrame): + # Send finalize command to flush the transcription session + if self._connection and self._connection.open: + await self._connection.send("finalize") diff --git a/src/pipecat/services/elevenlabs/tts.py b/src/pipecat/services/elevenlabs/tts.py index aeab4e787..37261a4ef 100644 --- a/src/pipecat/services/elevenlabs/tts.py +++ b/src/pipecat/services/elevenlabs/tts.py @@ -279,9 +279,12 @@ class ElevenLabsTTSService(AudioContextWordTTSService): await self._disconnect() async def flush_audio(self): - if self._websocket and self._context_id: - msg = {"context_id": self._context_id, "flush": True} - await self._websocket.send(json.dumps(msg)) + if not self._context_id or not self._websocket: + return + logger.trace(f"{self}: flushing audio") + msg = {"context_id": self._context_id, "flush": True} + await self._websocket.send(json.dumps(msg)) + self._context_id = None async def push_frame(self, frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM): await super().push_frame(frame, direction) @@ -389,14 +392,9 @@ class ElevenLabsTTSService(AudioContextWordTTSService): async for message in self._get_websocket(): msg = json.loads(message) # Check if this message belongs to the current context - # The default context may return null/None for context_id received_ctx_id = msg.get("contextId") - if ( - self._context_id is not None - and received_ctx_id is not None - and received_ctx_id != self._context_id - ): - logger.trace(f"Ignoring message from different context: {received_ctx_id}") + if not self.audio_context_available(received_ctx_id): + logger.trace(f"Ignoring message from unavailable context: {received_ctx_id}") continue if msg.get("audio"): @@ -405,14 +403,15 @@ class ElevenLabsTTSService(AudioContextWordTTSService): audio = base64.b64decode(msg["audio"]) frame = TTSAudioRawFrame(audio, self.sample_rate, 1) - await self.push_frame(frame) + await self.append_to_audio_context(received_ctx_id, frame) if msg.get("alignment"): word_times = calculate_word_times(msg["alignment"], self._cumulative_time) await self.add_word_timestamps(word_times) self._cumulative_time = word_times[-1][1] if msg.get("isFinal"): logger.trace(f"Received final message for context {received_ctx_id}") - # Context has finished + await self.remove_audio_context(received_ctx_id) + # Reset context tracking if this was our active context if self._context_id == received_ctx_id: self._context_id = None self._started = False @@ -464,6 +463,7 @@ class ElevenLabsTTSService(AudioContextWordTTSService): await self._websocket.send( json.dumps({"context_id": self._context_id, "close_context": True}) ) + await self.remove_audio_context(self._context_id) self._context_id = None if not self._started: @@ -471,6 +471,9 @@ class ElevenLabsTTSService(AudioContextWordTTSService): yield TTSStartedFrame() self._started = True self._cumulative_time = 0 + # Create new context ID and register it + self._context_id = str(uuid.uuid4()) + await self.create_audio_context(self._context_id) await self._send_text(text) await self.start_tts_usage_metrics(text) @@ -478,7 +481,9 @@ class ElevenLabsTTSService(AudioContextWordTTSService): logger.error(f"{self} error sending message: {e}") yield TTSStoppedFrame() self._started = False - self._context_id = None + if self._context_id: + await self.remove_audio_context(self._context_id) + self._context_id = None return yield None except Exception as e: diff --git a/src/pipecat/services/gemini_multimodal_live/gemini.py b/src/pipecat/services/gemini_multimodal_live/gemini.py index 25377e183..7ecf7e442 100644 --- a/src/pipecat/services/gemini_multimodal_live/gemini.py +++ b/src/pipecat/services/gemini_multimodal_live/gemini.py @@ -52,7 +52,7 @@ from pipecat.processors.aggregators.openai_llm_context import ( OpenAILLMContextFrame, ) from pipecat.processors.frame_processor import FrameDirection -from pipecat.services.llm_service import LLMService +from pipecat.services.llm_service import FunctionCallFromLLM, LLMService from pipecat.services.openai.llm import ( OpenAIAssistantContextAggregator, OpenAIUserContextAggregator, @@ -60,6 +60,7 @@ from pipecat.services.openai.llm import ( from pipecat.transcriptions.language import Language from pipecat.utils.string import match_endofsentence from pipecat.utils.time import time_now_iso8601 +from pipecat.utils.tracing.service_decorators import traced_gemini_live, traced_stt, traced_tts from . import events @@ -335,7 +336,7 @@ class GeminiMultimodalLiveLLMService(LLMService): *, api_key: str, base_url: str = "generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent", - model="models/gemini-2.5-flash-preview-native-audio-dialog", + model="models/gemini-2.0-flash-live-001", voice_id: str = "Charon", start_audio_paused: bool = False, start_video_paused: bool = False, @@ -378,6 +379,7 @@ class GeminiMultimodalLiveLLMService(LLMService): self._last_transcription_sent = "" self._bot_audio_buffer = bytearray() self._bot_text_buffer = "" + self._llm_output_buffer = "" self._sample_rate = 24000 @@ -471,6 +473,7 @@ class GeminiMultimodalLiveLLMService(LLMService): async def _handle_user_stopped_speaking(self, frame): self._user_is_speaking = False self._user_audio_buffer = bytearray() + await self.start_ttfb_metrics() if self._needs_turn_complete_message: self._needs_turn_complete_message = False evt = events.ClientContentMessage.model_validate( @@ -752,6 +755,8 @@ class GeminiMultimodalLiveLLMService(LLMService): logger.debug(f"Creating initial response: {messages}") + await self.start_ttfb_metrics() + evt = events.ClientContentMessage.model_validate( { "clientContent": { @@ -793,6 +798,8 @@ class GeminiMultimodalLiveLLMService(LLMService): return logger.debug(f"Creating response: {messages}") + await self.start_ttfb_metrics() + evt = events.ClientContentMessage.model_validate( { "clientContent": { @@ -803,6 +810,7 @@ class GeminiMultimodalLiveLLMService(LLMService): ) await self.send_client_event(evt) + @traced_gemini_live(operation="llm_tool_result") async def _tool_result(self, tool_result_message): # For now we're shoving the name into the tool_call_id field, so this # will work until we revisit that. @@ -827,6 +835,7 @@ class GeminiMultimodalLiveLLMService(LLMService): await self._websocket.send(response_message) # await self._websocket.send(json.dumps({"clientContent": {"turnComplete": True}})) + @traced_gemini_live(operation="llm_setup") async def _handle_evt_setup_complete(self, evt): # If this is our first context frame, run the LLM self._api_session_ready = True @@ -840,6 +849,8 @@ class GeminiMultimodalLiveLLMService(LLMService): if not part: return + await self.stop_ttfb_metrics() + # part.text is added when `modalities` is set to TEXT; otherwise, it's None text = part.text if text: @@ -873,26 +884,48 @@ class GeminiMultimodalLiveLLMService(LLMService): ) await self.push_frame(frame) + @traced_gemini_live(operation="llm_tool_call") async def _handle_evt_tool_call(self, evt): function_calls = evt.toolCall.functionCalls if not function_calls: return if not self._context: logger.error("Function calls are not supported without a context object.") - for call in function_calls: - await self.call_function( - context=self._context, - tool_call_id=call.id, - function_name=call.name, - arguments=call.args, - ) + function_calls_llm = [ + FunctionCallFromLLM( + context=self._context, + tool_call_id=f.id, + function_name=f.name, + arguments=f.args, + ) + for f in function_calls + ] + + await self.run_function_calls(function_calls_llm) + + @traced_gemini_live(operation="llm_response") async def _handle_evt_turn_complete(self, evt): self._bot_is_speaking = False text = self._bot_text_buffer - self._bot_text_buffer = "" - # Only push the TTSStoppedFrame the bot is outputting audio + # Determine output and modality for tracing + if text: + # TEXT modality + output_text = text + output_modality = "TEXT" + else: + # AUDIO modality + output_text = self._llm_output_buffer + output_modality = "AUDIO" + + # Trace the complete LLM response (this will be handled by the decorator) + # The decorator will extract the output text and usage metadata from the event + + self._bot_text_buffer = "" + self._llm_output_buffer = "" + + # Only push the TTSStoppedFrame if the bot is outputting audio # when text is found, modalities is set to TEXT and no audio # is produced. if not text: @@ -900,6 +933,13 @@ class GeminiMultimodalLiveLLMService(LLMService): await self.push_frame(LLMFullResponseEndFrame()) + @traced_stt + async def _handle_user_transcription( + self, transcript: str, is_final: bool, language: Optional[Language] = None + ): + """Handle a transcription result with tracing.""" + pass + async def _handle_evt_input_transcription(self, evt): """Handle the input transcription event. @@ -935,6 +975,9 @@ class GeminiMultimodalLiveLLMService(LLMService): # Send a TranscriptionFrame with the complete sentence logger.debug(f"[Transcription:user] [{complete_sentence}]") + await self._handle_user_transcription( + complete_sentence, True, self._settings["language"] + ) await self.push_frame( TranscriptionFrame( text=complete_sentence, @@ -957,6 +1000,9 @@ class GeminiMultimodalLiveLLMService(LLMService): if not text: return + # Collect text for tracing + self._llm_output_buffer += text + await self.push_frame(LLMTextFrame(text=text)) await self.push_frame(TTSTextFrame(text=text)) diff --git a/src/pipecat/services/gladia/config.py b/src/pipecat/services/gladia/config.py index 1e325686f..662996f1d 100644 --- a/src/pipecat/services/gladia/config.py +++ b/src/pipecat/services/gladia/config.py @@ -77,6 +77,7 @@ class TranslationConfig(BaseModel): lipsync: Whether to enable lip-sync optimization for translations context_adaptation: Whether to enable context-aware translation adaptation context: Additional context to help with translation accuracy + informal: Force informal language forms when available """ target_languages: Optional[List[str]] = None @@ -85,6 +86,7 @@ class TranslationConfig(BaseModel): lipsync: Optional[bool] = None context_adaptation: Optional[bool] = None context: Optional[str] = None + informal: Optional[bool] = None class RealtimeProcessingConfig(BaseModel): diff --git a/src/pipecat/services/google/llm.py b/src/pipecat/services/google/llm.py index 5efbbe8c0..f983b7342 100644 --- a/src/pipecat/services/google/llm.py +++ b/src/pipecat/services/google/llm.py @@ -42,7 +42,7 @@ from pipecat.processors.aggregators.openai_llm_context import ( ) from pipecat.processors.frame_processor import FrameDirection from pipecat.services.google.frames import LLMSearchResponseFrame -from pipecat.services.llm_service import LLMService +from pipecat.services.llm_service import FunctionCallFromLLM, LLMService from pipecat.services.openai.llm import ( OpenAIAssistantContextAggregator, OpenAIUserContextAggregator, @@ -83,7 +83,7 @@ class GoogleUserContextAggregator(OpenAIUserContextAggregator): await self.push_frame(frame) # Reset our accumulator state. - self.reset() + await self.reset() class GoogleAssistantContextAggregator(OpenAIAssistantContextAggregator): @@ -555,9 +555,11 @@ class GoogleLLMService(LLMService): contents=messages, config=generation_config, ) - await self.stop_ttfb_metrics() + function_calls = [] async for chunk in response: + # Stop TTFB metrics after the first chunk + await self.stop_ttfb_metrics() if chunk.usage_metadata: prompt_tokens += chunk.usage_metadata.prompt_token_count or 0 completion_tokens += chunk.usage_metadata.candidates_token_count or 0 @@ -576,11 +578,13 @@ class GoogleLLMService(LLMService): function_call = part.function_call id = function_call.id or str(uuid.uuid4()) logger.debug(f"Function call: {function_call.name}:{id}") - await self.call_function( - context=context, - tool_call_id=id, - function_name=function_call.name, - arguments=function_call.args or {}, + function_calls.append( + FunctionCallFromLLM( + context=context, + tool_call_id=id, + function_name=function_call.name, + arguments=function_call.args or {}, + ) ) if ( @@ -621,6 +625,8 @@ class GoogleLLMService(LLMService): "rendered_content": rendered_content, "origins": origins, } + + await self.run_function_calls(function_calls) except DeadlineExceeded: await self._call_event_handler("on_completion_timeout") except Exception as e: diff --git a/src/pipecat/services/google/llm_openai.py b/src/pipecat/services/google/llm_openai.py index 94b99072a..a497cb229 100644 --- a/src/pipecat/services/google/llm_openai.py +++ b/src/pipecat/services/google/llm_openai.py @@ -10,6 +10,8 @@ import os from openai import AsyncStream from openai.types.chat import ChatCompletionChunk +from pipecat.services.llm_service import FunctionCallFromLLM + # Suppress gRPC fork warnings os.environ["GRPC_ENABLE_FORK_SUPPORT"] = "false" @@ -18,7 +20,6 @@ from loguru import logger from pipecat.frames.frames import LLMTextFrame from pipecat.metrics.metrics import LLMTokenUsage from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext -from pipecat.services.openai.base_llm import OpenAIUnhandledFunctionException from pipecat.services.openai.llm import OpenAILLMService @@ -112,25 +113,26 @@ class GoogleLLMOpenAIBetaService(OpenAILLMService): logger.debug( f"Function list: {functions_list}, Arguments list: {arguments_list}, Tool ID list: {tool_id_list}" ) - for index, (function_name, arguments, tool_id) in enumerate( - zip(functions_list, arguments_list, tool_id_list), start=1 + + function_calls = [] + for function_name, arguments, tool_id in zip( + functions_list, arguments_list, tool_id_list ): if function_name == "": # TODO: Remove the _process_context method once Google resolves the bug # where the index is incorrectly set to None instead of returning the actual index, # which currently results in an empty function name(''). continue - if self.has_function(function_name): - run_llm = False - arguments = json.loads(arguments) - await self.call_function( + + arguments = json.loads(arguments) + + function_calls.append( + FunctionCallFromLLM( context=context, + tool_call_id=tool_id, function_name=function_name, arguments=arguments, - tool_call_id=tool_id, - run_llm=run_llm, - ) - else: - raise OpenAIUnhandledFunctionException( - f"The LLM tried to call a function named '{function_name}', but there isn't a callback registered for that function." ) + ) + + await self.run_function_calls(function_calls) diff --git a/src/pipecat/services/google/stt.py b/src/pipecat/services/google/stt.py index 4fd129af3..bf60541f5 100644 --- a/src/pipecat/services/google/stt.py +++ b/src/pipecat/services/google/stt.py @@ -747,6 +747,11 @@ class GoogleSTTService(STTService): try: while True: try: + if self._request_queue.empty(): + # wait for 10ms in case we don't have audio + await asyncio.sleep(0.01) + continue + # Start bi-directional streaming streaming_recognize = await self._client.streaming_recognize( requests=self._request_generator() diff --git a/src/pipecat/services/google/tts.py b/src/pipecat/services/google/tts.py index e28f9fadb..6e57b7b8d 100644 --- a/src/pipecat/services/google/tts.py +++ b/src/pipecat/services/google/tts.py @@ -362,8 +362,8 @@ class GoogleHttpTTSService(TTSService): # Skip the first 44 bytes to remove the WAV header audio_content = response.audio_content[44:] - # Read and yield audio data in chunks - CHUNK_SIZE = 1024 + CHUNK_SIZE = self.chunk_size + for i in range(0, len(audio_content), CHUNK_SIZE): chunk = audio_content[i : i + CHUNK_SIZE] if not chunk: @@ -505,9 +505,10 @@ class GoogleTTSService(TTSService): yield TTSStartedFrame() audio_buffer = b"" - CHUNK_SIZE = 1024 first_chunk_for_ttfb = False + CHUNK_SIZE = self.chunk_size + async for response in streaming_responses: chunk = response.audio_content if not chunk: diff --git a/src/pipecat/services/groq/tts.py b/src/pipecat/services/groq/tts.py index 6f73b1629..33fd3ce34 100644 --- a/src/pipecat/services/groq/tts.py +++ b/src/pipecat/services/groq/tts.py @@ -4,6 +4,8 @@ # SPDX-License-Identifier: BSD 2-Clause License # +import io +import wave from typing import AsyncGenerator, Optional from loguru import logger @@ -78,22 +80,26 @@ class GroqTTSService(TTSService): await self.start_ttfb_metrics() yield TTSStartedFrame() - response = await self._client.audio.speech.create( - model=self._model_name, - voice=self._voice_id, - response_format=self._output_format, - input=text, - ) + try: + response = await self._client.audio.speech.create( + model=self._model_name, + voice=self._voice_id, + response_format=self._output_format, + input=text, + ) - async for data in response.iter_bytes(): - if measuring_ttfb: - await self.stop_ttfb_metrics() - measuring_ttfb = False - # remove wav header if present - if data.startswith(b"RIFF"): - data = data[44:] - if len(data) == 0: - continue - yield TTSAudioRawFrame(data, self.sample_rate, 1) + async for data in response.iter_bytes(): + if measuring_ttfb: + await self.stop_ttfb_metrics() + measuring_ttfb = False + + with wave.open(io.BytesIO(data)) as w: + channels = w.getnchannels() + frame_rate = w.getframerate() + num_frames = w.getnframes() + bytes = w.readframes(num_frames) + yield TTSAudioRawFrame(bytes, frame_rate, channels) + except Exception as e: + logger.error(f"{self} exception: {e}") yield TTSStoppedFrame() diff --git a/src/pipecat/services/llm_service.py b/src/pipecat/services/llm_service.py index 21b62325d..5619cd35e 100644 --- a/src/pipecat/services/llm_service.py +++ b/src/pipecat/services/llm_service.py @@ -7,18 +7,23 @@ import asyncio import inspect from dataclasses import dataclass -from typing import Any, Awaitable, Callable, Mapping, Optional, Protocol, Set, Tuple, Type +from typing import Any, Awaitable, Callable, Dict, Mapping, Optional, Protocol, Sequence, Type from loguru import logger from pipecat.adapters.base_llm_adapter import BaseLLMAdapter from pipecat.adapters.services.open_ai_adapter import OpenAILLMAdapter from pipecat.frames.frames import ( + CancelFrame, + EndFrame, Frame, FunctionCallCancelFrame, + FunctionCallFromLLM, FunctionCallInProgressFrame, FunctionCallResultFrame, FunctionCallResultProperties, + FunctionCallsStartedFrame, + StartFrame, StartInterruptionFrame, UserImageRequestFrame, ) @@ -41,22 +46,6 @@ class FunctionCallResultCallback(Protocol): ) -> None: ... -@dataclass -class FunctionCallEntry: - """Represents an internal entry for a function call. - - Attributes: - function_name (Optional[str]): The name of the function. - handler (FunctionCallHandler): The handler for processing function call parameters. - cancel_on_interruption (bool): Flag indicating whether to cancel the call on interruption. - - """ - - function_name: Optional[str] - handler: FunctionCallHandler - cancel_on_interruption: bool - - @dataclass class FunctionCallParams: """Parameters for a function call. @@ -79,20 +68,78 @@ class FunctionCallParams: result_callback: FunctionCallResultCallback +@dataclass +class FunctionCallRegistryItem: + """Represents an entry in our function call registry. This is what the user + registers. + + Attributes: + function_name (Optional[str]): The name of the function. + handler (FunctionCallHandler): The handler for processing function call parameters. + cancel_on_interruption (bool): Flag indicating whether to cancel the call on interruption. + + """ + + function_name: Optional[str] + handler: FunctionCallHandler + cancel_on_interruption: bool + + +@dataclass +class FunctionCallRunnerItem: + """Represents an internal function call entry to our function call + runner. The runner executes function calls in order. + + Attributes: + registry_name (Optional[str]): The function call name registration (could be None). + function_name (str): The name of the function. + tool_call_id (str): A unique identifier for the function call. + arguments (Mapping[str, Any]): The arguments for the function. + context (OpenAILLMContext): The LLM context. + + """ + + registry_item: FunctionCallRegistryItem + function_name: str + tool_call_id: str + arguments: Mapping[str, Any] + context: OpenAILLMContext + run_llm: Optional[bool] = None + + class LLMService(AIService): - """This class is a no-op but serves as a base class for LLM services.""" + """This is the base class for all LLM services. It handles function calling + registration and execution. The class also provides event handlers. + + An event to know when an LLM service completion timeout occurs: + + @task.event_handler("on_completion_timeout") + async def on_completion_timeout(service): + ... + + And an event to know that function calls have been received from the LLM + service and that we are going to start executing them: + + @task.event_handler("on_function_calls_started") + async def on_function_calls_started(service, function_calls: Sequence[FunctionCallFromLLM]): + ... + + """ # OpenAILLMAdapter is used as the default adapter since it aligns with most LLM implementations. # However, subclasses should override this with a more specific adapter when necessary. adapter_class: Type[BaseLLMAdapter] = OpenAILLMAdapter - def __init__(self, **kwargs): + def __init__(self, run_in_parallel: bool = True, **kwargs): super().__init__(**kwargs) - self._functions = {} + self._run_in_parallel = run_in_parallel self._start_callbacks = {} self._adapter = self.adapter_class() - self._function_call_tasks: Set[Tuple[asyncio.Task, str, str]] = set() + self._functions: Dict[Optional[str], FunctionCallRegistryItem] = {} + self._function_call_tasks: Dict[asyncio.Task, FunctionCallRunnerItem] = {} + self._sequential_runner_task: Optional[asyncio.Task] = None + self._register_event_handler("on_function_calls_started") self._register_event_handler("on_completion_timeout") def get_llm_adapter(self) -> BaseLLMAdapter: @@ -107,13 +154,28 @@ class LLMService(AIService): ) -> Any: pass + async def start(self, frame: StartFrame): + await super().start(frame) + if not self._run_in_parallel: + await self._create_sequential_runner_task() + + async def stop(self, frame: EndFrame): + await super().stop(frame) + if not self._run_in_parallel: + await self._cancel_sequential_runner_task() + + async def cancel(self, frame: CancelFrame): + await super().cancel(frame) + if not self._run_in_parallel: + await self._cancel_sequential_runner_task() + async def process_frame(self, frame: Frame, direction: FrameDirection): await super().process_frame(frame, direction) if isinstance(frame, StartInterruptionFrame): await self._handle_interruptions(frame) - async def _handle_interruptions(self, frame: StartInterruptionFrame): + async def _handle_interruptions(self, _: StartInterruptionFrame): for function_name, entry in self._functions.items(): if entry.cancel_on_interruption: await self._cancel_function_call(function_name) @@ -124,11 +186,11 @@ class LLMService(AIService): handler: Any, start_callback=None, *, - cancel_on_interruption: bool = False, + cancel_on_interruption: bool = True, ): # Registering a function with the function_name set to None will run # that handler for all functions - self._functions[function_name] = FunctionCallEntry( + self._functions[function_name] = FunctionCallRegistryItem( function_name=function_name, handler=handler, cancel_on_interruption=cancel_on_interruption, @@ -157,25 +219,43 @@ class LLMService(AIService): return True return function_name in self._functions.keys() - async def call_function( - self, - *, - context: OpenAILLMContext, - tool_call_id: str, - function_name: str, - arguments: Mapping[str, Any], - run_llm: bool = True, - ): - if not function_name in self._functions.keys() and not None in self._functions.keys(): + async def run_function_calls(self, function_calls: Sequence[FunctionCallFromLLM]): + if len(function_calls) == 0: return - task = self.create_task( - self._run_function_call(context, tool_call_id, function_name, arguments, run_llm) - ) + await self._call_event_handler("on_function_calls_started", function_calls) - self._function_call_tasks.add((task, tool_call_id, function_name)) + # Push frame both downstream and upstream + started_frame_downstream = FunctionCallsStartedFrame(function_calls=function_calls) + started_frame_upstream = FunctionCallsStartedFrame(function_calls=function_calls) + await self.push_frame(started_frame_downstream, FrameDirection.DOWNSTREAM) + await self.push_frame(started_frame_upstream, FrameDirection.UPSTREAM) - task.add_done_callback(self._function_call_task_finished) + for function_call in function_calls: + if function_call.function_name in self._functions.keys(): + item = self._functions[function_call.function_name] + elif None in self._functions.keys(): + item = self._functions[None] + else: + logger.warning( + f"{self} is calling '{function_call.function_name}', but it's not registered." + ) + continue + + runner_item = FunctionCallRunnerItem( + registry_item=item, + function_name=function_call.function_name, + tool_call_id=function_call.tool_call_id, + arguments=function_call.arguments, + context=function_call.context, + ) + + if self._run_in_parallel: + task = self.create_task(self._run_function_call(runner_item)) + self._function_call_tasks[task] = runner_item + task.add_done_callback(self._function_call_task_finished) + else: + await self._sequential_runner_queue.put(runner_item) async def call_start_function(self, context: OpenAILLMContext, function_name: str): if function_name in self._start_callbacks.keys(): @@ -203,43 +283,57 @@ class LLMService(AIService): FrameDirection.UPSTREAM, ) - async def _run_function_call( - self, - context: OpenAILLMContext, - tool_call_id: str, - function_name: str, - arguments: Mapping[str, Any], - run_llm: bool = True, - ): - if function_name in self._functions.keys(): - entry = self._functions[function_name] + async def _create_sequential_runner_task(self): + if not self._sequential_runner_task: + self._sequential_runner_queue = asyncio.Queue() + self._sequential_runner_task = self.create_task(self._sequential_runner_handler()) + + async def _cancel_sequential_runner_task(self): + if self._sequential_runner_task: + await self.cancel_task(self._sequential_runner_task) + self._sequential_runner_task = None + + async def _sequential_runner_handler(self): + while True: + runner_item = await self._sequential_runner_queue.get() + task = self.create_task(self._run_function_call(runner_item)) + self._function_call_tasks[task] = runner_item + # Since we run tasks sequentially we don't need to call + # task.add_done_callback(self._function_call_task_finished). + await self.wait_for_task(task) + del self._function_call_tasks[task] + + async def _run_function_call(self, runner_item: FunctionCallRunnerItem): + if runner_item.function_name in self._functions.keys(): + item = self._functions[runner_item.function_name] elif None in self._functions.keys(): - entry = self._functions[None] + item = self._functions[None] else: return logger.debug( - f"{self} Calling function [{function_name}:{tool_call_id}] with arguments {arguments}" + f"{self} Calling function [{runner_item.function_name}:{runner_item.tool_call_id}] with arguments {runner_item.arguments}" ) # NOTE(aleix): This needs to be removed after we remove the deprecation. - await self.call_start_function(context, function_name) + await self.call_start_function(runner_item.context, runner_item.function_name) - # Push a SystemFrame downstream. This frame will let our assistant context aggregator - # know that we are in the middle of a function call. Some contexts/aggregators may - # not need this. But some definitely do (Anthropic, for example). - # Also push a SystemFrame upstream for use by other processors, like STTMuteFilter. + # Push a function call in-progress downstream. This frame will let our + # assistant context aggregator know that we are in the middle of a + # function call. Some contexts/aggregators may not need this. But some + # definitely do (Anthropic, for example). Also push it upstream for use + # by other processors, like STTMuteFilter. progress_frame_downstream = FunctionCallInProgressFrame( - function_name=function_name, - tool_call_id=tool_call_id, - arguments=arguments, - cancel_on_interruption=entry.cancel_on_interruption, + function_name=runner_item.function_name, + tool_call_id=runner_item.tool_call_id, + arguments=runner_item.arguments, + cancel_on_interruption=item.cancel_on_interruption, ) progress_frame_upstream = FunctionCallInProgressFrame( - function_name=function_name, - tool_call_id=tool_call_id, - arguments=arguments, - cancel_on_interruption=entry.cancel_on_interruption, + function_name=runner_item.function_name, + tool_call_id=runner_item.tool_call_id, + arguments=runner_item.arguments, + cancel_on_interruption=item.cancel_on_interruption, ) # Push frame both downstream and upstream @@ -251,24 +345,26 @@ class LLMService(AIService): result: Any, *, properties: Optional[FunctionCallResultProperties] = None ): result_frame_downstream = FunctionCallResultFrame( - function_name=function_name, - tool_call_id=tool_call_id, - arguments=arguments, + function_name=runner_item.function_name, + tool_call_id=runner_item.tool_call_id, + arguments=runner_item.arguments, result=result, + run_llm=runner_item.run_llm, properties=properties, ) result_frame_upstream = FunctionCallResultFrame( - function_name=function_name, - tool_call_id=tool_call_id, - arguments=arguments, + function_name=runner_item.function_name, + tool_call_id=runner_item.tool_call_id, + arguments=runner_item.arguments, result=result, + run_llm=runner_item.run_llm, properties=properties, ) await self.push_frame(result_frame_downstream, FrameDirection.DOWNSTREAM) await self.push_frame(result_frame_upstream, FrameDirection.UPSTREAM) - signature = inspect.signature(entry.handler) + signature = inspect.signature(item.handler) if len(signature.parameters) > 1: import warnings @@ -279,24 +375,32 @@ class LLMService(AIService): DeprecationWarning, ) - await entry.handler( - function_name, tool_call_id, arguments, self, context, function_call_result_callback + await item.handler( + runner_item.function_name, + runner_item.tool_call_id, + runner_item.arguments, + self, + runner_item.context, + function_call_result_callback, ) else: params = FunctionCallParams( - function_name=function_name, - tool_call_id=tool_call_id, - arguments=arguments, + function_name=runner_item.function_name, + tool_call_id=runner_item.tool_call_id, + arguments=runner_item.arguments, llm=self, - context=context, + context=runner_item.context, result_callback=function_call_result_callback, ) - await entry.handler(params) + await item.handler(params) - async def _cancel_function_call(self, function_name: str): + async def _cancel_function_call(self, function_name: Optional[str]): cancelled_tasks = set() - for task, tool_call_id, name in self._function_call_tasks: - if name == function_name: + for task, runner_item in self._function_call_tasks.items(): + if runner_item.registry_item.function_name == function_name: + name = runner_item.function_name + tool_call_id = runner_item.tool_call_id + # We remove the callback because we are going to cancel the task # now, otherwise we will be removing it from the set while we # are iterating. @@ -306,23 +410,20 @@ class LLMService(AIService): await self.cancel_task(task) - frame = FunctionCallCancelFrame( - function_name=function_name, tool_call_id=tool_call_id - ) + frame = FunctionCallCancelFrame(function_name=name, tool_call_id=tool_call_id) await self.push_frame(frame) - logger.debug(f"{self} Function call [{name}:{tool_call_id}] has been cancelled") - cancelled_tasks.add(task) + logger.debug(f"{self} Function call [{name}:{tool_call_id}] has been cancelled") + # Remove all cancelled tasks from our set. for task in cancelled_tasks: self._function_call_task_finished(task) def _function_call_task_finished(self, task: asyncio.Task): - tuple_to_remove = next((t for t in self._function_call_tasks if t[0] == task), None) - if tuple_to_remove: - self._function_call_tasks.discard(tuple_to_remove) + if task in self._function_call_tasks: + del self._function_call_tasks[task] # The task is finished so this should exit immediately. We need to # do this because otherwise the task manager would report a dangling # task if we don't remove it. diff --git a/src/pipecat/services/minimax/tts.py b/src/pipecat/services/minimax/tts.py index 932996751..86f954755 100644 --- a/src/pipecat/services/minimax/tts.py +++ b/src/pipecat/services/minimax/tts.py @@ -227,7 +227,8 @@ class MiniMaxHttpTTSService(TTSService): # Process the streaming response buffer = bytearray() - CHUNK_SIZE = 1024 + + CHUNK_SIZE = self.chunk_size async for chunk in response.content.iter_chunked(CHUNK_SIZE): if not chunk: @@ -279,10 +280,8 @@ class MiniMaxHttpTTSService(TTSService): await self.stop_ttfb_metrics() yield TTSAudioRawFrame( audio=audio_chunk, - sample_rate=self._settings["audio_setting"][ - "sample_rate" - ], - num_channels=self._settings["audio_setting"]["channel"], + sample_rate=self.sample_rate, + num_channels=1, ) except ValueError as e: logger.error(f"Error converting hex to binary: {e}") diff --git a/src/pipecat/services/openai/base_llm.py b/src/pipecat/services/openai/base_llm.py index e90f87f3c..2badfed96 100644 --- a/src/pipecat/services/openai/base_llm.py +++ b/src/pipecat/services/openai/base_llm.py @@ -34,14 +34,10 @@ from pipecat.processors.aggregators.openai_llm_context import ( OpenAILLMContextFrame, ) from pipecat.processors.frame_processor import FrameDirection -from pipecat.services.llm_service import LLMService +from pipecat.services.llm_service import FunctionCallFromLLM, LLMService from pipecat.utils.tracing.service_decorators import traced_llm -class OpenAIUnhandledFunctionException(Exception): - pass - - class BaseOpenAILLMService(LLMService): """This is the base for all services that use the AsyncOpenAI client. @@ -260,23 +256,22 @@ class BaseOpenAILLMService(LLMService): arguments_list.append(arguments) tool_id_list.append(tool_call_id) - for index, (function_name, arguments, tool_id) in enumerate( - zip(functions_list, arguments_list, tool_id_list), start=1 + function_calls = [] + + for function_name, arguments, tool_id in zip( + functions_list, arguments_list, tool_id_list ): - if self.has_function(function_name): - run_llm = False - arguments = json.loads(arguments) - await self.call_function( + arguments = json.loads(arguments) + function_calls.append( + FunctionCallFromLLM( context=context, + tool_call_id=tool_id, function_name=function_name, arguments=arguments, - tool_call_id=tool_id, - run_llm=run_llm, - ) - else: - raise OpenAIUnhandledFunctionException( - f"The LLM tried to call a function named '{function_name}', but there isn't a callback registered for that function." ) + ) + + await self.run_function_calls(function_calls) async def process_frame(self, frame: Frame, direction: FrameDirection): await super().process_frame(frame, direction) diff --git a/src/pipecat/services/openai/tts.py b/src/pipecat/services/openai/tts.py index 61fb3e77c..946d5e396 100644 --- a/src/pipecat/services/openai/tts.py +++ b/src/pipecat/services/openai/tts.py @@ -125,7 +125,7 @@ class OpenAITTSService(TTSService): await self.start_tts_usage_metrics(text) - CHUNK_SIZE = 1024 + CHUNK_SIZE = self.chunk_size yield TTSStartedFrame() async for chunk in r.iter_bytes(CHUNK_SIZE): diff --git a/src/pipecat/services/openai_realtime_beta/openai.py b/src/pipecat/services/openai_realtime_beta/openai.py index 579be2ebe..4ea5843bf 100644 --- a/src/pipecat/services/openai_realtime_beta/openai.py +++ b/src/pipecat/services/openai_realtime_beta/openai.py @@ -48,9 +48,11 @@ from pipecat.processors.aggregators.openai_llm_context import ( OpenAILLMContextFrame, ) from pipecat.processors.frame_processor import FrameDirection -from pipecat.services.llm_service import LLMService +from pipecat.services.llm_service import FunctionCallFromLLM, LLMService from pipecat.services.openai.llm import OpenAIContextAggregatorPair +from pipecat.transcriptions.language import Language from pipecat.utils.time import time_now_iso8601 +from pipecat.utils.tracing.service_decorators import traced_openai_realtime, traced_stt, traced_tts from . import events from .context import ( @@ -76,10 +78,6 @@ class CurrentAudioResponse: total_size: int = 0 -class OpenAIUnhandledFunctionException(Exception): - pass - - class OpenAIRealtimeBetaLLMService(LLMService): # Overriding the default adapter to use the OpenAIRealtimeLLMAdapter one. adapter_class = OpenAIRealtimeLLMAdapter @@ -100,6 +98,7 @@ class OpenAIRealtimeBetaLLMService(LLMService): self.api_key = api_key self.base_url = full_url + self.set_model_name(model) self._session_properties: events.SessionProperties = ( session_properties or events.SessionProperties() @@ -402,6 +401,7 @@ class OpenAIRealtimeBetaLLMService(LLMService): # errors are fatal, so exit the receive loop return + @traced_openai_realtime(operation="llm_setup") async def _handle_evt_session_created(self, evt): # session.created is received right after connecting. Send a message # to configure the session properties. @@ -467,6 +467,13 @@ class OpenAIRealtimeBetaLLMService(LLMService): InterimTranscriptionFrame(evt.delta, "", time_now_iso8601(), result=evt) ) + @traced_stt + async def _handle_user_transcription( + self, transcript: str, is_final: bool, language: Optional[Language] = None + ): + """Handle a transcription result with tracing.""" + pass + async def handle_evt_input_audio_transcription_completed(self, evt): await self._call_event_handler("on_conversation_item_updated", evt.item_id, None) @@ -475,6 +482,7 @@ class OpenAIRealtimeBetaLLMService(LLMService): # no way to get a language code? TranscriptionFrame(evt.transcript, "", time_now_iso8601(), result=evt) ) + await self._handle_user_transcription(evt.transcript, True, Language.EN) pair = self._user_and_response_message_tuple if pair: user, assistant = pair @@ -493,6 +501,7 @@ class OpenAIRealtimeBetaLLMService(LLMService): for future in futures: future.set_result(evt.item) + @traced_openai_realtime(operation="llm_response") async def _handle_evt_response_done(self, evt): # todo: figure out whether there's anything we need to do for "cancelled" events # usage metrics @@ -574,25 +583,18 @@ class OpenAIRealtimeBetaLLMService(LLMService): await self._handle_function_call_items(function_calls) async def _handle_function_call_items(self, items): - total_items = len(items) - for index, item in enumerate(items): - function_name = item.name - tool_id = item.call_id - arguments = json.loads(item.arguments) - if self.has_function(function_name): - run_llm = index == total_items - 1 - if function_name in self._functions.keys() or None in self._functions.keys(): - await self.call_function( - context=self._context, - tool_call_id=tool_id, - function_name=function_name, - arguments=arguments, - run_llm=run_llm, - ) - else: - raise OpenAIUnhandledFunctionException( - f"The LLM tried to call a function named '{function_name}', but there isn't a callback registered for that function." + function_calls = [] + for item in items: + args = json.loads(item.arguments) + function_calls.append( + FunctionCallFromLLM( + context=self._context, + tool_call_id=item.call_id, + function_name=item.name, + arguments=args, ) + ) + await self.run_function_calls(function_calls) # # state and client events for the current conversation @@ -609,6 +611,7 @@ class OpenAIRealtimeBetaLLMService(LLMService): self._context.llm_needs_initial_messages = True await self._connect() + @traced_openai_realtime(operation="llm_request") async def _create_response(self): if not self._api_session_ready: self._run_llm_when_api_session_ready = True diff --git a/src/pipecat/services/piper/tts.py b/src/pipecat/services/piper/tts.py index 7686196de..65caa3650 100644 --- a/src/pipecat/services/piper/tts.py +++ b/src/pipecat/services/piper/tts.py @@ -74,19 +74,18 @@ class PiperTTSService(TTSService): async with self._session.post(self._base_url, data=text, headers=headers) as response: if response.status != 200: - eror = await response.text() + error = await response.text() logger.error( - f"{self} error getting audio (status: {response.status}, error: {eror})" + f"{self} error getting audio (status: {response.status}, error: {error})" ) yield ErrorFrame( - f"Error getting audio (status: {response.status}, error: {eror})" + f"Error getting audio (status: {response.status}, error: {error})" ) return await self.start_tts_usage_metrics(text) - # Process the streaming response - CHUNK_SIZE = 1024 + CHUNK_SIZE = self.chunk_size yield TTSStartedFrame() async for chunk in response.content.iter_chunked(CHUNK_SIZE): diff --git a/src/pipecat/services/rime/tts.py b/src/pipecat/services/rime/tts.py index b3f46961f..821eafb23 100644 --- a/src/pipecat/services/rime/tts.py +++ b/src/pipecat/services/rime/tts.py @@ -26,6 +26,7 @@ from pipecat.frames.frames import ( ) from pipecat.processors.frame_processor import FrameDirection from pipecat.services.tts_service import AudioContextWordTTSService, TTSService +from pipecat.transcriptions import language from pipecat.transcriptions.language import Language from pipecat.utils.text.base_text_aggregator import BaseTextAggregator from pipecat.utils.text.skip_tags_aggregator import SkipTagsAggregator @@ -49,6 +50,8 @@ def language_to_rime_language(language: Language) -> str: str: Three-letter language code used by Rime (e.g., 'eng' for English). """ LANGUAGE_MAP = { + Language.DE: "ger", + Language.FR: "fra", Language.EN: "eng", Language.ES: "spa", } @@ -352,6 +355,7 @@ class RimeTTSService(AudioContextWordTTSService): class RimeHttpTTSService(TTSService): class InputParams(BaseModel): + language: Optional[Language] = Language.EN pause_between_brackets: Optional[bool] = False phonemize_between_brackets: Optional[bool] = False inline_speed_alpha: Optional[str] = None @@ -377,6 +381,9 @@ class RimeHttpTTSService(TTSService): self._session = aiohttp_session self._base_url = "https://users.rime.ai/v1/rime-tts" self._settings = { + "lang": self.language_to_service_language(params.language) + if params.language + else "eng", "speedAlpha": params.speed_alpha, "reduceLatency": params.reduce_latency, "pauseBetweenBrackets": params.pause_between_brackets, @@ -391,6 +398,10 @@ class RimeHttpTTSService(TTSService): def can_generate_metrics(self) -> bool: return True + def language_to_service_language(self, language: Language) -> str | None: + """Convert pipecat language to Rime language code.""" + return language_to_rime_language(language) + @traced_tts async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]: logger.debug(f"{self}: Generating TTS [{text}]") @@ -430,8 +441,7 @@ class RimeHttpTTSService(TTSService): yield TTSStartedFrame() - # Process the streaming response - CHUNK_SIZE = 1024 + CHUNK_SIZE = self.chunk_size async for chunk in response.content.iter_chunked(CHUNK_SIZE): if need_to_strip_wav_header and chunk.startswith(b"RIFF"): diff --git a/src/pipecat/services/tts_service.py b/src/pipecat/services/tts_service.py index 0bdcd0d1c..904f603a9 100644 --- a/src/pipecat/services/tts_service.py +++ b/src/pipecat/services/tts_service.py @@ -106,6 +106,19 @@ class TTSService(AIService): def sample_rate(self) -> int: return self._sample_rate + @property + def chunk_size(self) -> int: + """This property indicates how much audio we download (from TTS services + that require chunking) before we start pushing the first audio + frame. This will make sure we download the rest of the audio while audio + is being played without causing audio glitches (specially at the + beginning). Of course, this will also depend on how fast the TTS service + generates bytes. + + """ + CHUNK_SECONDS = 0.5 + return int(self.sample_rate * CHUNK_SECONDS * 2) # 2 bytes/sample + async def set_model(self, model: str): self.set_model_name(model) diff --git a/src/pipecat/services/xtts/tts.py b/src/pipecat/services/xtts/tts.py index 18528f0ea..2111e4d72 100644 --- a/src/pipecat/services/xtts/tts.py +++ b/src/pipecat/services/xtts/tts.py @@ -152,7 +152,7 @@ class XTTSService(TTSService): yield TTSStartedFrame() - CHUNK_SIZE = 1024 + CHUNK_SIZE = self.chunk_size buffer = bytearray() async for chunk in r.content.iter_chunked(CHUNK_SIZE): diff --git a/src/pipecat/transports/base_input.py b/src/pipecat/transports/base_input.py index 2a2343883..68f58694a 100644 --- a/src/pipecat/transports/base_input.py +++ b/src/pipecat/transports/base_input.py @@ -17,6 +17,8 @@ from pipecat.audio.turn.base_turn_analyzer import ( from pipecat.audio.vad.vad_analyzer import VADAnalyzer, VADState from pipecat.frames.frames import ( BotInterruptionFrame, + BotStartedSpeakingFrame, + BotStoppedSpeakingFrame, CancelFrame, EmulateUserStartedSpeakingFrame, EmulateUserStoppedSpeakingFrame, @@ -51,6 +53,9 @@ class BaseInputTransport(FrameProcessor): # Input sample rate. It will be initialized on StartFrame. self._sample_rate = 0 + # Track bot speaking state for interruption logic + self._bot_speaking = False + # We read audio from a single queue one at a time and we then run VAD in # a thread. Therefore, only one thread should be necessary. self._executor = ThreadPoolExecutor(max_workers=1) @@ -189,6 +194,12 @@ class BaseInputTransport(FrameProcessor): await self.push_frame(frame, direction) elif isinstance(frame, BotInterruptionFrame): await self._handle_bot_interruption(frame) + elif isinstance(frame, BotStartedSpeakingFrame): + await self._handle_bot_started_speaking(frame) + await self.push_frame(frame, direction) + elif isinstance(frame, BotStoppedSpeakingFrame): + await self._handle_bot_stopped_speaking(frame) + await self.push_frame(frame, direction) elif isinstance(frame, EmulateUserStartedSpeakingFrame): logger.debug("Emulating user started speaking") await self._handle_user_interruption(UserStartedSpeakingFrame(emulated=True)) @@ -230,13 +241,26 @@ class BaseInputTransport(FrameProcessor): if isinstance(frame, UserStartedSpeakingFrame): logger.debug("User started speaking") await self.push_frame(frame) + + # Only push StartInterruptionFrame if: + # 1. No interruption config is set, OR + # 2. Interruption config is set but bot is not speaking + should_push_immediate_interruption = ( + not self.interruption_strategies or not self._bot_speaking + ) + # Make sure we notify about interruptions quickly out-of-band. - if self.interruptions_allowed: + if should_push_immediate_interruption and self.interruptions_allowed: await self._start_interruption() # Push an out-of-band frame (i.e. not using the ordered push # frame task) to stop everything, specially at the output # transport. await self.push_frame(StartInterruptionFrame()) + elif self.interruption_strategies and self._bot_speaking: + logger.debug( + "User started speaking while bot is speaking with interruption config - " + "deferring interruption to aggregator" + ) elif isinstance(frame, UserStoppedSpeakingFrame): logger.debug("User stopped speaking") await self.push_frame(frame) @@ -244,6 +268,16 @@ class BaseInputTransport(FrameProcessor): await self._stop_interruption() await self.push_frame(StopInterruptionFrame()) + # + # Handle bot speaking state + # + + async def _handle_bot_started_speaking(self, frame: BotStartedSpeakingFrame): + self._bot_speaking = True + + async def _handle_bot_stopped_speaking(self, frame: BotStoppedSpeakingFrame): + self._bot_speaking = False + # # Audio input # diff --git a/src/pipecat/transports/base_output.py b/src/pipecat/transports/base_output.py index dabd92829..386f223d7 100644 --- a/src/pipecat/transports/base_output.py +++ b/src/pipecat/transports/base_output.py @@ -351,6 +351,7 @@ class BaseOutputTransport(FrameProcessor): sample_rate=self._sample_rate, num_channels=frame.num_channels, ) + chunk.transport_destination = self._destination await self._audio_queue.put(chunk) self._audio_buffer = self._audio_buffer[self._audio_chunk_size :] diff --git a/src/pipecat/transports/network/fastapi_websocket.py b/src/pipecat/transports/network/fastapi_websocket.py index 205fb8c9e..97ff8e03a 100644 --- a/src/pipecat/transports/network/fastapi_websocket.py +++ b/src/pipecat/transports/network/fastapi_websocket.py @@ -92,7 +92,7 @@ class FastAPIWebsocketClient: async def trigger_client_connected(self): await self._callbacks.on_client_connected(self._websocket) - async def trigger_client_timout(self): + async def trigger_client_timeout(self): await self._callbacks.on_session_timeout(self._websocket) def _can_send(self): @@ -188,7 +188,7 @@ class FastAPIWebsocketInputTransport(BaseInputTransport): async def _monitor_websocket(self): """Wait for self._params.session_timeout seconds, if the websocket is still open, trigger timeout event.""" await asyncio.sleep(self._params.session_timeout) - await self._client.trigger_client_timout() + await self._client.trigger_client_timeout() class FastAPIWebsocketOutputTransport(BaseOutputTransport): diff --git a/src/pipecat/transports/services/livekit.py b/src/pipecat/transports/services/livekit.py index 736054ab9..6381c1dd4 100644 --- a/src/pipecat/transports/services/livekit.py +++ b/src/pipecat/transports/services/livekit.py @@ -363,8 +363,6 @@ class LiveKitInputTransport(BaseInputTransport): self._audio_in_task = None self._vad_analyzer: Optional[VADAnalyzer] = params.vad_analyzer self._resampler = create_default_resampler() - if self._initialized: - return # Whether we have seen a StartFrame already. self._initialized = False diff --git a/src/pipecat/utils/tracing/service_attributes.py b/src/pipecat/utils/tracing/service_attributes.py index 619dfbf11..8f90ad3be 100644 --- a/src/pipecat/utils/tracing/service_attributes.py +++ b/src/pipecat/utils/tracing/service_attributes.py @@ -6,7 +6,7 @@ """Functions for adding attributes to OpenTelemetry spans.""" -from typing import TYPE_CHECKING, Any, Dict, Optional +from typing import TYPE_CHECKING, Any, Dict, List, Optional # Import for type checking only if TYPE_CHECKING: @@ -256,3 +256,207 @@ def add_llm_span_attributes( for key, value in kwargs.items(): if isinstance(value, (str, int, float, bool)): span.set_attribute(key, value) + + +def add_gemini_live_span_attributes( + span: "Span", + service_name: str, + model: str, + operation_name: str, + voice_id: Optional[str] = None, + language: Optional[str] = None, + modalities: Optional[str] = None, + settings: Optional[Dict[str, Any]] = None, + tools: Optional[List[Dict]] = None, + tools_serialized: Optional[str] = None, + transcript: Optional[str] = None, + is_input: Optional[bool] = None, + text_output: Optional[str] = None, + audio_data_size: Optional[int] = None, + **kwargs, +) -> None: + """Add Gemini Live specific attributes to a span. + + Args: + span: The span to add attributes to + service_name: Name of the service + model: Model name/identifier + operation_name: Name of the operation (setup, model_turn, tool_call, etc.) + voice_id: Voice identifier used for output + language: Language code for the session + modalities: Supported modalities (e.g., "AUDIO", "TEXT") + settings: Service configuration settings + tools: Available tools/functions list + tools_serialized: JSON-serialized tools for detailed inspection + transcript: Transcription text + is_input: Whether transcript is input (True) or output (False) + text_output: Text output from model + audio_data_size: Size of audio data in bytes + **kwargs: Additional attributes to add + """ + # Add standard attributes + span.set_attribute("gen_ai.system", "gcp.gemini") + span.set_attribute("gen_ai.request.model", model) + span.set_attribute("gen_ai.operation.name", operation_name) + span.set_attribute("service.operation", operation_name) + + # Add optional attributes + if voice_id: + span.set_attribute("voice_id", voice_id) + + if language: + span.set_attribute("language", language) + + if modalities: + span.set_attribute("modalities", modalities) + + if transcript: + span.set_attribute("transcript", transcript) + if is_input is not None: + span.set_attribute("transcript.is_input", is_input) + + if text_output: + span.set_attribute("text_output", text_output) + + if audio_data_size is not None: + span.set_attribute("audio.data_size_bytes", audio_data_size) + + if tools: + span.set_attribute("tools.count", len(tools)) + span.set_attribute("tools.available", True) + + # Add individual tool names for easier filtering + tool_names = [] + for tool in tools: + if isinstance(tool, dict) and "name" in tool: + tool_names.append(tool["name"]) + elif hasattr(tool, "name"): + tool_name = getattr(tool, "name", None) + if tool_name is not None: + tool_names.append(tool_name) + + if tool_names: + span.set_attribute("tools.names", ",".join(tool_names)) + + if tools_serialized: + span.set_attribute("tools.definitions", tools_serialized) + + # Add settings if provided + if settings: + for key, value in settings.items(): + if isinstance(value, (str, int, float, bool)): + span.set_attribute(f"settings.{key}", value) + elif key == "vad" and value: + # Handle VAD settings specially + if hasattr(value, "disabled") and value.disabled is not None: + span.set_attribute("settings.vad.disabled", value.disabled) + if hasattr(value, "start_sensitivity") and value.start_sensitivity: + span.set_attribute( + "settings.vad.start_sensitivity", value.start_sensitivity.value + ) + if hasattr(value, "end_sensitivity") and value.end_sensitivity: + span.set_attribute("settings.vad.end_sensitivity", value.end_sensitivity.value) + + # Add any additional keyword arguments as attributes + for key, value in kwargs.items(): + if isinstance(value, (str, int, float, bool)): + span.set_attribute(key, value) + + +def add_openai_realtime_span_attributes( + span: "Span", + service_name: str, + model: str, + operation_name: str, + session_properties: Optional[Dict[str, Any]] = None, + transcript: Optional[str] = None, + is_input: Optional[bool] = None, + context_messages: Optional[str] = None, + function_calls: Optional[List[Dict]] = None, + tools: Optional[List[Dict]] = None, + tools_serialized: Optional[str] = None, + audio_data_size: Optional[int] = None, + **kwargs, +) -> None: + """Add OpenAI Realtime specific attributes to a span. + + Args: + span: The span to add attributes to + service_name: Name of the service + model: Model name/identifier + operation_name: Name of the operation (setup, transcription, response, etc.) + session_properties: Session configuration properties + transcript: Transcription text + is_input: Whether transcript is input (True) or output (False) + context_messages: JSON-serialized context messages + function_calls: Function calls being made + tools: Available tools/functions list + tools_serialized: JSON-serialized tools for detailed inspection + audio_data_size: Size of audio data in bytes + **kwargs: Additional attributes to add + """ + # Add standard attributes + span.set_attribute("gen_ai.system", "openai") + span.set_attribute("gen_ai.request.model", model) + span.set_attribute("gen_ai.operation.name", operation_name) + span.set_attribute("service.operation", operation_name) + + # Add optional attributes + if transcript: + span.set_attribute("transcript", transcript) + if is_input is not None: + span.set_attribute("transcript.is_input", is_input) + + if context_messages: + span.set_attribute("input", context_messages) + + if audio_data_size is not None: + span.set_attribute("audio.data_size_bytes", audio_data_size) + + if tools: + span.set_attribute("tools.count", len(tools)) + span.set_attribute("tools.available", True) + + # Add individual tool names for easier filtering + tool_names = [] + for tool in tools: + if isinstance(tool, dict) and "name" in tool: + tool_names.append(tool["name"]) + elif hasattr(tool, "name"): + tool_names.append(tool.name) + elif isinstance(tool, dict) and "function" in tool and "name" in tool["function"]: + tool_names.append(tool["function"]["name"]) + + if tool_names: + span.set_attribute("tools.names", ",".join(tool_names)) + + if tools_serialized: + span.set_attribute("tools.definitions", tools_serialized) + + if function_calls: + span.set_attribute("function_calls.count", len(function_calls)) + if function_calls: + call = function_calls[0] + if hasattr(call, "name"): + span.set_attribute("function_calls.first_name", call.name) + elif isinstance(call, dict) and "name" in call: + span.set_attribute("function_calls.first_name", call["name"]) + + # Add session properties if provided + if session_properties: + for key, value in session_properties.items(): + if isinstance(value, (str, int, float, bool)): + span.set_attribute(f"session.{key}", value) + elif key == "turn_detection" and value is not None: + if isinstance(value, bool): + span.set_attribute("session.turn_detection.enabled", value) + elif isinstance(value, dict): + span.set_attribute("session.turn_detection.enabled", True) + for td_key, td_value in value.items(): + if isinstance(td_value, (str, int, float, bool)): + span.set_attribute(f"session.turn_detection.{td_key}", td_value) + + # Add any additional keyword arguments as attributes + for key, value in kwargs.items(): + if isinstance(value, (str, int, float, bool)): + span.set_attribute(key, value) diff --git a/src/pipecat/utils/tracing/service_decorators.py b/src/pipecat/utils/tracing/service_decorators.py index 4341d308c..c016827d6 100644 --- a/src/pipecat/utils/tracing/service_decorators.py +++ b/src/pipecat/utils/tracing/service_decorators.py @@ -24,7 +24,9 @@ if TYPE_CHECKING: from opentelemetry import trace from pipecat.utils.tracing.service_attributes import ( + add_gemini_live_span_attributes, add_llm_span_attributes, + add_openai_realtime_span_attributes, add_stt_span_attributes, add_tts_span_attributes, ) @@ -477,3 +479,525 @@ def traced_llm(func: Optional[Callable] = None, *, name: Optional[str] = None) - if func is not None: return decorator(func) return decorator + + +def traced_gemini_live(operation: str) -> Callable: + """Traces Gemini Live service methods with operation-specific attributes. + + This decorator automatically captures relevant information based on the operation type: + - llm_setup: Configuration, tools definitions, and system instructions + - llm_tool_call: Function call information + - llm_tool_result: Function execution results + - llm_response: Complete LLM response with usage and output + + Args: + operation: The operation name (matches the event type being handled) + + Returns: + Wrapped method with Gemini Live specific tracing. + """ + if not is_tracing_available(): + return _noop_decorator + + def decorator(func): + @functools.wraps(func) + async def wrapper(self, *args, **kwargs): + try: + if not is_tracing_available(): + return await func(self, *args, **kwargs) + + service_class_name = self.__class__.__name__ + span_name = f"{operation}" + + # Get the parent context - turn context if available, otherwise service context + turn_context = get_current_turn_context() + parent_context = turn_context or _get_parent_service_context(self) + + # Create a new span as child of the turn span or service span + tracer = trace.get_tracer("pipecat") + with tracer.start_as_current_span( + span_name, context=parent_context + ) as current_span: + try: + # Base service attributes + model_name = getattr( + self, "model_name", getattr(self, "_model_name", "unknown") + ) + voice_id = getattr(self, "_voice_id", None) + language_code = getattr(self, "_language_code", None) + settings = getattr(self, "_settings", {}) + + # Get modalities if available + modalities = None + if hasattr(self, "_settings") and "modalities" in self._settings: + modality_obj = self._settings["modalities"] + if hasattr(modality_obj, "value"): + modalities = modality_obj.value + else: + modalities = str(modality_obj) + + # Operation-specific attribute collection + operation_attrs = {} + + if operation == "llm_setup": + # Capture detailed tool information + tools = getattr(self, "_tools", None) + if tools: + # Handle different tool formats + tools_list = [] + tools_serialized = None + + try: + if hasattr(tools, "standard_tools"): + # ToolsSchema object + tools_list = tools.standard_tools + # Serialize the tools for detailed inspection + tools_serialized = json.dumps( + [ + { + "name": tool.name + if hasattr(tool, "name") + else tool.get("name", "unknown"), + "description": tool.description + if hasattr(tool, "description") + else tool.get("description", ""), + "properties": tool.properties + if hasattr(tool, "properties") + else tool.get("properties", {}), + "required": tool.required + if hasattr(tool, "required") + else tool.get("required", []), + } + for tool in tools_list + ] + ) + elif isinstance(tools, list): + # List of tool dictionaries or objects + tools_list = tools + tools_serialized = json.dumps( + [ + { + "name": tool.get("name", "unknown") + if isinstance(tool, dict) + else getattr(tool, "name", "unknown"), + "description": tool.get("description", "") + if isinstance(tool, dict) + else getattr(tool, "description", ""), + "properties": tool.get("properties", {}) + if isinstance(tool, dict) + else getattr(tool, "properties", {}), + "required": tool.get("required", []) + if isinstance(tool, dict) + else getattr(tool, "required", []), + } + for tool in tools_list + ] + ) + + if tools_list: + operation_attrs["tools"] = tools_list + operation_attrs["tools_serialized"] = tools_serialized + + except Exception as e: + logging.warning(f"Error serializing tools for tracing: {e}") + # Fallback to basic tool count + if tools_list: + operation_attrs["tools"] = tools_list + + # Capture system instruction information + system_instruction = getattr(self, "_system_instruction", None) + if system_instruction: + operation_attrs["system_instruction"] = system_instruction[ + :500 + ] # Truncate if very long + + # Capture context system instructions if available + if hasattr(self, "_context") and self._context: + try: + context_system = self._context.extract_system_instructions() + if context_system: + operation_attrs["context_system_instruction"] = ( + context_system[:500] + ) # Truncate if very long + except Exception as e: + logging.warning( + f"Error extracting context system instructions: {e}" + ) + + elif operation == "llm_tool_call" and args: + # Extract tool call information + evt = args[0] if args else None + if evt and hasattr(evt, "toolCall") and evt.toolCall.functionCalls: + function_calls = evt.toolCall.functionCalls + if function_calls: + # Add information about the first function call + call = function_calls[0] + operation_attrs["tool.function_name"] = call.name + operation_attrs["tool.call_id"] = call.id + operation_attrs["tool.calls_count"] = len(function_calls) + + # Add all function names being called + all_function_names = [c.name for c in function_calls] + operation_attrs["tool.all_function_names"] = ",".join( + all_function_names + ) + + # Add arguments for the first call (truncated if too long) + try: + args_str = json.dumps(call.args) if call.args else "{}" + if len(args_str) > 1000: + args_str = args_str[:1000] + "..." + operation_attrs["tool.arguments"] = args_str + except Exception: + operation_attrs["tool.arguments"] = str(call.args)[:1000] + + elif operation == "llm_tool_result" and args: + # Extract tool result information + tool_result_message = args[0] if args else None + if tool_result_message and isinstance(tool_result_message, dict): + # Extract the tool call information + tool_call_id = tool_result_message.get("tool_call_id") + tool_call_name = tool_result_message.get("tool_call_name") + result_content = tool_result_message.get("content") + + if tool_call_id: + operation_attrs["tool.call_id"] = tool_call_id + if tool_call_name: + operation_attrs["tool.function_name"] = tool_call_name + + # Parse and capture the result + if result_content: + try: + result = json.loads(result_content) + # Serialize the result, truncating if too long + result_str = json.dumps(result) + if len(result_str) > 2000: # Larger limit for results + result_str = result_str[:2000] + "..." + operation_attrs["tool.result"] = result_str + + # Add result status/success indicator if present + if isinstance(result, dict): + if "error" in result: + operation_attrs["tool.result_status"] = "error" + elif "success" in result: + operation_attrs["tool.result_status"] = "success" + else: + operation_attrs["tool.result_status"] = "completed" + + except json.JSONDecodeError as e: + operation_attrs["tool.result"] = ( + f"Invalid JSON: {str(result_content)[:500]}" + ) + operation_attrs["tool.result_status"] = "parse_error" + except Exception as e: + operation_attrs["tool.result"] = ( + f"Error processing result: {str(e)}" + ) + operation_attrs["tool.result_status"] = "processing_error" + + elif operation == "llm_response" and args: + # Extract usage and response metadata from turn complete event + evt = args[0] if args else None + if evt and hasattr(evt, "usageMetadata") and evt.usageMetadata: + usage = evt.usageMetadata + + # Token usage - basic attributes for span visibility + if hasattr(usage, "promptTokenCount"): + operation_attrs["tokens.prompt"] = usage.promptTokenCount or 0 + if hasattr(usage, "responseTokenCount"): + operation_attrs["tokens.completion"] = ( + usage.responseTokenCount or 0 + ) + if hasattr(usage, "totalTokenCount"): + operation_attrs["tokens.total"] = usage.totalTokenCount or 0 + + # Get output text and modality from service state + text = getattr(self, "_bot_text_buffer", "") + audio_text = getattr(self, "_llm_output_buffer", "") + + if text: + # TEXT modality + operation_attrs["output"] = text + operation_attrs["output_modality"] = "TEXT" + elif audio_text: + # AUDIO modality + operation_attrs["output"] = audio_text + operation_attrs["output_modality"] = "AUDIO" + + # Add turn completion status + if ( + evt + and hasattr(evt, "serverContent") + and evt.serverContent.turnComplete + ): + operation_attrs["turn_complete"] = True + + # Add all attributes to the span + add_gemini_live_span_attributes( + span=current_span, + service_name=service_class_name, + model=model_name, + operation_name=operation, + voice_id=voice_id, + language=language_code, + modalities=modalities, + settings=settings, + **operation_attrs, + ) + + # For llm_response operation, also handle token usage metrics + if operation == "llm_response" and hasattr(self, "start_llm_usage_metrics"): + evt = args[0] if args else None + if evt and hasattr(evt, "usageMetadata") and evt.usageMetadata: + usage = evt.usageMetadata + # Create LLMTokenUsage object + from pipecat.metrics.metrics import LLMTokenUsage + + tokens = LLMTokenUsage( + prompt_tokens=usage.promptTokenCount or 0, + completion_tokens=usage.responseTokenCount or 0, + total_tokens=usage.totalTokenCount or 0, + ) + _add_token_usage_to_span(current_span, tokens) + + # Capture TTFB metric if available + ttfb = getattr(getattr(self, "_metrics", None), "ttfb", None) + if ttfb is not None: + current_span.set_attribute("metrics.ttfb", ttfb) + + # Run the original function + result = await func(self, *args, **kwargs) + + return result + + except Exception as e: + current_span.record_exception(e) + current_span.set_status(trace.Status(trace.StatusCode.ERROR, str(e))) + raise + + except Exception as e: + logging.error(f"Error in Gemini Live tracing (continuing without tracing): {e}") + # If tracing fails, fall back to the original function + return await func(self, *args, **kwargs) + + return wrapper + + return decorator + + +def traced_openai_realtime(operation: str) -> Callable: + """Traces OpenAI Realtime service methods with operation-specific attributes. + + This decorator automatically captures relevant information based on the operation type: + - llm_setup: Session configuration and tools + - llm_request: Context and input messages + - llm_response: Usage metadata, output, and function calls + + Args: + operation: The operation name (matches the event type being handled) + + Returns: + Wrapped method with OpenAI Realtime specific tracing. + """ + if not is_tracing_available(): + return _noop_decorator + + def decorator(func): + @functools.wraps(func) + async def wrapper(self, *args, **kwargs): + try: + if not is_tracing_available(): + return await func(self, *args, **kwargs) + + service_class_name = self.__class__.__name__ + span_name = f"{operation}" + + # Get the parent context - turn context if available, otherwise service context + turn_context = get_current_turn_context() + parent_context = turn_context or _get_parent_service_context(self) + + # Create a new span as child of the turn span or service span + tracer = trace.get_tracer("pipecat") + with tracer.start_as_current_span( + span_name, context=parent_context + ) as current_span: + try: + # Base service attributes + model_name = getattr( + self, "model_name", getattr(self, "_model_name", "unknown") + ) + + # Operation-specific attribute collection + operation_attrs = {} + + if operation == "llm_setup": + # Capture session properties and tools + session_properties = getattr(self, "_session_properties", None) + if session_properties: + try: + # Convert to dict for easier processing + if hasattr(session_properties, "model_dump"): + props_dict = session_properties.model_dump() + elif hasattr(session_properties, "__dict__"): + props_dict = session_properties.__dict__ + else: + props_dict = {} + + operation_attrs["session_properties"] = props_dict + + # Extract tools if available + tools = props_dict.get("tools") + if tools: + operation_attrs["tools"] = tools + try: + operation_attrs["tools_serialized"] = json.dumps(tools) + except Exception as e: + logging.warning(f"Error serializing OpenAI tools: {e}") + + # Extract instructions + instructions = props_dict.get("instructions") + if instructions: + operation_attrs["instructions"] = instructions[:500] + + except Exception as e: + logging.warning(f"Error processing session properties: {e}") + + # Also check context for tools + if hasattr(self, "_context") and self._context: + try: + context_tools = getattr(self._context, "tools", None) + if context_tools and not operation_attrs.get("tools"): + operation_attrs["tools"] = context_tools + operation_attrs["tools_serialized"] = json.dumps( + context_tools + ) + except Exception as e: + logging.warning(f"Error extracting context tools: {e}") + + elif operation == "llm_request": + # Capture context messages being sent + if hasattr(self, "_context") and self._context: + try: + messages = self._context.get_messages_for_logging() + if messages: + operation_attrs["context_messages"] = json.dumps(messages) + except Exception as e: + logging.warning(f"Error getting context messages: {e}") + + elif operation == "llm_response" and args: + # Extract usage and response metadata + evt = args[0] if args else None + if evt and hasattr(evt, "response"): + response = evt.response + + # Token usage - basic attributes for span visibility + if hasattr(response, "usage"): + usage = response.usage + if hasattr(usage, "input_tokens"): + operation_attrs["tokens.prompt"] = usage.input_tokens + if hasattr(usage, "output_tokens"): + operation_attrs["tokens.completion"] = usage.output_tokens + if hasattr(usage, "total_tokens"): + operation_attrs["tokens.total"] = usage.total_tokens + + # Response status and metadata + if hasattr(response, "status"): + operation_attrs["response.status"] = response.status + + if hasattr(response, "id"): + operation_attrs["response.id"] = response.id + + # Output items and extract transcript for output field + if hasattr(response, "output") and response.output: + operation_attrs["response.output_items"] = len(response.output) + + # Extract assistant transcript and function calls + assistant_transcript = "" + function_calls = [] + + for item in response.output: + if ( + hasattr(item, "content") + and item.content + and hasattr(item, "role") + and item.role == "assistant" + ): + for content in item.content: + if ( + hasattr(content, "transcript") + and content.transcript + ): + assistant_transcript += content.transcript + " " + + elif hasattr(item, "type") and item.type == "function_call": + function_call_info = { + "name": getattr(item, "name", "unknown"), + "call_id": getattr(item, "call_id", "unknown"), + } + if hasattr(item, "arguments"): + args_str = item.arguments + if len(args_str) > 500: + args_str = args_str[:500] + "..." + function_call_info["arguments"] = args_str + function_calls.append(function_call_info) + + if assistant_transcript.strip(): + operation_attrs["output"] = assistant_transcript.strip() + + if function_calls: + operation_attrs["function_calls"] = function_calls + operation_attrs["function_calls.count"] = len( + function_calls + ) + all_names = [call["name"] for call in function_calls] + operation_attrs["function_calls.all_names"] = ",".join( + all_names + ) + + # Add all attributes to the span + add_openai_realtime_span_attributes( + span=current_span, + service_name=service_class_name, + model=model_name, + operation_name=operation, + **operation_attrs, + ) + + # For llm_response operation, also handle token usage metrics + if operation == "llm_response" and hasattr(self, "start_llm_usage_metrics"): + evt = args[0] if args else None + if evt and hasattr(evt, "response") and hasattr(evt.response, "usage"): + usage = evt.response.usage + # Create LLMTokenUsage object + from pipecat.metrics.metrics import LLMTokenUsage + + tokens = LLMTokenUsage( + prompt_tokens=getattr(usage, "input_tokens", 0), + completion_tokens=getattr(usage, "output_tokens", 0), + total_tokens=getattr(usage, "total_tokens", 0), + ) + _add_token_usage_to_span(current_span, tokens) + + # Capture TTFB metric if available + ttfb = getattr(getattr(self, "_metrics", None), "ttfb", None) + if ttfb is not None: + current_span.set_attribute("metrics.ttfb", ttfb) + + # Run the original function + result = await func(self, *args, **kwargs) + + return result + + except Exception as e: + current_span.record_exception(e) + current_span.set_status(trace.Status(trace.StatusCode.ERROR, str(e))) + raise + + except Exception as e: + logging.error(f"Error in OpenAI Realtime tracing (continuing without tracing): {e}") + # If tracing fails, fall back to the original function + return await func(self, *args, **kwargs) + + return wrapper + + return decorator diff --git a/src/pipecat/utils/tracing/turn_trace_observer.py b/src/pipecat/utils/tracing/turn_trace_observer.py index 26fc5b38a..f67ca3b28 100644 --- a/src/pipecat/utils/tracing/turn_trace_observer.py +++ b/src/pipecat/utils/tracing/turn_trace_observer.py @@ -35,7 +35,11 @@ class TurnTraceObserver(BaseObserver): """ def __init__( - self, turn_tracker: TurnTrackingObserver, conversation_id: Optional[str] = None, **kwargs + self, + turn_tracker: TurnTrackingObserver, + conversation_id: Optional[str] = None, + additional_span_attributes: Optional[dict] = None, + **kwargs, ): super().__init__(**kwargs) self._turn_tracker = turn_tracker @@ -47,6 +51,7 @@ class TurnTraceObserver(BaseObserver): # Conversation tracking properties self._conversation_span: Optional["Span"] = None self._conversation_id = conversation_id + self._additional_span_attributes = additional_span_attributes or {} if turn_tracker: @@ -89,6 +94,9 @@ class TurnTraceObserver(BaseObserver): # Set span attributes self._conversation_span.set_attribute("conversation.id", conversation_id) self._conversation_span.set_attribute("conversation.type", "voice") + # Set custom otel attributes if provided + for k, v in (self._additional_span_attributes or {}).items(): + self._conversation_span.set_attribute(k, v) # Update the conversation context provider context_provider.set_current_conversation_context( diff --git a/tests/test_interruption_strategies.py b/tests/test_interruption_strategies.py new file mode 100644 index 000000000..aa1bd7625 --- /dev/null +++ b/tests/test_interruption_strategies.py @@ -0,0 +1,24 @@ +# +# Copyright (c) 2024-2025 Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +import unittest + +from pipecat.audio.interruptions.min_words_interruption_strategy import MinWordsInterruptionStrategy + + +class TestInterruptionStrategy(unittest.IsolatedAsyncioTestCase): + async def test_min_words(self): + strategy = MinWordsInterruptionStrategy(min_words=2) + await strategy.append_text("Hello") + self.assertEqual(await strategy.should_interrupt(), False) + await strategy.append_text(" there!") + self.assertEqual(await strategy.should_interrupt(), True) + # Reset and check again + await strategy.reset() + await strategy.append_text("Hello!") + self.assertEqual(await strategy.should_interrupt(), False) + await strategy.append_text(" How are you?") + self.assertEqual(await strategy.should_interrupt(), True) diff --git a/tests/test_pipeline.py b/tests/test_pipeline.py index 9bc737b1a..84485098d 100644 --- a/tests/test_pipeline.py +++ b/tests/test_pipeline.py @@ -117,7 +117,8 @@ class TestPipelineTask(unittest.IsolatedAsyncioTestCase): async def test_task_add_observer(self): frame_received = False - frame_add_count = 0 + frame_count_1 = 0 + frame_count_2 = 0 class CustomObserver(BaseObserver): async def on_push_frame(self, data: FramePushed): @@ -126,28 +127,41 @@ class TestPipelineTask(unittest.IsolatedAsyncioTestCase): if isinstance(data.frame, TextFrame): frame_received = True - class CustomAddObserver(BaseObserver): + class CustomAddObserver1(BaseObserver): async def on_push_frame(self, data: FramePushed): - nonlocal frame_add_count + nonlocal frame_count_1 if isinstance(data.source, IdentityFilter) and isinstance(data.frame, TextFrame): - frame_add_count += 1 + frame_count_1 += 1 + + class CustomAddObserver2(BaseObserver): + async def on_push_frame(self, data: FramePushed): + nonlocal frame_count_2 + + if isinstance(data.source, IdentityFilter) and isinstance(data.frame, TextFrame): + frame_count_2 += 1 identity = IdentityFilter() pipeline = Pipeline([identity]) task = PipelineTask(pipeline, observers=[CustomObserver()]) + + # Add a new observer right away, before doing anything else with the task. + observer1 = CustomAddObserver1() + task.add_observer(observer1) + task.set_event_loop(asyncio.get_event_loop()) async def delayed_add_observer(): - observer = CustomAddObserver() - # Wait after the pipeline is started and add an observer. + observer2 = CustomAddObserver2() + # Wait after the pipeline is started and add another observer. await asyncio.sleep(0.1) - await task.add_observer(observer) + task.add_observer(observer2) # Push a TextFrame and wait for the observer to pick it up. await task.queue_frame(TextFrame(text="Hello Downstream!")) await asyncio.sleep(0.1) - # Remove the observer - await task.remove_observer(observer) + # Remove both observers. + await task.remove_observer(observer1) + await task.remove_observer(observer2) # Push another TextFrame. This time the counter should not # increments since we have removed the observer. await task.queue_frame(TextFrame(text="Hello Downstream!")) @@ -158,7 +172,8 @@ class TestPipelineTask(unittest.IsolatedAsyncioTestCase): await asyncio.gather(task.run(), delayed_add_observer()) assert frame_received - assert frame_add_count == 1 + assert frame_count_1 == 1 + assert frame_count_2 == 1 async def test_task_started_ended_event_handler(self): start_received = False diff --git a/tests/test_piper_tts.py b/tests/test_piper_tts.py index 673ea087a..75893f93f 100644 --- a/tests/test_piper_tts.py +++ b/tests/test_piper_tts.py @@ -47,8 +47,9 @@ async def test_run_piper_tts_success(aiohttp_client): # Write out some chunked byte data # In reality, you’d return WAV data or similar - data_chunk_1 = b"\x00\x01\x02\x03" * 1024 # 4096 bytes, 04 TTSAudioRawFrame - data_chunk_2 = b"\x04\x05\x06\x07" * 1024 # another chunk + CHUNK_SIZE = 24000 + data_chunk_1 = b"\x00\x01\x02\x03" * CHUNK_SIZE # 4xTTSAudioRawFrame + data_chunk_2 = b"\x04\x05\x06\x07" * CHUNK_SIZE # another chunk await resp.write(data_chunk_1) await asyncio.sleep(0.01) # simulate async chunk delay await resp.write(data_chunk_2)