diff --git a/CHANGELOG.md b/CHANGELOG.md index 564650905..7c734120f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- Added `GoogleHttpTTSService` which uses Google's HTTP TTS API. + +- Added `TavusTransport`, a new transport implementation compatible with any + Pipecat pipeline. When using the `TavusTransport`the Pipecat bot will + connect in the same room as the Tavus Avatar and the user. + +- Added `PlivoFrameSerializer` to support Plivo calls. A full running example + has also been added to `examples/plivo-chatbot`. + +- Added `UserBotLatencyLogObserver`. This is an observer that logs the latency + between when the user stops speaking and when the bot starts speaking. This + gives you an initial idea on how quickly the AI services respond. + - Added `SarvamTTSService`, which implements Sarvam AI's TTS API: https://docs.sarvam.ai/api-reference-docs/text-to-speech/convert. @@ -26,8 +39,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 correspond to the `StartFrame`, `StopFrame`, `EndFrame` and `CancelFrame` respectively. -- Added additional languages to `LmntTTSService`. Languages include: `hi`, `id`, - `it`, `ja`, `nl`, `pl`, `ru`, `sv`, `th`, `tr`, `uk`, `vi`. +- Added additional languages to `LmntTTSService`. Languages include: `hi`, + `id`, `it`, `ja`, `nl`, `pl`, `ru`, `sv`, `th`, `tr`, `uk`, `vi`. - Added a `model` parameter to the `LmntTTSService` constructor, allowing switching between LMNT models. @@ -65,8 +78,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ``` By default, Pipecat has implemented service decorators to trace execution of - STT, LLM, and TTS services. You can enable tracing by setting `enable_tracing` - to `True` in the PipelineTask. + STT, LLM, and TTS services. You can enable tracing by setting + `enable_tracing` to `True` in the PipelineTask. - Added `TurnTrackingObserver`, which tracks the start and end of a user/bot turn pair and emits events `on_turn_started` and `on_turn_stopped` @@ -76,6 +89,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- Updated `GoogleTTSService` to use Google's streaming TTS API. The default + voice also updated to `en-US-Chirp3-HD-Charon`. + +- ⚠️ Refactored the `TavusVideoService`, so it acts like a proxy, sending audio + to Tavus and receiving both audio and video. This will make + `TavusVideoService` usable with any Pipecat pipeline and with any transport. + This is a **breaking change**, check the + `examples/foundational/21a-tavus-layer-small-webrtc.py` to see how to use it. + +- `DailyTransport` now uses custom microphone audio tracks instead of virtual + microphones. Now, multiple Daily transports can be used in the same process. + +- `DailyTransport` now captures audio from individual participants instead of + the whole room. This allows identifying audio frames per participant. + - Updated the default model for `AnthropicLLMService` to `claude-sonnet-4-20250514`. @@ -117,6 +145,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- Fixed a `DailyTransport` issue that would cause images needing resize to block + the event loop. + - Fixed an issue with `ElevenLabsTTSService` where changing the model or voice while the service is running wasn't working. @@ -130,6 +161,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Performance +- `DailyTransport`: process audio, video and events in separate tasks. + - Don't create event handler tasks if no user event handlers have been registered. diff --git a/README.md b/README.md index ae6c3cc53..3966e8d65 100644 --- a/README.md +++ b/README.md @@ -58,11 +58,12 @@ You can connect to Pipecat from any platform using our official SDKs: | Text-to-Speech | [AWS](https://docs.pipecat.ai/server/services/tts/aws), [Azure](https://docs.pipecat.ai/server/services/tts/azure), [Cartesia](https://docs.pipecat.ai/server/services/tts/cartesia), [Deepgram](https://docs.pipecat.ai/server/services/tts/deepgram), [ElevenLabs](https://docs.pipecat.ai/server/services/tts/elevenlabs), [FastPitch (NVIDIA)](https://docs.pipecat.ai/server/services/tts/fastpitch), [Fish](https://docs.pipecat.ai/server/services/tts/fish), [Google](https://docs.pipecat.ai/server/services/tts/google), [LMNT](https://docs.pipecat.ai/server/services/tts/lmnt), [MiniMax](https://docs.pipecat.ai/server/services/tts/minimax), [Neuphonic](https://docs.pipecat.ai/server/services/tts/neuphonic), [OpenAI](https://docs.pipecat.ai/server/services/tts/openai), [Piper](https://docs.pipecat.ai/server/services/tts/piper), [PlayHT](https://docs.pipecat.ai/server/services/tts/playht), [Rime](https://docs.pipecat.ai/server/services/tts/rime), [Sarvam](https://docs.pipecat.ai/server/services/tts/sarvam), [XTTS](https://docs.pipecat.ai/server/services/tts/xtts) | | Speech-to-Speech | [AWS Nova Sonic](https://docs.pipecat.ai/server/services/s2s/aws), [Gemini Multimodal Live](https://docs.pipecat.ai/server/services/s2s/gemini), [OpenAI Realtime](https://docs.pipecat.ai/server/services/s2s/openai) | | Transport | [Daily (WebRTC)](https://docs.pipecat.ai/server/services/transport/daily), [FastAPI Websocket](https://docs.pipecat.ai/server/services/transport/fastapi-websocket), [SmallWebRTCTransport](https://docs.pipecat.ai/server/services/transport/small-webrtc), [WebSocket Server](https://docs.pipecat.ai/server/services/transport/websocket-server), Local | +| Serializers | [Plivo](https://docs.pipecat.ai/server/utilities/serializers/plivo), [Twilio](https://docs.pipecat.ai/server/utilities/serializers/twilio), [Telnyx](https://docs.pipecat.ai/server/utilities/serializers/telnyx) | | Video | [Tavus](https://docs.pipecat.ai/server/services/video/tavus), [Simli](https://docs.pipecat.ai/server/services/video/simli) | | Memory | [mem0](https://docs.pipecat.ai/server/services/memory/mem0) | | Vision & Image | [fal](https://docs.pipecat.ai/server/services/image-generation/fal), [Google Imagen](https://docs.pipecat.ai/server/services/image-generation/fal), [Moondream](https://docs.pipecat.ai/server/services/vision/moondream) | | Audio Processing | [Silero VAD](https://docs.pipecat.ai/server/utilities/audio/silero-vad-analyzer), [Krisp](https://docs.pipecat.ai/server/utilities/audio/krisp-filter), [Koala](https://docs.pipecat.ai/server/utilities/audio/koala-filter), [Noisereduce](https://docs.pipecat.ai/server/utilities/audio/noisereduce-filter) | -| Analytics & Metrics | [Sentry](https://docs.pipecat.ai/server/services/analytics/sentry) | +| Analytics & Metrics | [OpenTelemetry](https://docs.pipecat.ai/server/utilities/opentelemetry), [Sentry](https://docs.pipecat.ai/server/services/analytics/sentry) | πŸ“š [View full services documentation β†’](https://docs.pipecat.ai/server/services/supported-services) diff --git a/examples/chatbot-audio-recording/bot.py b/examples/chatbot-audio-recording/bot.py index 5428a0d2f..128c97c7e 100644 --- a/examples/chatbot-audio-recording/bot.py +++ b/examples/chatbot-audio-recording/bot.py @@ -128,7 +128,14 @@ async def main(): ] ) - task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True)) + task = PipelineTask( + pipeline, + params=PipelineParams( + audio_in_sample_rate=16000, + audio_out_sample_rate=16000, + allow_interruptions=True, + ), + ) @audiobuffer.event_handler("on_audio_data") async def on_audio_data(buffer, audio, sample_rate, num_channels): diff --git a/examples/foundational/04a-transports-daily.py b/examples/foundational/04a-transports-daily.py index b8125c5da..98060dbab 100644 --- a/examples/foundational/04a-transports-daily.py +++ b/examples/foundational/04a-transports-daily.py @@ -37,9 +37,9 @@ async def main(): token, "Respond bot", DailyParams( + audio_in_enabled=True, audio_out_enabled=True, transcription_enabled=True, - vad_enabled=True, vad_analyzer=SileroVADAnalyzer(), ), ) diff --git a/examples/foundational/21-tavus-layer-tavus-transport.py b/examples/foundational/21-tavus-layer-tavus-transport.py new file mode 100644 index 000000000..c9bcd2501 --- /dev/null +++ b/examples/foundational/21-tavus-layer-tavus-transport.py @@ -0,0 +1,112 @@ +# +# Copyright (c) 2024–2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +import asyncio +import os +import sys + +import aiohttp +from dotenv import load_dotenv +from loguru import logger + +from pipecat.audio.vad.silero import SileroVADAnalyzer +from pipecat.pipeline.pipeline import Pipeline +from pipecat.pipeline.runner import PipelineRunner +from pipecat.pipeline.task import PipelineParams, PipelineTask +from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext +from pipecat.services.cartesia.tts import CartesiaTTSService +from pipecat.services.deepgram.stt import DeepgramSTTService +from pipecat.services.google.llm import GoogleLLMService +from pipecat.transports.services.tavus import TavusParams, TavusTransport + +load_dotenv(override=True) + +logger.remove(0) +logger.add(sys.stderr, level="DEBUG") + + +async def main(): + async with aiohttp.ClientSession() as session: + transport = TavusTransport( + bot_name="Pipecat bot", + api_key=os.getenv("TAVUS_API_KEY"), + replica_id=os.getenv("TAVUS_REPLICA_ID"), + session=session, + params=TavusParams( + audio_in_enabled=True, + audio_out_enabled=True, + microphone_out_enabled=False, + vad_analyzer=SileroVADAnalyzer(), + ), + ) + + stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY")) + + tts = CartesiaTTSService( + api_key=os.getenv("CARTESIA_API_KEY"), + voice_id="a167e0f3-df7e-4d52-a9c3-f949145efdab", + ) + + llm = GoogleLLMService(api_key=os.getenv("GOOGLE_API_KEY")) + + messages = [ + { + "role": "system", + "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.", + }, + ] + + context = OpenAILLMContext(messages) + context_aggregator = llm.create_context_aggregator(context) + + pipeline = Pipeline( + [ + transport.input(), # Transport user input + stt, # STT + context_aggregator.user(), # User responses + llm, # LLM + tts, # TTS + transport.output(), # Transport bot output + context_aggregator.assistant(), # Assistant spoken responses + ] + ) + + task = PipelineTask( + pipeline, + params=PipelineParams( + audio_in_sample_rate=16000, + audio_out_sample_rate=24000, + allow_interruptions=True, + enable_metrics=True, + enable_usage_metrics=True, + report_only_initial_ttfb=True, + ), + ) + + @transport.event_handler("on_client_connected") + async def on_client_connected(transport, participant): + logger.info(f"Client connected") + # Kick off the conversation. + messages.append( + { + "role": "system", + "content": "Start by greeting the user and ask how you can help.", + } + ) + await task.queue_frames([context_aggregator.user().get_context_frame()]) + + @transport.event_handler("on_client_disconnected") + async def on_client_disconnected(transport, participant): + logger.info(f"Client disconnected") + await task.cancel() + + runner = PipelineRunner(handle_sigint=False) + + await runner.run(task) + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/examples/foundational/21a-tavus-layer-small-webrtc.py b/examples/foundational/21a-tavus-layer-small-webrtc.py new file mode 100644 index 000000000..2f557b5cd --- /dev/null +++ b/examples/foundational/21a-tavus-layer-small-webrtc.py @@ -0,0 +1,125 @@ +# +# Copyright (c) 2024–2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +import argparse +import os + +import aiohttp +from dotenv import load_dotenv +from loguru import logger + +from pipecat.audio.vad.silero import SileroVADAnalyzer +from pipecat.pipeline.pipeline import Pipeline +from pipecat.pipeline.runner import PipelineRunner +from pipecat.pipeline.task import PipelineParams, PipelineTask +from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext +from pipecat.services.cartesia.tts import CartesiaTTSService +from pipecat.services.deepgram.stt import DeepgramSTTService +from pipecat.services.google.llm import GoogleLLMService +from pipecat.services.tavus.video import TavusVideoService +from pipecat.transports.base_transport import TransportParams +from pipecat.transports.network.small_webrtc import SmallWebRTCTransport +from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection + +load_dotenv(override=True) + + +async def run_bot(webrtc_connection: SmallWebRTCConnection, _: argparse.Namespace): + logger.info(f"Starting bot") + async with aiohttp.ClientSession() as session: + transport = SmallWebRTCTransport( + webrtc_connection=webrtc_connection, + params=TransportParams( + audio_in_enabled=True, + audio_out_enabled=True, + video_out_enabled=True, + video_out_is_live=True, + vad_analyzer=SileroVADAnalyzer(), + video_out_width=1280, + video_out_height=720, + ), + ) + + stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY")) + + tts = CartesiaTTSService( + api_key=os.getenv("CARTESIA_API_KEY"), + voice_id="a167e0f3-df7e-4d52-a9c3-f949145efdab", + ) + + llm = GoogleLLMService(api_key=os.getenv("GOOGLE_API_KEY")) + + tavus = TavusVideoService( + api_key=os.getenv("TAVUS_API_KEY"), + replica_id=os.getenv("TAVUS_REPLICA_ID"), + session=session, + ) + + messages = [ + { + "role": "system", + "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.", + }, + ] + + context = OpenAILLMContext(messages) + context_aggregator = llm.create_context_aggregator(context) + + pipeline = Pipeline( + [ + transport.input(), # Transport user input + stt, # STT + context_aggregator.user(), # User responses + llm, # LLM + tts, # TTS + tavus, # Tavus output layer + transport.output(), # Transport bot output + context_aggregator.assistant(), # Assistant spoken responses + ] + ) + + task = PipelineTask( + pipeline, + params=PipelineParams( + audio_in_sample_rate=16000, + audio_out_sample_rate=24000, + allow_interruptions=True, + enable_metrics=True, + enable_usage_metrics=True, + report_only_initial_ttfb=True, + ), + ) + + @transport.event_handler("on_client_connected") + async def on_client_connected(transport, client): + logger.info(f"Client connected") + # Kick off the conversation. + messages.append( + { + "role": "system", + "content": "Start by greeting the user and ask how you can help.", + } + ) + await task.queue_frames([context_aggregator.user().get_context_frame()]) + + @transport.event_handler("on_client_disconnected") + async def on_client_disconnected(transport, client): + logger.info(f"Client disconnected") + + @transport.event_handler("on_client_closed") + async def on_client_closed(transport, client): + logger.info(f"Client closed connection") + await task.cancel() + + runner = PipelineRunner(handle_sigint=False) + + await runner.run(task) + + +if __name__ == "__main__": + from run import main + + main() diff --git a/examples/foundational/21-tavus-layer.py b/examples/foundational/21b-tavus-layer-daily-transport.py similarity index 64% rename from examples/foundational/21-tavus-layer.py rename to examples/foundational/21b-tavus-layer-daily-transport.py index ffe95e074..564828136 100644 --- a/examples/foundational/21-tavus-layer.py +++ b/examples/foundational/21b-tavus-layer-daily-transport.py @@ -7,9 +7,9 @@ import asyncio import os import sys -from typing import Any, Mapping import aiohttp +from daily_runner import configure from dotenv import load_dotenv from loguru import logger @@ -20,7 +20,7 @@ from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.services.deepgram.stt import DeepgramSTTService -from pipecat.services.openai.llm import OpenAILLMService +from pipecat.services.google.llm import GoogleLLMService from pipecat.services.tavus.video import TavusVideoService from pipecat.transports.services.daily import DailyParams, DailyTransport @@ -32,23 +32,20 @@ logger.add(sys.stderr, level="DEBUG") async def main(): async with aiohttp.ClientSession() as session: - tavus = TavusVideoService( - api_key=os.getenv("TAVUS_API_KEY"), - replica_id=os.getenv("TAVUS_REPLICA_ID"), - session=session, - ) - - # get persona, look up persona_name, set this as the bot name to ignore - persona_name = await tavus.get_persona_name() - room_url = await tavus.initialize() + (room_url, token) = await configure(session) transport = DailyTransport( - room_url=room_url, - token=None, - bot_name="Pipecat bot", - params=DailyParams( + room_url, + token, + "Pipecat bot", + DailyParams( audio_in_enabled=True, + audio_out_enabled=True, + video_out_enabled=True, + video_out_is_live=True, vad_analyzer=SileroVADAnalyzer(), + video_out_width=1280, + video_out_height=720, ), ) @@ -59,7 +56,13 @@ async def main(): voice_id="a167e0f3-df7e-4d52-a9c3-f949145efdab", ) - llm = OpenAILLMService(model="gpt-4o-mini") + llm = GoogleLLMService(api_key=os.getenv("GOOGLE_API_KEY")) + + tavus = TavusVideoService( + api_key=os.getenv("TAVUS_API_KEY"), + replica_id=os.getenv("TAVUS_REPLICA_ID"), + session=session, + ) messages = [ { @@ -87,10 +90,8 @@ async def main(): task = PipelineTask( pipeline, params=PipelineParams( - # We just use 16000 because that's what Tavus is expecting and - # we avoid resampling. audio_in_sample_rate=16000, - audio_out_sample_rate=16000, + audio_out_sample_rate=24000, allow_interruptions=True, enable_metrics=True, enable_usage_metrics=True, @@ -98,33 +99,22 @@ async def main(): ), ) - @transport.event_handler("on_participant_joined") - async def on_participant_joined( - transport: DailyTransport, participant: Mapping[str, Any] - ) -> None: - # Ignore the Tavus replica's microphone - if participant.get("info", {}).get("userName", "") == persona_name: - logger.debug(f"Ignoring {participant['id']}'s microphone") - await transport.update_subscriptions( - participant_settings={ - participant["id"]: { - "media": {"microphone": "unsubscribed"}, - } - } - ) - - if participant.get("info", {}).get("userName", "") != persona_name: - # Kick off the conversation. - messages.append( - {"role": "system", "content": "Please introduce yourself to the user."} - ) - await task.queue_frames([context_aggregator.user().get_context_frame()]) + @transport.event_handler("on_first_participant_joined") + async def on_first_participant_joined(transport, participant): + # Kick off the conversation. + messages.append( + { + "role": "system", + "content": "Start by greeting the user and ask how you can help.", + } + ) + await task.queue_frames([context_aggregator.user().get_context_frame()]) @transport.event_handler("on_participant_left") async def on_participant_left(transport, participant, reason): await task.cancel() - runner = PipelineRunner() + runner = PipelineRunner(handle_sigint=False) await runner.run(task) diff --git a/examples/foundational/29-turn-tracking-observer.py b/examples/foundational/29-turn-tracking-observer.py index 0e9c7acec..08c39fe55 100644 --- a/examples/foundational/29-turn-tracking-observer.py +++ b/examples/foundational/29-turn-tracking-observer.py @@ -11,6 +11,7 @@ from dotenv import load_dotenv from loguru import logger from pipecat.audio.vad.silero import SileroVADAnalyzer +from pipecat.observers.loggers.user_bot_latency_log_observer import UserBotLatencyLogObserver from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask @@ -76,6 +77,7 @@ async def run_bot(webrtc_connection: SmallWebRTCConnection, _: argparse.Namespac enable_usage_metrics=True, report_only_initial_ttfb=True, ), + observers=[UserBotLatencyLogObserver()], ) turn_observer = task.turn_tracking_observer diff --git a/examples/foundational/README.md b/examples/foundational/README.md index 14323d189..3c77a7bef 100644 --- a/examples/foundational/README.md +++ b/examples/foundational/README.md @@ -95,7 +95,7 @@ Depending on what you're trying to build, these learning paths will guide you th - **[18-gstreamer-filesrc.py](./18-gstreamer-filesrc.py)**: GStreamer video streaming (Video processing) - **[19-openai-realtime-beta.py](./19-openai-realtime-beta.py)**: OpenAI Speech-to-Speech (Direct S2S, Function calls) -- **[21-tavus-layer.py](./21-tavus-layer.py)**: Tavus digital twin (Avatar integration) +- **[21-tavus-layer-tavus-transport.py](./21-tavus-layer-tavus-transport.py)**: Tavus digital twin (Avatar integration) - **[27-simli-layer.py](./27-simli-layer.py)**: Simli avatar integration (Video synchronization) ### Performance & Optimization diff --git a/examples/open-telemetry/jaeger/README.md b/examples/open-telemetry/jaeger/README.md index a19d3ee9d..57a9bb7d3 100644 --- a/examples/open-telemetry/jaeger/README.md +++ b/examples/open-telemetry/jaeger/README.md @@ -38,6 +38,8 @@ OPENAI_API_KEY=your_key_here pip install -r requirements.txt ``` +> Install only the grpc exporter. If you have a conflict, uninstall the http exporter. + ### 4. Run the Demo ```bash diff --git a/examples/open-telemetry/langfuse/README.md b/examples/open-telemetry/langfuse/README.md index 002a3f64a..c17de8394 100644 --- a/examples/open-telemetry/langfuse/README.md +++ b/examples/open-telemetry/langfuse/README.md @@ -43,6 +43,8 @@ OPENAI_API_KEY=your_key_here pip install -r requirements.txt ``` +> Install only the http exporter. If you have a conflict, uninstall the grpc exporter. + ### 4. Run the Demo ```bash diff --git a/examples/open-telemetry/langfuse/env.example b/examples/open-telemetry/langfuse/env.example index c86485674..622854799 100644 --- a/examples/open-telemetry/langfuse/env.example +++ b/examples/open-telemetry/langfuse/env.example @@ -4,8 +4,15 @@ OPENAI_API_KEY=your_openai_key # Set to any value to enable tracing ENABLE_TRACING=true -# OTLP endpoint (change to us.cloud.langfuse.com if you use the US data region) -OTEL_EXPORTER_OTLP_ENDPOINT=http://cloud.langfuse.com/api/public/otel -OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic%20 + +# πŸ‡ͺπŸ‡Ί EU data region +OTEL_EXPORTER_OTLP_ENDPOINT="https://cloud.langfuse.com/api/public/otel" +# πŸ‡ΊπŸ‡Έ US data region +# OTEL_EXPORTER_OTLP_ENDPOINT="https://us.cloud.langfuse.com/api/public/otel" +# 🏠 Local deployment (>= v3.22.0) +# OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:3000/api/public/otel" + +OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic " + # Set to any value to enable console output for debugging # OTEL_CONSOLE_EXPORT=true \ No newline at end of file diff --git a/examples/plivo-chatbot/.gitignore b/examples/plivo-chatbot/.gitignore new file mode 100644 index 000000000..2bc1403d1 --- /dev/null +++ b/examples/plivo-chatbot/.gitignore @@ -0,0 +1,161 @@ +# Byte-compiled / optimized / DLL files +__pycache__/ +*.py[cod] +*$py.class + +# C extensions +*.so + +# Distribution / packaging +.Python +build/ +develop-eggs/ +dist/ +downloads/ +eggs/ +.eggs/ +lib/ +lib64/ +parts/ +sdist/ +var/ +wheels/ +share/python-wheels/ +*.egg-info/ +.installed.cfg +*.egg +MANIFEST + +# PyInstaller +# Usually these files are written by a python script from a template +# before PyInstaller builds the exe, so as to inject date/other infos into it. +*.manifest +*.spec + +# Installer logs +pip-log.txt +pip-delete-this-directory.txt + +# Unit test / coverage reports +htmlcov/ +.tox/ +.nox/ +.coverage +.coverage.* +.cache +nosetests.xml +coverage.xml +*.cover +*.py,cover +.hypothesis/ +.pytest_cache/ +cover/ + +# Translations +*.mo +*.pot + +# Django stuff: +*.log +local_settings.py +db.sqlite3 +db.sqlite3-journal + +# Flask stuff: +instance/ +.webassets-cache + +# Scrapy stuff: +.scrapy + +# Sphinx documentation +docs/_build/ + +# PyBuilder +.pybuilder/ +target/ + +# Jupyter Notebook +.ipynb_checkpoints + +# IPython +profile_default/ +ipython_config.py + +# pyenv +# For a library or package, you might want to ignore these files since the code is +# intended to run in multiple environments; otherwise, check them in: +# .python-version + +# pipenv +# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. +# However, in case of collaboration, if having platform-specific dependencies or dependencies +# having no cross-platform support, pipenv may install dependencies that don't work, or not +# install all needed dependencies. +#Pipfile.lock + +# poetry +# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. +# This is especially recommended for binary packages to ensure reproducibility, and is more +# commonly ignored for libraries. +# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control +#poetry.lock + +# pdm +# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. +#pdm.lock +# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it +# in version control. +# https://pdm.fming.dev/#use-with-ide +.pdm.toml + +# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm +__pypackages__/ + +# Celery stuff +celerybeat-schedule +celerybeat.pid + +# SageMath parsed files +*.sage.py + +# Environments +.env +.venv +env/ +venv/ +ENV/ +env.bak/ +venv.bak/ + +# Spyder project settings +.spyderproject +.spyproject + +# Rope project settings +.ropeproject + +# mkdocs documentation +/site + +# mypy +.mypy_cache/ +.dmypy.json +dmypy.json + +# Pyre type checker +.pyre/ + +# pytype static type analyzer +.pytype/ + +# Cython debug symbols +cython_debug/ + +# PyCharm +# JetBrains specific template is maintained in a separate JetBrains.gitignore that can +# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore +# and can be added to the global gitignore or merged into this file. For a more nuclear +# option (not recommended) you can uncomment the following to ignore the entire idea folder. +#.idea/ +runpod.toml diff --git a/examples/plivo-chatbot/Dockerfile b/examples/plivo-chatbot/Dockerfile new file mode 100644 index 000000000..4f7a1960c --- /dev/null +++ b/examples/plivo-chatbot/Dockerfile @@ -0,0 +1,20 @@ +# Use an official Python runtime as a parent image +FROM python:3.10-bullseye + +# Set the working directory in the container +WORKDIR /plivo-chatbot + +# Copy the requirements file into the container +COPY requirements.txt . + +# Install any needed packages specified in requirements.txt +RUN pip install --no-cache-dir -r requirements.txt + +# Copy the current directory contents into the container +COPY . . + +# Expose the desired port +EXPOSE 8765 + +# Run the application +CMD ["uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8765"] diff --git a/examples/plivo-chatbot/README.md b/examples/plivo-chatbot/README.md new file mode 100644 index 000000000..f8d12e4b9 --- /dev/null +++ b/examples/plivo-chatbot/README.md @@ -0,0 +1,128 @@ +# Plivo Chatbot + +This project is a FastAPI-based chatbot that integrates with Plivo to handle WebSocket connections and provide real-time communication. The project includes endpoints for starting a call and handling WebSocket connections. + +## Table of Contents + +- [Features](#features) +- [Requirements](#requirements) +- [Installation](#installation) +- [Configure Plivo URLs](#configure-plivo-urls) +- [Running the Application](#running-the-application) +- [Usage](#usage) + +## Features + +- **FastAPI**: A modern, fast (high-performance), web framework for building APIs with Python 3.6+. +- **WebSocket Support**: Real-time communication using WebSockets. +- **CORS Middleware**: Allowing cross-origin requests for testing. +- **Dockerized**: Easily deployable using Docker. + +## Requirements + +- Python 3.10 +- Docker (for containerized deployment) +- ngrok (for tunneling) +- Plivo Account + +## Installation + +1. **Set up a virtual environment** (optional but recommended): + + ```sh + python -m venv venv + source venv/bin/activate # On Windows, use `venv\Scripts\activate` + ``` + +2. **Install dependencies**: + + ```sh + pip install -r requirements.txt + ``` + +3. **Create .env**: + Copy the example environment file and update with your settings: + + ```sh + cp env.example .env + ``` + +4. **Install ngrok**: + Follow the instructions on the [ngrok website](https://ngrok.com/download) to download and install ngrok. + +## Configure Plivo URLs + +1. **Start ngrok**: + In a new terminal, start ngrok to tunnel the local server: + + ```sh + ngrok http 8765 + ``` + +2. **Update the Plivo Application**: + + - Go to your Plivo console and navigate to Voice > Applications > XML + - Select "Add New Application" or edit an existing one + - Set the Primary Answer URL to your ngrok URL (e.g., https:///) + - Ensure the Answer Method is set to POST + - Save the application + - Configure your number to use the newly created (or updated) application + - Phone Numbers > Active > Your number + - Select Application Type: XML Application + - Plivo Application: Your application + - Click "Update" to save + +3. **Configure streams.xml**: + + - Copy the template file to create your local version: + ```sh + cp templates/streams.xml.template templates/streams.xml + ``` + - In `templates/streams.xml`, replace `` with your ngrok URL (without `https://`) + - The final URL should look like: `wss://abc123.ngrok.io/ws` + +4. **Assign the Application to a Plivo Number**: + - Go to Phone Numbers > Your Numbers in the Plivo console + - Edit your Plivo number + - Select the application you created/updated in the previous step + - Save the configuration + +## Running the Application + +Choose one of these two methods to run the application: + +### Using Python (Option 1) + +**Run the FastAPI application**: + +```sh +# Make sure you're in the project directory and your virtual environment is activated +python server.py +``` + +### Using Docker (Option 2) + +1. **Build the Docker image**: + + ```sh + docker build -t plivo-chatbot . + ``` + +2. **Run the Docker container**: + ```sh + docker run -it --rm -p 8765:8765 plivo-chatbot + ``` + +The server will start on port 8765. Keep this running while you test with Plivo. + +## Usage + +To start a call, simply make a call to your configured Plivo phone number. The Answer URL will direct the call to your FastAPI application, which will handle it accordingly. + +## Key Differences from Twilio + +- Plivo uses `streamId` instead of `streamSid` +- Plivo uses `callId` instead of `callSid` +- Plivo uses `` element instead of `` +- Plivo's Stream element has `bidirectional`, `keepCallAlive`, and `contentType` attributes +- Plivo API authentication uses Auth ID and Auth Token (similar to Twilio's Account SID and Auth Token) diff --git a/examples/plivo-chatbot/bot.py b/examples/plivo-chatbot/bot.py new file mode 100644 index 000000000..5ce4be0c6 --- /dev/null +++ b/examples/plivo-chatbot/bot.py @@ -0,0 +1,111 @@ +# +# Copyright (c) 2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +import os +import sys +from typing import Optional + +from dotenv import load_dotenv +from fastapi import WebSocket +from loguru import logger + +from pipecat.audio.vad.silero import SileroVADAnalyzer +from pipecat.pipeline.pipeline import Pipeline +from pipecat.pipeline.runner import PipelineRunner +from pipecat.pipeline.task import PipelineParams, PipelineTask +from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext +from pipecat.serializers.plivo import PlivoFrameSerializer +from pipecat.services.cartesia import CartesiaTTSService +from pipecat.services.deepgram.stt import DeepgramSTTService +from pipecat.services.openai import OpenAILLMService +from pipecat.transports.network.fastapi_websocket import ( + FastAPIWebsocketParams, + FastAPIWebsocketTransport, +) + +load_dotenv() +logger.remove(0) +logger.add(sys.stderr, level="DEBUG") + + +async def run_bot(websocket_client: WebSocket, stream_id: str, call_id: Optional[str]): + logger.info(f"Starting bot for stream: {stream_id}") + + serializer = PlivoFrameSerializer( + stream_id=stream_id, + call_id=call_id, + auth_id=os.getenv("PLIVO_AUTH_ID"), + auth_token=os.getenv("PLIVO_AUTH_TOKEN"), + ) + + transport = FastAPIWebsocketTransport( + websocket=websocket_client, + params=FastAPIWebsocketParams( + audio_in_enabled=True, + audio_out_enabled=True, + add_wav_header=False, + vad_analyzer=SileroVADAnalyzer(), + serializer=serializer, + ), + ) + + llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY")) + + stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY")) + + tts = CartesiaTTSService( + api_key=os.getenv("CARTESIA_API_KEY"), + voice_id="71a7ad14-091c-4e8e-a314-022ece01c121", # British Reading Lady + ) + + messages = [ + { + "role": "system", + "content": "You are an elementary teacher in an audio call. Your output will be converted to audio so don't include special characters in your answers. Respond to what the student said in a short short sentence.", + }, + ] + + context = OpenAILLMContext(messages) + context_aggregator = llm.create_context_aggregator(context) + + pipeline = Pipeline( + [ + transport.input(), # Websocket input from client + stt, # Speech-To-Text + context_aggregator.user(), + llm, # LLM + tts, # Text-To-Speech + transport.output(), # Websocket output to client + context_aggregator.assistant(), + ] + ) + + task = PipelineTask( + pipeline, + params=PipelineParams( + audio_in_sample_rate=8000, + audio_out_sample_rate=8000, + allow_interruptions=True, + ), + ) + + @transport.event_handler("on_client_connected") + async def on_client_connected(transport, client): + # Kick off the conversation. + messages.append({"role": "system", "content": "Please introduce yourself to the user."}) + await task.queue_frames([context_aggregator.user().get_context_frame()]) + + @transport.event_handler("on_client_disconnected") + async def on_client_disconnected(transport, client): + await task.cancel() + + # We use `handle_sigint=False` because `uvicorn` is controlling keyboard + # interruptions. We use `force_gc=True` to force garbage collection after + # the runner finishes running a task which could be useful for long running + # applications with multiple clients connecting. + runner = PipelineRunner(handle_sigint=False, force_gc=True) + + await runner.run(task) diff --git a/examples/plivo-chatbot/env.example b/examples/plivo-chatbot/env.example new file mode 100644 index 000000000..7e9278758 --- /dev/null +++ b/examples/plivo-chatbot/env.example @@ -0,0 +1,5 @@ +OPENAI_API_KEY= +DEEPGRAM_API_KEY= +CARTESIA_API_KEY= +PLIVO_AUTH_ID= +PLIVO_AUTH_TOKEN= \ No newline at end of file diff --git a/examples/plivo-chatbot/requirements.txt b/examples/plivo-chatbot/requirements.txt new file mode 100644 index 000000000..ff134bae3 --- /dev/null +++ b/examples/plivo-chatbot/requirements.txt @@ -0,0 +1,5 @@ +pipecat-ai[cartesia,openai,silero,deepgram] +fastapi +uvicorn +python-dotenv +loguru \ No newline at end of file diff --git a/examples/plivo-chatbot/server.py b/examples/plivo-chatbot/server.py new file mode 100644 index 000000000..b6d6ebd10 --- /dev/null +++ b/examples/plivo-chatbot/server.py @@ -0,0 +1,59 @@ +# +# Copyright (c) 2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +import argparse +import json + +import uvicorn +from bot import run_bot +from fastapi import FastAPI, WebSocket +from fastapi.middleware.cors import CORSMiddleware +from loguru import logger +from starlette.responses import HTMLResponse + +app = FastAPI() + +app.add_middleware( + CORSMiddleware, + allow_origins=["*"], # Allow all origins for testing + allow_credentials=True, + allow_methods=["*"], + allow_headers=["*"], +) + + +@app.post("/") +async def start_call(): + print("POST Plivo XML") + return HTMLResponse(content=open("templates/streams.xml").read(), media_type="application/xml") + + +@app.websocket("/ws") +async def websocket_endpoint(websocket: WebSocket): + await websocket.accept() + + # Plivo sends a start event when the stream begins + start_data = websocket.iter_text() + start_message = json.loads(await start_data.__anext__()) + + print("Received start message:", start_message, flush=True) + + # Extract stream_id and call_id from the start event + start_info = start_message.get("start", {}) + stream_id = start_info.get("streamId") + call_id = start_info.get("callId") + + if not stream_id: + logger.error("No streamId found in start message") + await websocket.close() + return + + print(f"WebSocket connection accepted for stream: {stream_id}, call: {call_id}") + await run_bot(websocket, stream_id, call_id) + + +if __name__ == "__main__": + uvicorn.run(app, host="0.0.0.0", port=8765) diff --git a/examples/plivo-chatbot/templates/streams.xml.template b/examples/plivo-chatbot/templates/streams.xml.template new file mode 100644 index 000000000..8f657672d --- /dev/null +++ b/examples/plivo-chatbot/templates/streams.xml.template @@ -0,0 +1,4 @@ + + + wss:///ws + \ No newline at end of file diff --git a/examples/websocket-server/index.html b/examples/websocket-server/index.html index ac93c62d7..6f0bd1ada 100644 --- a/examples/websocket-server/index.html +++ b/examples/websocket-server/index.html @@ -10,7 +10,7 @@

Pipecat WebSocket Client Example

-

Loading, wait...

+

Loading, wait...