From 3ab9c155833631a72728ca7365541637350f5515 Mon Sep 17 00:00:00 2001 From: Mark Backman Date: Tue, 29 Jul 2025 18:13:07 -0400 Subject: [PATCH] Add README to client-server-web, add phone-bot-twilio files --- examples/client-server-web/README.md | 111 ++++++++++++ examples/client-server-web/server/README.md | 90 ---------- examples/phone-bot-twilio/.gitignore | 164 ++++++++++++++++++ examples/phone-bot-twilio/README.md | 115 ++++++++++++ examples/phone-bot-twilio/bot.py | 121 +++++++++++++ examples/phone-bot-twilio/env.example | 3 + examples/phone-bot-twilio/requirements.txt | 1 + .../templates/streams.xml.template | 7 + examples/quickstart/README.md | 5 +- 9 files changed, 524 insertions(+), 93 deletions(-) create mode 100644 examples/client-server-web/README.md delete mode 100644 examples/client-server-web/server/README.md create mode 100644 examples/phone-bot-twilio/.gitignore create mode 100644 examples/phone-bot-twilio/README.md create mode 100644 examples/phone-bot-twilio/bot.py create mode 100644 examples/phone-bot-twilio/env.example create mode 100644 examples/phone-bot-twilio/requirements.txt create mode 100644 examples/phone-bot-twilio/templates/streams.xml.template diff --git a/examples/client-server-web/README.md b/examples/client-server-web/README.md new file mode 100644 index 000000000..aa6318334 --- /dev/null +++ b/examples/client-server-web/README.md @@ -0,0 +1,111 @@ +# Client Server Web Example + +Learn how to build web applications using Pipecat's client/server architecture. This approach separates your bot logic from your user interface, giving you full control over the client experience while maintaining real-time voice communication. + +This example demonstrates: + +- Server-side bot running with Pipecat +- React client using [Pipecat's client SDK](https://docs.pipecat.ai/client/introduction) +- Real-time voice communication between client and server +- UI components from [voice-ui-kit](https://github.com/pipecat-ai/voice-ui-kit) for common voice interface patterns + +This is the recommended architecture for web applications that need custom interfaces or client-side functionality. + +## Prerequisites + +- Python 3.10+ +- `npm` installed +- AI Service API keys for: [Deepgram](https://console.deepgram.com/signup), [OpenAI](https://auth.openai.com/create-account), and [Cartesia](https://play.cartesia.ai/sign-up) + +## Setup + +This example requires running both a server and client in **two separate terminal windows**. + +### Terminal 1: Server Setup + +1. Set up a virtual environment + +From the `examples/client-server-web` directory, run: + +```bash +cd server +python -m venv .venv +source .venv/bin/activate # On Windows: .venv\Scripts\activate +``` + +> Using `uv`? Create your venv using: `uv venv && source .venv/bin/activate`. + +2. Install packages + +Then, install the requirements: + +```bash +pip install -r requirements.txt +``` + +> Using `uv`? Install requirements using: `uv pip install -r requirements.txt`. + +3. Configure environment variables + +Create a `.env` file: + +```bash +cp env.example .env +``` + +Then, add your API keys: + +``` +DEEPGRAM_API_KEY=your_deepgram_api_key +OPENAI_API_KEY=your_openai_api_key +CARTESIA_API_KEY=your_cartesia_api_key +``` + +4. Run the example + +Run your bot using: + +```bash +python bot.py +``` + +> Using `uv`? Run your bot using: `uv run bot.py`. + +> 💡 First run note: The initial startup may take ~10 seconds as Pipecat downloads required models, like the Silero VAD model. + +### Terminal 2: Client Setup + +1. Open a new terminal window and navigate to the `client` folder: + +From the `examples/client-server-web` directory, run: + +```bash +cd client +``` + +2. Install dependencies: + +```bash +npm i +``` + +3. Run the client: + +```bash +npm run dev +``` + +4. **Open http://localhost:5173 in your browser** and click `Connect` to start talking to your bot. + +## Troubleshooting + +- **Browser permissions**: Make sure to allow microphone access when prompted by your browser. +- **Connection issues**: If the WebRTC connection fails, first try a different browser. If that fails, make sure you don't have a VPN or firewall rules blocking traffic. WebRTC uses UDP to communicate. +- **Audio issues**: Check that your microphone and speakers are working and not muted. + +## Next Steps + +- **Explore the client SDK**: Learn more about [Pipecat's client SDKs](https://docs.pipecat.ai/client/introduction) for web, mobile, and other platforms +- **Learn about the voice-ui-kit**: Explore [voice-ui-kit](https://github.com/pipecat-ai/voice-ui-kit) to simplify your front end development +- **Advanced examples**: Check out [pipecat-examples](https://github.com/pipecat-ai/pipecat-examples) for more complex client/server applications +- **Join Discord**: Connect with other developers on [Discord](https://discord.gg/pipecat) diff --git a/examples/client-server-web/server/README.md b/examples/client-server-web/server/README.md deleted file mode 100644 index a05964812..000000000 --- a/examples/client-server-web/server/README.md +++ /dev/null @@ -1,90 +0,0 @@ -# Pipecat Quickstart - -Run your first Pipecat bot in under 5 minutes. This example creates a voice AI bot that you can talk to in your browser. - -## Prerequisites - -### Python 3.10+ - -Pipecat requires Python 3.10 or newer. Check your version: - -```bash -python --version -``` - -If you need to upgrade Python, we recommend using a version manager like `uv` or `pyenv`. - -### AI Service API keys - -Pipecat orchestrates different AI services in a pipeline, ensuring low latency communication. In this quickstart example, we'll use: - -- [Deepgram](https://console.deepgram.com/signup) for Speech-to-Text transcriptions -- [OpenAI](https://auth.openai.com/create-account) for LLM inference -- [Cartesia](https://play.cartesia.ai/sign-up) for Text-to-Speech audio generation - -Have your API keys ready. We'll add them to your `.env` shortly. - -## Setup - -1. Set up a virtual environment - -From the root directory of the `pipecat` repo, run: - -```bash -cd examples/quickstart -python -m venv .venv -source .venv/bin/activate # On Windows: .venv\Scripts\activate -``` - -> Using `uv`? Create your venv using: `uv venv && source .venv/bin/activate`. - -2. Install packages - -Then, install the requirements: - -```bash -pip install -r requirements.txt -``` - -> Using `uv`? Install requirements using: `uv pip install -r requirements.txt`. - -3. Configure environment variables - -Create a `.env` file: - -```bash -cp env.example .env -``` - -Then, add your API keys: - -``` -DEEPGRAM_API_KEY=your_deepgram_api_key -OPENAI_API_KEY=your_openai_api_key -CARTESIA_API_KEY=your_cartesia_api_key -``` - -4. Run the example - -Run your bot using: - -```bash -python bot.py -``` - -> Using `uv`? Run your bot using: `uv run bot.py`. - -Connect to your bot in a browser at http://localhost:7860. - -> 💡 First run note: The initial startup may take ~10 seconds as Pipecat downloads required models, like the Silero VAD model. - -## Troubleshooting - -- **Browser permissions**: Make sure to allow microphone access when prompted by your browser. -- **Connection issues**: If the WebRTC connection fails, first try a different browser. If that fails, make sure you don't have a VPN or firewall rules blocking traffic. WebRTC uses UDP to communicate. -- **Audio issues**: Check that your microphone and speakers are working and not muted. - -## Next Steps - -- **Read the docs**: Check out [Pipecat's docs](https://docs.pipecat.ai/) for guides and reference information. -- **Join Discord**: Join [Pipecat's Discord server](https://discord.gg/pipecat) to get help and learn about what others are building. diff --git a/examples/phone-bot-twilio/.gitignore b/examples/phone-bot-twilio/.gitignore new file mode 100644 index 000000000..e68f56694 --- /dev/null +++ b/examples/phone-bot-twilio/.gitignore @@ -0,0 +1,164 @@ +# Byte-compiled / optimized / DLL files +__pycache__/ +*.py[cod] +*$py.class + +# C extensions +*.so + +# Distribution / packaging +.Python +build/ +develop-eggs/ +dist/ +downloads/ +eggs/ +.eggs/ +lib/ +lib64/ +parts/ +sdist/ +var/ +wheels/ +share/python-wheels/ +*.egg-info/ +.installed.cfg +*.egg +MANIFEST + +# PyInstaller +# Usually these files are written by a python script from a template +# before PyInstaller builds the exe, so as to inject date/other infos into it. +*.manifest +*.spec + +# Installer logs +pip-log.txt +pip-delete-this-directory.txt + +# Unit test / coverage reports +htmlcov/ +.tox/ +.nox/ +.coverage +.coverage.* +.cache +nosetests.xml +coverage.xml +*.cover +*.py,cover +.hypothesis/ +.pytest_cache/ +cover/ + +# Translations +*.mo +*.pot + +# Django stuff: +*.log +local_settings.py +db.sqlite3 +db.sqlite3-journal + +# Flask stuff: +instance/ +.webassets-cache + +# Scrapy stuff: +.scrapy + +# Sphinx documentation +docs/_build/ + +# PyBuilder +.pybuilder/ +target/ + +# Jupyter Notebook +.ipynb_checkpoints + +# IPython +profile_default/ +ipython_config.py + +# pyenv +# For a library or package, you might want to ignore these files since the code is +# intended to run in multiple environments; otherwise, check them in: +# .python-version + +# pipenv +# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. +# However, in case of collaboration, if having platform-specific dependencies or dependencies +# having no cross-platform support, pipenv may install dependencies that don't work, or not +# install all needed dependencies. +#Pipfile.lock + +# poetry +# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. +# This is especially recommended for binary packages to ensure reproducibility, and is more +# commonly ignored for libraries. +# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control +#poetry.lock + +# pdm +# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. +#pdm.lock +# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it +# in version control. +# https://pdm.fming.dev/#use-with-ide +.pdm.toml + +# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm +__pypackages__/ + +# Celery stuff +celerybeat-schedule +celerybeat.pid + +# SageMath parsed files +*.sage.py + +# Environments +.env +.venv +env/ +venv/ +ENV/ +env.bak/ +venv.bak/ + +# Spyder project settings +.spyderproject +.spyproject + +# Rope project settings +.ropeproject + +# mkdocs documentation +/site + +# mypy +.mypy_cache/ +.dmypy.json +dmypy.json + +# Pyre type checker +.pyre/ + +# pytype static type analyzer +.pytype/ + +# Cython debug symbols +cython_debug/ + +# PyCharm +# JetBrains specific template is maintained in a separate JetBrains.gitignore that can +# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore +# and can be added to the global gitignore or merged into this file. For a more nuclear +# option (not recommended) you can uncomment the following to ignore the entire idea folder. +#.idea/ +runpod.toml + +# Examples +templates/streams.xml \ No newline at end of file diff --git a/examples/phone-bot-twilio/README.md b/examples/phone-bot-twilio/README.md new file mode 100644 index 000000000..9c7a4be95 --- /dev/null +++ b/examples/phone-bot-twilio/README.md @@ -0,0 +1,115 @@ +# Twilio Chatbot + +This project is a FastAPI-based chatbot that integrates with Twilio to handle WebSocket connections and provide real-time communication. The project includes endpoints for starting a call and handling WebSocket connections. + +## Table of Contents + +- [Features](#features) +- [Requirements](#requirements) +- [Installation](#installation) +- [Configure Twilio URLs](#configure-twilio-urls) +- [Running the Application](#running-the-application) +- [Usage](#usage) + +## Features + +- **FastAPI**: A modern, fast (high-performance), web framework for building APIs with Python 3.6+. +- **WebSocket Support**: Real-time communication using WebSockets. +- **CORS Middleware**: Allowing cross-origin requests for testing. +- **Dockerized**: Easily deployable using Docker. + +## Requirements + +- Python 3.10 +- Docker (for containerized deployment) +- ngrok (for tunneling) +- Twilio Account + +## Installation + +1. **Set up a virtual environment** (optional but recommended): + + ```sh + python -m venv venv + source venv/bin/activate # On Windows, use `venv\Scripts\activate` + ``` + +2. **Install dependencies**: + + ```sh + pip install -r requirements.txt + ``` + +3. **Create .env**: + Copy the example environment file and update with your settings: + + ```sh + cp env.example .env + ``` + +4. **Install ngrok**: + Follow the instructions on the [ngrok website](https://ngrok.com/download) to download and install ngrok. + +## Configure Twilio URLs + +1. **Start ngrok**: + In a new terminal, start ngrok to tunnel the local server: + + ```sh + ngrok http 8765 + ``` + +2. **Update the Twilio Webhook**: + + - Go to your Twilio phone number's configuration page + - Under "Voice Configuration", in the "A call comes in" section: + - Select "Webhook" from the dropdown + - Enter your ngrok URL (e.g., http://) + - Ensure "HTTP POST" is selected + - Click Save at the bottom of the page + +3. **Configure streams.xml**: + - Copy the template file to create your local version: + ```sh + cp templates/streams.xml.template templates/streams.xml + ``` + - In `templates/streams.xml`, replace `` with your ngrok URL (without `https://`) + - The final URL should look like: `wss://abc123.ngrok.io/ws` + +## Running the Application + +Choose one of these two methods to run the application: + +### Using Python (Option 1) + +**Run the FastAPI application**: + +```sh +# Make sure you’re in the project directory and your virtual environment is activated +python server.py +``` + +### Using Docker (Option 2) + +1. **Build the Docker image**: + + ```sh + docker build -t twilio-chatbot . + ``` + +2. **Run the Docker container**: + ```sh + docker run -it --rm -p 8765:8765 twilio-chatbot + ``` + +The server will start on port 8765. Keep this running while you test with Twilio. + +## Usage + +To start a call, simply make a call to your configured Twilio phone number. The webhook URL will direct the call to your FastAPI application, which will handle it accordingly. + +## Testing + +It is also possible to test the server without making phone calls by using one of these clients. +- [python](client/python/README.md): This Python client enables automated testing of the server via WebSocket without the need to make actual phone calls. +- [typescript](client/typescript/README.md): This typescript client enables manual testing of the server via WebSocket without the need to make actual phone calls. \ No newline at end of file diff --git a/examples/phone-bot-twilio/bot.py b/examples/phone-bot-twilio/bot.py new file mode 100644 index 000000000..3417500bb --- /dev/null +++ b/examples/phone-bot-twilio/bot.py @@ -0,0 +1,121 @@ +# +# Copyright (c) 2024–2025, Daily +# +# SPDX-License-Identifier: BSD 2-Clause License +# + +"""Pipecat Twilio Phone Example. + +The example runs a simple voice AI bot that you can connect to using a +phone via Twilio. + +Required AI services: +- Deepgram (Speech-to-Text) +- OpenAI (LLM) +- Cartesia (Text-to-Speech) + +The example connects between client and server using a Twilio websocket +connection. + +Run the bot using:: + + python bot.py -t twilio +""" + +import argparse +import os + +from dotenv import load_dotenv +from loguru import logger + +from pipecat.audio.vad.silero import SileroVADAnalyzer +from pipecat.pipeline.pipeline import Pipeline +from pipecat.pipeline.runner import PipelineRunner +from pipecat.pipeline.task import PipelineParams, PipelineTask +from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext +from pipecat.processors.frameworks.rtvi import RTVIConfig, RTVIObserver, RTVIProcessor +from pipecat.services.cartesia.tts import CartesiaTTSService +from pipecat.services.deepgram.stt import DeepgramSTTService +from pipecat.services.openai.llm import OpenAILLMService +from pipecat.transports.base_transport import BaseTransport +from pipecat.transports.network.fastapi_websocket import FastAPIWebsocketParams + +load_dotenv(override=True) + + +async def run_bot(transport: BaseTransport, _: argparse.Namespace, handle_sigint: bool): + logger.info(f"Starting bot") + + stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY")) + + tts = CartesiaTTSService( + api_key=os.getenv("CARTESIA_API_KEY"), + voice_id="71a7ad14-091c-4e8e-a314-022ece01c121", # British Reading Lady + ) + + llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY")) + + messages = [ + { + "role": "system", + "content": "You are a friendly AI assistant. Respond naturally and keep your answers conversational.", + }, + ] + + context = OpenAILLMContext(messages) + context_aggregator = llm.create_context_aggregator(context) + + rtvi = RTVIProcessor(config=RTVIConfig(config=[])) + + pipeline = Pipeline( + [ + transport.input(), # Transport user input + rtvi, # RTVI processor + stt, + context_aggregator.user(), # User responses + llm, # LLM + tts, # TTS + transport.output(), # Transport bot output + context_aggregator.assistant(), # Assistant spoken responses + ] + ) + + task = PipelineTask( + pipeline, + params=PipelineParams( + enable_metrics=True, + enable_usage_metrics=True, + ), + observers=[RTVIObserver(rtvi)], + ) + + @transport.event_handler("on_client_connected") + async def on_client_connected(transport, client): + logger.info(f"Client connected") + # Kick off the conversation. + messages.append({"role": "system", "content": "Say hello and briefly introduce yourself."}) + await task.queue_frames([context_aggregator.user().get_context_frame()]) + + @transport.event_handler("on_client_disconnected") + async def on_client_disconnected(transport, client): + logger.info(f"Client disconnected") + await task.cancel() + + runner = PipelineRunner(handle_sigint=handle_sigint) + + await runner.run(task) + + +if __name__ == "__main__": + from pipecat.runner.local import main + + # SmallWebRTCTransport for a P2P WebRTC connection + transport_params = { + "twilio": lambda: FastAPIWebsocketParams( + audio_in_enabled=True, + audio_out_enabled=True, + vad_analyzer=SileroVADAnalyzer(), + ), + } + + main(run_bot, transport_params=transport_params) diff --git a/examples/phone-bot-twilio/env.example b/examples/phone-bot-twilio/env.example new file mode 100644 index 000000000..1da398649 --- /dev/null +++ b/examples/phone-bot-twilio/env.example @@ -0,0 +1,3 @@ +OPENAI_API_KEY= +DEEPGRAM_API_KEY= +CARTESIA_API_KEY= diff --git a/examples/phone-bot-twilio/requirements.txt b/examples/phone-bot-twilio/requirements.txt new file mode 100644 index 000000000..180726c7c --- /dev/null +++ b/examples/phone-bot-twilio/requirements.txt @@ -0,0 +1 @@ +pipecat-ai[cartesia,openai,silero,deepgram,websocket,runner] \ No newline at end of file diff --git a/examples/phone-bot-twilio/templates/streams.xml.template b/examples/phone-bot-twilio/templates/streams.xml.template new file mode 100644 index 000000000..3d81d53d7 --- /dev/null +++ b/examples/phone-bot-twilio/templates/streams.xml.template @@ -0,0 +1,7 @@ + + + + + + + diff --git a/examples/quickstart/README.md b/examples/quickstart/README.md index a05964812..d56aca703 100644 --- a/examples/quickstart/README.md +++ b/examples/quickstart/README.md @@ -28,10 +28,9 @@ Have your API keys ready. We'll add them to your `.env` shortly. 1. Set up a virtual environment -From the root directory of the `pipecat` repo, run: +From the `examples/quickstart` directory, run: ```bash -cd examples/quickstart python -m venv .venv source .venv/bin/activate # On Windows: .venv\Scripts\activate ``` @@ -74,7 +73,7 @@ python bot.py > Using `uv`? Run your bot using: `uv run bot.py`. -Connect to your bot in a browser at http://localhost:7860. +**Open http://localhost:7860 in your browser** and click `Connect` to start talking to your bot. > 💡 First run note: The initial startup may take ~10 seconds as Pipecat downloads required models, like the Silero VAD model.