wx44wx/pipecat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

merge from main

Browse Source
This commit is contained in:
Adrian Cowham
2024-10-03 09:42:06 -07:00
parent 387a36dd8a 65eeb0f1f6
commit 2d82702e04
191 changed files with 8988 additions and 8902 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
.github/workflows/lint.yaml → .github/workflows/format.yaml vendored
View File

@@ -1,4 +1,4 @@
name: lint name: format
on: on:
workflow_dispatch: workflow_dispatch:
@@ -12,12 +12,12 @@ on:
- "docs/**" - "docs/**"
concurrency: concurrency:
group: build-lint-${{ github.event.pull_request.number || github.ref }} group: build-format-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true cancel-in-progress: true
jobs: jobs:
autopep8: ruff-format:
name: "Formatting lints" name: "Formatting checker"
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- name: Checkout repo - name: Checkout repo
@@ -25,7 +25,7 @@ jobs:
- name: Set up Python - name: Set up Python
uses: actions/setup-python@v4 uses: actions/setup-python@v4
with: with:
python-version: '3.10' python-version: "3.10"
- name: Setup virtual environment - name: Setup virtual environment
run: | run: |
python -m venv .venv python -m venv .venv
@@ -34,11 +34,8 @@ jobs:
source .venv/bin/activate source .venv/bin/activate
python -m pip install --upgrade pip python -m pip install --upgrade pip
pip install -r dev-requirements.txt pip install -r dev-requirements.txt
- name: autopep8 - name: Ruff formatter
id: autopep8 id: ruff
run: | run: |
source .venv/bin/activate source .venv/bin/activate
autopep8 --max-line-length 100 --exit-code -r -d --exclude "*_pb2.py" -a -a src/ ruff format --config line-length=100 --diff --exclude "*_pb2.py"
- name: Fail if autopep8 requires changes
if: steps.autopep8.outputs.exit-code == 2
run: exit 1

18
.github/workflows/tests.yaml vendored
View File

@@ -20,14 +20,24 @@ jobs:
name: "Unit and Integration Tests" name: "Unit and Integration Tests"
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - name: Checkout repo
uses: actions/checkout@v4
- name: Set up Python - name: Set up Python
id: setup_python id: setup_python
uses: actions/setup-python@v4 uses: actions/setup-python@v4
with: with:
python-version: "3.10" python-version: "3.10"
- name: Cache virtual environment
uses: actions/cache@v3
with:
# We are hashing dev-requirements.txt and test-requirements.txt which
# contain all dependencies needed to run the tests.
key: venv-${{ runner.os }}-${{ steps.setup_python.outputs.python-version}}-${{ hashFiles('dev-requirements.txt') }}-${{ hashFiles('test-requirements.txt') }}
path: .venv
- name: Install system packages - name: Install system packages
run: sudo apt-get install -y portaudio19-dev id: install_system_packages
run: |
sudo apt-get install -y portaudio19-dev
- name: Setup virtual environment - name: Setup virtual environment
run: | run: |
python -m venv .venv python -m venv .venv
@@ -35,8 +45,8 @@ jobs:
run: | run: |
source .venv/bin/activate source .venv/bin/activate
python -m pip install --upgrade pip python -m pip install --upgrade pip
pip install -r dev-requirements.txt pip install -r dev-requirements.txt -r test-requirements.txt
- name: Test with pytest - name: Test with pytest
run: | run: |
source .venv/bin/activate source .venv/bin/activate
pytest --doctest-modules --ignore-glob="*to_be_updated*" src tests pytest --ignore-glob="*to_be_updated*" --ignore-glob=*pipeline_source* src tests

127
CHANGELOG.md
View File

@@ -1,14 +1,79 @@
# Changelog # Changelog
All notable changes to **pipecat** will be documented in this file. All notable changes to **Pipecat** will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased] ## [0.0.42] - 2024-10-02
### Added ### Added
- `SentryMetrics` has been added to report frame processor metrics to
Sentry. This is now possible because `FrameProcessorMetrics` can now be passed
to `FrameProcessor`.
- Added Google TTS service and corresponding foundational example
`07n-interruptible-google.py`
- Added AWS Polly TTS support and `07m-interruptible-aws.py` as an example.
- Added InputParams to Azure TTS service.
- Added `LivekitTransport` (audio-only for now).
- RTVI 0.2.0 is now supported.
- All `FrameProcessors` can now register event handlers.
```
tts = SomeTTSService(...)
@tts.event_handler("on_connected"):
async def on_connected(processor):
...
```
- Added `AsyncGeneratorProcessor`. This processor can be used together with a
`FrameSerializer` as an async generator. It provides a `generator()` function
that returns an `AsyncGenerator` and that yields serialized frames.
- Added `EndTaskFrame` and `CancelTaskFrame`. These are new frames that are
meant to be pushed upstream to tell the pipeline task to stop nicely or
immediately respectively.
- Added configurable LLM parameters (e.g., temperature, top_p, max_tokens, seed)
for OpenAI, Anthropic, and Together AI services along with corresponding
setter functions.
- Added `sample_rate` as a constructor parameter for TTS services.
- Pipecat has a pipeline-based architecture. The pipeline consists of frame
processors linked to each other. The elements traveling across the pipeline
are called frames.
To have a deterministic behavior the frames traveling through the pipeline
should always be ordered, except system frames which are out-of-band
frames. To achieve that, each frame processor should only output frames from a
single task.
In this version all the frame processors have their own task to push
frames. That is, when `push_frame()` is called the given frame will be put
into an internal queue (with the exception of system frames) and a frame
processor task will push it out.
- Added pipeline clocks. A pipeline clock is used by the output transport to
know when a frame needs to be presented. For that, all frames now have an
optional `pts` field (prensentation timestamp). There's currently just one
clock implementation `SystemClock` and the `pts` field is currently only used
for `TextFrame`s (audio and image frames will be next).
- A clock can now be specified to `PipelineTask` (defaults to
`SystemClock`). This clock will be passed to each frame processor via the
`StartFrame`.
- Added `CartesiaHttpTTSService`.
- `DailyTransport` now supports setting the audio bitrate to improve audio - `DailyTransport` now supports setting the audio bitrate to improve audio
quality through the `DailyParams.audio_out_bitrate` parameter. The new quality through the `DailyParams.audio_out_bitrate` parameter. The new
default is 96kbps. default is 96kbps.
@@ -30,6 +95,43 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed ### Changed
- Context frames are now pushed downstream from assistant context aggregators.
- Removed Silero VAD torch dependency.
- Updated individual update settings frame classes into a single
`ServiceUpdateSettingsFrame` class.
- We now distinguish between input and output audio and image frames. We
introduce `InputAudioRawFrame`, `OutputAudioRawFrame`, `InputImageRawFrame`
and `OutputImageRawFrame` (and other subclasses of those). The input frames
usually come from an input transport and are meant to be processed inside the
pipeline to generate new frames. However, the input frames will not be sent
through an output transport. The output frames can also be processed by any
frame processor in the pipeline and they are allowed to be sent by the output
transport.
- `ParallelTask` has been renamed to `SyncParallelPipeline`. A
`SyncParallelPipeline` is a frame processor that contains a list of different
pipelines to be executed concurrently. The difference between a
`SyncParallelPipeline` and a `ParallelPipeline` is that, given an input frame,
the `SyncParallelPipeline` will wait for all the internal pipelines to
complete. This is achieved by making sure the last processor in each of the
pipelines is synchronous (e.g. an HTTP-based service that waits for the
response).
- `StartFrame` is back a system frame to make sure it's processed immediately by
all processors. `EndFrame` stays a control frame since it needs to be ordered
allowing the frames in the pipeline to be processed.
- Updated `MoondreamService` revision to `2024-08-26`.
- `CartesiaTTSService` and `ElevenLabsTTSService` now add presentation
timestamps to their text output. This allows the output transport to push the
text frames downstream at almost the same time the words are spoken. We say
"almost" because currently the audio frames don't have presentation timestamp
but they should be played at roughly the same time.
- `DailyTransport.on_joined` event now returns the full session data instead of - `DailyTransport.on_joined` event now returns the full session data instead of
just the participant. just the participant.
@@ -44,6 +146,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed ### Fixed
- Fixed OpenAI multiple function calls.
- Fixed a Cartesia TTS issue that would cause audio to be truncated in some
cases.
- Fixed a `BaseOutputTransport` issue that would stop audio and video rendering
tasks (after receiving and `EndFrame`) before the internal queue was emptied,
causing the pipeline to finish prematurely.
- `StartFrame` should be the first frame every processor receives to avoid - `StartFrame` should be the first frame every processor receives to avoid
situations where things are not initialized (because initialization happens on situations where things are not initialized (because initialization happens on
`StartFrame`) and other frames come in resulting in undesired behavior. `StartFrame`) and other frames come in resulting in undesired behavior.
@@ -53,6 +164,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `obj_id()` and `obj_count()` now use `itertools.count` avoiding the need of - `obj_id()` and `obj_count()` now use `itertools.count` avoiding the need of
`threading.Lock`. `threading.Lock`.
### Other
- Pipecat now uses Ruff as its formatter (https://github.com/astral-sh/ruff).
## [0.0.41] - 2024-08-22 ## [0.0.41] - 2024-08-22
### Added ### Added
@@ -277,7 +392,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- It is now possible to specify a Silero VAD version when using `SileroVADAnalyzer` - It is now possible to specify a Silero VAD version when using `SileroVADAnalyzer`
or `SileroVAD`. or `SileroVAD`.
- Added `AysncFrameProcessor` and `AsyncAIService`. Some services like - Added `AysncFrameProcessor` and `AsyncAIService`. Some services like
`DeepgramSTTService` need to process things asynchronously. For example, audio `DeepgramSTTService` need to process things asynchronously. For example, audio
is sent to Deepgram but transcriptions are not returned immediately. In these is sent to Deepgram but transcriptions are not returned immediately. In these
cases we still require all frames (except system frames) to be pushed cases we still require all frames (except system frames) to be pushed
@@ -294,7 +409,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `WhisperSTTService` model can now also be a string. - `WhisperSTTService` model can now also be a string.
- Added missing * keyword separators in services. - Added missing \* keyword separators in services.
### Fixed ### Fixed
@@ -371,7 +486,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Added new `TwilioFrameSerializer`. This is a new serializer that knows how to - Added new `TwilioFrameSerializer`. This is a new serializer that knows how to
serialize and deserialize audio frames from Twilio. serialize and deserialize audio frames from Twilio.
- Added Daily transport event: `on_dialout_answered`. See - Added Daily transport event: `on_dialout_answered`. See
https://reference-python.daily.co/api_reference.html#daily.EventHandler https://reference-python.daily.co/api_reference.html#daily.EventHandler
- Added new `AzureSTTService`. This allows you to use Azure Speech-To-Text. - Added new `AzureSTTService`. This allows you to use Azure Speech-To-Text.
@@ -611,7 +726,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Added Daily transport support for dial-in use cases. - Added Daily transport support for dial-in use cases.
- Added Daily transport events: `on_dialout_connected`, `on_dialout_stopped`, - Added Daily transport events: `on_dialout_connected`, `on_dialout_stopped`,
`on_dialout_error` and `on_dialout_warning`. See `on_dialout_error` and `on_dialout_warning`. See
https://reference-python.daily.co/api_reference.html#daily.EventHandler https://reference-python.daily.co/api_reference.html#daily.EventHandler
## [0.0.21] - 2024-05-22 ## [0.0.21] - 2024-05-22

32
README.md
View File

@@ -38,7 +38,7 @@ pip install "pipecat-ai[option,...]"
Your project may or may not need these, so they're made available as optional requirements. Here is a list: Your project may or may not need these, so they're made available as optional requirements. Here is a list:
- **AI services**: `anthropic`, `azure`, `deepgram`, `gladia`, `google`, `fal`, `lmnt`, `moondream`, `openai`, `openpipe`, `playht`, `silero`, `whisper`, `xtts` - **AI services**: `anthropic`, `aws`, `azure`, `deepgram`, `gladia`, `google`, `fal`, `lmnt`, `moondream`, `openai`, `openpipe`, `playht`, `silero`, `whisper`, `xtts`
- **Transports**: `local`, `websocket`, `daily` - **Transports**: `local`, `websocket`, `daily`
## Code examples ## Code examples
@@ -110,7 +110,6 @@ python app.py
Daily provides a prebuilt WebRTC user interface. Whilst the app is running, you can visit at `https://<yourdomain>.daily.co/<room_url>` and listen to the bot say hello! Daily provides a prebuilt WebRTC user interface. Whilst the app is running, you can visit at `https://<yourdomain>.daily.co/<room_url>` and listen to the bot say hello!
## WebRTC for production use ## WebRTC for production use
WebSockets are fine for server-to-server communication or for initial development. But for production use, you’ll need client-server audio to use a protocol designed for real-time media transport. (For an explanation of the difference between WebSockets and WebRTC, see [this post.](https://www.daily.co/blog/how-to-talk-to-an-llm-with-your-voice/#webrtc)) WebSockets are fine for server-to-server communication or for initial development. But for production use, you’ll need client-server audio to use a protocol designed for real-time media transport. (For an explanation of the difference between WebSockets and WebRTC, see [this post.](https://www.daily.co/blog/how-to-talk-to-an-llm-with-your-voice/#webrtc))
@@ -131,7 +130,6 @@ pip install pipecat-ai[silero]
The first time your run your bot with Silero, startup may take a while whilst it downloads and caches the model in the background. You can check the progress of this in the console. The first time your run your bot with Silero, startup may take a while whilst it downloads and caches the model in the background. You can check the progress of this in the console.
## Hacking on the framework itself ## Hacking on the framework itself
_Note that you may need to set up a virtual environment before following the instructions below. For instance, you might need to run the following from the root of the repo:_ _Note that you may need to set up a virtual environment before following the instructions below. For instance, you might need to run the following from the root of the repo:_
@@ -165,27 +163,29 @@ pip install "path_to_this_repo[option,...]"
From the root directory, run: From the root directory, run:
```shell ```shell
pytest --doctest-modules --ignore-glob="*to_be_updated*" src tests pytest --doctest-modules --ignore-glob="*to_be_updated*" --ignore-glob=*pipeline_source* src tests
``` ```
## Setting up your editor ## Setting up your editor
This project uses strict [PEP 8](https://peps.python.org/pep-0008/) formatting. This project uses strict [PEP 8](https://peps.python.org/pep-0008/) formatting via [Ruff](https://github.com/astral-sh/ruff).
### Emacs ### Emacs
You can use [use-package](https://github.com/jwiegley/use-package) to install [py-autopep8](https://codeberg.org/ideasman42/emacs-py-autopep8) package and configure `autopep8` arguments: You can use [use-package](https://github.com/jwiegley/use-package) to install [emacs-lazy-ruff](https://github.com/christophermadsen/emacs-lazy-ruff) package and configure `ruff` arguments:
```elisp ```elisp
(use-package py-autopep8 (use-package lazy-ruff
:ensure t :ensure t
:defer t :hook ((python-mode . lazy-ruff-mode))
:hook ((python-mode . py-autopep8-mode))
:config :config
(setq py-autopep8-options '("-a" "-a", "--max-line-length=100"))) (setq lazy-ruff-format-command "ruff format --config line-length=100")
(setq lazy-ruff-only-format-block t)
(setq lazy-ruff-only-format-region t)
(setq lazy-ruff-only-format-buffer t))
``` ```
`autopep8` was installed in the `venv` environment described before, so you should be able to use [pyvenv-auto](https://github.com/ryotaro612/pyvenv-auto) to automatically load that environment inside Emacs. `ruff` was installed in the `venv` environment described before, so you should be able to use [pyvenv-auto](https://github.com/ryotaro612/pyvenv-auto) to automatically load that environment inside Emacs.
```elisp ```elisp
(use-package pyvenv-auto (use-package pyvenv-auto
@@ -198,18 +198,14 @@ You can use [use-package](https://github.com/jwiegley/use-package) to install [p
### Visual Studio Code ### Visual Studio Code
Install the Install the
[autopep8](https://marketplace.visualstudio.com/items?itemName=ms-python.autopep8) extension. Then edit the user settings (_Ctrl-Shift-P_ `Open User Settings (JSON)`) and set it as the default Python formatter, enable formatting on save and configure `autopep8` arguments: [Ruff](https://marketplace.visualstudio.com/items?itemName=charliermarsh.ruff) extension. Then edit the user settings (_Ctrl-Shift-P_ `Open User Settings (JSON)`) and set it as the default Python formatter, enable formatting on save and configure `ruff` arguments:
```json ```json
"[python]": { "[python]": {
"editor.defaultFormatter": "ms-python.autopep8", "editor.defaultFormatter": "charliermarsh.ruff",
"editor.formatOnSave": true "editor.formatOnSave": true
}, },
"autopep8.args": [ "ruff.format.args": ["--config", "line-length=100"]
"-a",
"-a",
"--max-line-length=100"
],
``` ```
## Getting help ## Getting help

2
dev-requirements.txt
View File

@@ -1,8 +1,8 @@
autopep8~=2.3.1
build~=1.2.1 build~=1.2.1
grpcio-tools~=1.62.2 grpcio-tools~=1.62.2
pip-tools~=7.4.1 pip-tools~=7.4.1
pyright~=1.1.376 pyright~=1.1.376
pytest~=8.3.2 pytest~=8.3.2
ruff~=0.6.7
setuptools~=72.2.0 setuptools~=72.2.0
setuptools_scm~=8.1.0 setuptools_scm~=8.1.0

5
dot-env.template
View File

@@ -1,6 +1,11 @@
# Anthropic # Anthropic
ANTHROPIC_API_KEY=... ANTHROPIC_API_KEY=...
# AWS
AWS_SECRET_ACCESS_KEY=...
AWS_ACCESS_KEY_ID=...
AWS_REGION=...
# Azure # Azure
AZURE_SPEECH_REGION=... AZURE_SPEECH_REGION=...
AZURE_SPEECH_API_KEY=... AZURE_SPEECH_API_KEY=...

30
examples/deployment/flyio-example/bot.py
View File

@@ -6,7 +6,10 @@ import argparse
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import LLMAssistantResponseAggregator, LLMUserResponseAggregator from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.frames.frames import LLMMessagesFrame, EndFrame from pipecat.frames.frames import LLMMessagesFrame, EndFrame
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
from pipecat.services.elevenlabs import ElevenLabsTTSService from pipecat.services.elevenlabs import ElevenLabsTTSService
@@ -16,6 +19,7 @@ from pipecat.vad.silero import SileroVADAnalyzer
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -39,7 +43,7 @@ async def main(room_url: str, token: str):
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(), vad_analyzer=SileroVADAnalyzer(),
transcription_enabled=True, transcription_enabled=True,
) ),
) )
tts = ElevenLabsTTSService( tts = ElevenLabsTTSService(
@@ -47,9 +51,7 @@ async def main(room_url: str, token: str):
voice_id=os.getenv("ELEVENLABS_VOICE_ID", ""), voice_id=os.getenv("ELEVENLABS_VOICE_ID", ""),
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
messages = [ messages = [
{ {
@@ -61,14 +63,16 @@ async def main(room_url: str, token: str):
tma_in = LLMUserResponseAggregator(messages) tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), [
tma_in, transport.input(),
llm, tma_in,
tts, llm,
transport.output(), tts,
tma_out, transport.output(),
]) tma_out,
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))

118
examples/deployment/flyio-example/bot_runner.py
View File

@@ -16,9 +16,14 @@ from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import JSONResponse from fastapi.responses import JSONResponse
from pipecat.transports.services.helpers.daily_rest import ( from pipecat.transports.services.helpers.daily_rest import (
DailyRESTHelper, DailyRoomObject, DailyRoomProperties, DailyRoomParams) DailyRESTHelper,
DailyRoomObject,
DailyRoomProperties,
DailyRoomParams,
)
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
@@ -26,37 +31,37 @@ load_dotenv(override=True)
MAX_SESSION_TIME = 5 * 60 # 5 minutes MAX_SESSION_TIME = 5 * 60 # 5 minutes
REQUIRED_ENV_VARS = [ REQUIRED_ENV_VARS = [
'DAILY_API_KEY', "DAILY_API_KEY",
'OPENAI_API_KEY', "OPENAI_API_KEY",
'ELEVENLABS_API_KEY', "ELEVENLABS_API_KEY",
'ELEVENLABS_VOICE_ID', "ELEVENLABS_VOICE_ID",
'FLY_API_KEY', "FLY_API_KEY",
'FLY_APP_NAME',] "FLY_APP_NAME",
]
FLY_API_HOST = os.getenv("FLY_API_HOST", "https://api.machines.dev/v1") FLY_API_HOST = os.getenv("FLY_API_HOST", "https://api.machines.dev/v1")
FLY_APP_NAME = os.getenv("FLY_APP_NAME", "pipecat-fly-example") FLY_APP_NAME = os.getenv("FLY_APP_NAME", "pipecat-fly-example")
FLY_API_KEY = os.getenv("FLY_API_KEY", "") FLY_API_KEY = os.getenv("FLY_API_KEY", "")
FLY_HEADERS = { FLY_HEADERS = {"Authorization": f"Bearer {FLY_API_KEY}", "Content-Type": "application/json"}
'Authorization': f"Bearer {FLY_API_KEY}",
'Content-Type': 'application/json'
}
daily_helpers = {} daily_helpers = {}
# ----------------- API ----------------- # # ----------------- API ----------------- #
@asynccontextmanager @asynccontextmanager
async def lifespan(app: FastAPI): async def lifespan(app: FastAPI):
aiohttp_session = aiohttp.ClientSession() aiohttp_session = aiohttp.ClientSession()
daily_helpers["rest"] = DailyRESTHelper( daily_helpers["rest"] = DailyRESTHelper(
daily_api_key=os.getenv("DAILY_API_KEY", ""), daily_api_key=os.getenv("DAILY_API_KEY", ""),
daily_api_url=os.getenv("DAILY_API_URL", 'https://api.daily.co/v1'), daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
aiohttp_session=aiohttp_session aiohttp_session=aiohttp_session,
) )
yield yield
await aiohttp_session.close() await aiohttp_session.close()
app = FastAPI(lifespan=lifespan) app = FastAPI(lifespan=lifespan)
app.add_middleware( app.add_middleware(
@@ -64,7 +69,7 @@ app.add_middleware(
allow_origins=["*"], allow_origins=["*"],
allow_credentials=True, allow_credentials=True,
allow_methods=["*"], allow_methods=["*"],
allow_headers=["*"] allow_headers=["*"],
) )
# ----------------- Main ----------------- # # ----------------- Main ----------------- #
@@ -73,13 +78,15 @@ app.add_middleware(
async def spawn_fly_machine(room_url: str, token: str): async def spawn_fly_machine(room_url: str, token: str):
async with aiohttp.ClientSession() as session: async with aiohttp.ClientSession() as session:
# Use the same image as the bot runner # Use the same image as the bot runner
async with session.get(f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines", headers=FLY_HEADERS) as r: async with session.get(
f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines", headers=FLY_HEADERS
) as r:
if r.status != 200: if r.status != 200:
text = await r.text() text = await r.text()
raise Exception(f"Unable to get machine info from Fly: {text}") raise Exception(f"Unable to get machine info from Fly: {text}")
data = await r.json() data = await r.json()
image = data[0]['config']['image'] image = data[0]["config"]["image"]
# Machine configuration # Machine configuration
cmd = f"python3 bot.py -u {room_url} -t {token}" cmd = f"python3 bot.py -u {room_url} -t {token}"
@@ -88,31 +95,28 @@ async def spawn_fly_machine(room_url: str, token: str):
"config": { "config": {
"image": image, "image": image,
"auto_destroy": True, "auto_destroy": True,
"init": { "init": {"cmd": cmd},
"cmd": cmd "restart": {"policy": "no"},
}, "guest": {"cpu_kind": "shared", "cpus": 1, "memory_mb": 1024},
"restart": {
"policy": "no"
},
"guest": {
"cpu_kind": "shared",
"cpus": 1,
"memory_mb": 1024
}
}, },
} }
# Spawn a new machine instance # Spawn a new machine instance
async with session.post(f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines", headers=FLY_HEADERS, json=worker_props) as r: async with session.post(
f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines", headers=FLY_HEADERS, json=worker_props
) as r:
if r.status != 200: if r.status != 200:
text = await r.text() text = await r.text()
raise Exception(f"Problem starting a bot worker: {text}") raise Exception(f"Problem starting a bot worker: {text}")
data = await r.json() data = await r.json()
# Wait for the machine to enter the started state # Wait for the machine to enter the started state
vm_id = data['id'] vm_id = data["id"]
async with session.get(f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines/{vm_id}/wait?state=started", headers=FLY_HEADERS) as r: async with session.get(
f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines/{vm_id}/wait?state=started",
headers=FLY_HEADERS,
) as r:
if r.status != 200: if r.status != 200:
text = await r.text() text = await r.text()
raise Exception(f"Bot was unable to enter started state: {text}") raise Exception(f"Bot was unable to enter started state: {text}")
@@ -134,29 +138,23 @@ async def start_bot(request: Request) -> JSONResponse:
room_url = os.getenv("DAILY_SAMPLE_ROOM_URL", "") room_url = os.getenv("DAILY_SAMPLE_ROOM_URL", "")
if not room_url: if not room_url:
params = DailyRoomParams( params = DailyRoomParams(properties=DailyRoomProperties())
properties=DailyRoomProperties()
)
try: try:
room: DailyRoomObject = await daily_helpers["rest"].create_room(params=params) room: DailyRoomObject = await daily_helpers["rest"].create_room(params=params)
except Exception as e: except Exception as e:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Unable to provision room {e}")
status_code=500,
detail=f"Unable to provision room {e}")
else: else:
# Check passed room URL exists, we should assume that it already has a sip set up # Check passed room URL exists, we should assume that it already has a sip set up
try: try:
room: DailyRoomObject = await daily_helpers["rest"].get_room_from_url(room_url) room: DailyRoomObject = await daily_helpers["rest"].get_room_from_url(room_url)
except Exception: except Exception:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Room not found: {room_url}")
status_code=500, detail=f"Room not found: {room_url}")
# Give the agent a token to join the session # Give the agent a token to join the session
token = await daily_helpers["rest"].get_token(room.url, MAX_SESSION_TIME) token = await daily_helpers["rest"].get_token(room.url, MAX_SESSION_TIME)
if not room or not token: if not room or not token:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Failed to get token for room: {room_url}")
status_code=500, detail=f"Failed to get token for room: {room_url}")
# Launch a new fly.io machine, or run as a shell process (not recommended) # Launch a new fly.io machine, or run as a shell process (not recommended)
run_as_process = os.getenv("RUN_AS_PROCESS", False) run_as_process = os.getenv("RUN_AS_PROCESS", False)
@@ -167,24 +165,26 @@ async def start_bot(request: Request) -> JSONResponse:
[f"python3 -m bot -u {room.url} -t {token}"], [f"python3 -m bot -u {room.url} -t {token}"],
shell=True, shell=True,
bufsize=1, bufsize=1,
cwd=os.path.dirname(os.path.abspath(__file__))) cwd=os.path.dirname(os.path.abspath(__file__)),
)
except Exception as e: except Exception as e:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Failed to start subprocess: {e}")
status_code=500, detail=f"Failed to start subprocess: {e}")
else: else:
try: try:
await spawn_fly_machine(room.url, token) await spawn_fly_machine(room.url, token)
except Exception as e: except Exception as e:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Failed to spawn VM: {e}")
status_code=500, detail=f"Failed to spawn VM: {e}")
# Grab a token for the user to join with # Grab a token for the user to join with
user_token = await daily_helpers["rest"].get_token(room.url, MAX_SESSION_TIME) user_token = await daily_helpers["rest"].get_token(room.url, MAX_SESSION_TIME)
return JSONResponse({ return JSONResponse(
"room_url": room.url, {
"token": user_token, "room_url": room.url,
}) "token": user_token,
}
)
if __name__ == "__main__": if __name__ == "__main__":
# Check environment variables # Check environment variables
@@ -193,23 +193,19 @@ if __name__ == "__main__":
raise Exception(f"Missing environment variable: {env_var}.") raise Exception(f"Missing environment variable: {env_var}.")
parser = argparse.ArgumentParser(description="Pipecat Bot Runner") parser = argparse.ArgumentParser(description="Pipecat Bot Runner")
parser.add_argument("--host", type=str, parser.add_argument(
default=os.getenv("HOST", "0.0.0.0"), help="Host address") "--host", type=str, default=os.getenv("HOST", "0.0.0.0"), help="Host address"
parser.add_argument("--port", type=int, )
default=os.getenv("PORT", 7860), help="Port number") parser.add_argument("--port", type=int, default=os.getenv("PORT", 7860), help="Port number")
parser.add_argument("--reload", action="store_true", parser.add_argument(
default=False, help="Reload code on change") "--reload", action="store_true", default=False, help="Reload code on change"
)
config = parser.parse_args() config = parser.parse_args()
try: try:
import uvicorn import uvicorn
uvicorn.run( uvicorn.run("bot_runner:app", host=config.host, port=config.port, reload=config.reload)
"bot_runner:app",
host=config.host,
port=config.port,
reload=config.reload
)
except KeyboardInterrupt: except KeyboardInterrupt:
print("Pipecat runner shutting down...") print("Pipecat runner shutting down...")

39
examples/dialin-chatbot/bot_daily.py
View File

@@ -6,11 +6,11 @@ import argparse
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import LLMAssistantResponseAggregator, LLMUserResponseAggregator from pipecat.processors.aggregators.llm_response import (
from pipecat.frames.frames import ( LLMAssistantResponseAggregator,
LLMMessagesFrame, LLMUserResponseAggregator,
EndFrame
) )
from pipecat.frames.frames import LLMMessagesFrame, EndFrame
from pipecat.services.elevenlabs import ElevenLabsTTSService from pipecat.services.elevenlabs import ElevenLabsTTSService
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport, DailyDialinSettings from pipecat.transports.services.daily import DailyParams, DailyTransport, DailyDialinSettings
@@ -18,6 +18,7 @@ from pipecat.vad.silero import SileroVADAnalyzer
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -31,10 +32,7 @@ async def main(room_url: str, token: str, callId: str, callDomain: str):
# diallin_settings are only needed if Daily's SIP URI is used # diallin_settings are only needed if Daily's SIP URI is used
# If you are handling this via Twilio, Telnyx, set this to None # If you are handling this via Twilio, Telnyx, set this to None
# and handle call-forwarding when on_dialin_ready fires. # and handle call-forwarding when on_dialin_ready fires.
diallin_settings = DailyDialinSettings( diallin_settings = DailyDialinSettings(call_id=callId, call_domain=callDomain)
call_id=callId,
call_domain=callDomain
)
transport = DailyTransport( transport = DailyTransport(
room_url, room_url,
@@ -50,7 +48,7 @@ async def main(room_url: str, token: str, callId: str, callDomain: str):
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(), vad_analyzer=SileroVADAnalyzer(),
transcription_enabled=True, transcription_enabled=True,
) ),
) )
tts = ElevenLabsTTSService( tts = ElevenLabsTTSService(
@@ -58,10 +56,7 @@ async def main(room_url: str, token: str, callId: str, callDomain: str):
voice_id=os.getenv("ELEVENLABS_VOICE_ID", ""), voice_id=os.getenv("ELEVENLABS_VOICE_ID", ""),
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o"
)
messages = [ messages = [
{ {
@@ -73,14 +68,16 @@ async def main(room_url: str, token: str, callId: str, callDomain: str):
tma_in = LLMUserResponseAggregator(messages) tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), [
tma_in, transport.input(),
llm, tma_in,
tts, llm,
transport.output(), tts,
tma_out, transport.output(),
]) tma_out,
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))

70
examples/dialin-chatbot/bot_runner.py
View File

@@ -7,7 +7,6 @@ provisioning a room and starting a Pipecat bot in response.
Refer to README for more information. Refer to README for more information.
""" """
import aiohttp import aiohttp
import os import os
import argparse import argparse
@@ -25,17 +24,18 @@ from pipecat.transports.services.helpers.daily_rest import (
DailyRoomObject, DailyRoomObject,
DailyRoomProperties, DailyRoomProperties,
DailyRoomSipParams, DailyRoomSipParams,
DailyRoomParams) DailyRoomParams,
)
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
# ------------ Configuration ------------ # # ------------ Configuration ------------ #
MAX_SESSION_TIME = 5 * 60 # 5 minutes MAX_SESSION_TIME = 5 * 60 # 5 minutes
REQUIRED_ENV_VARS = ['OPENAI_API_KEY', 'DAILY_API_KEY', REQUIRED_ENV_VARS = ["OPENAI_API_KEY", "DAILY_API_KEY", "ELEVENLABS_API_KEY", "ELEVENLABS_VOICE_ID"]
'ELEVENLABS_API_KEY', 'ELEVENLABS_VOICE_ID']
daily_helpers = {} daily_helpers = {}
@@ -47,12 +47,13 @@ async def lifespan(app: FastAPI):
aiohttp_session = aiohttp.ClientSession() aiohttp_session = aiohttp.ClientSession()
daily_helpers["rest"] = DailyRESTHelper( daily_helpers["rest"] = DailyRESTHelper(
daily_api_key=os.getenv("DAILY_API_KEY", ""), daily_api_key=os.getenv("DAILY_API_KEY", ""),
daily_api_url=os.getenv("DAILY_API_URL", 'https://api.daily.co/v1'), daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
aiohttp_session=aiohttp_session aiohttp_session=aiohttp_session,
) )
yield yield
await aiohttp_session.close() await aiohttp_session.close()
app = FastAPI(lifespan=lifespan) app = FastAPI(lifespan=lifespan)
app.add_middleware( app.add_middleware(
@@ -60,7 +61,7 @@ app.add_middleware(
allow_origins=["*"], allow_origins=["*"],
allow_credentials=True, allow_credentials=True,
allow_methods=["*"], allow_methods=["*"],
allow_headers=["*"] allow_headers=["*"],
) )
""" """
@@ -80,10 +81,7 @@ async def _create_daily_room(room_url, callId, callDomain=None, vendor="daily"):
properties=DailyRoomProperties( properties=DailyRoomProperties(
# Note: these are the default values, except for the display name # Note: these are the default values, except for the display name
sip=DailyRoomSipParams( sip=DailyRoomSipParams(
display_name="dialin-user", display_name="dialin-user", video=False, sip_mode="dial-in", num_endpoints=1
video=False,
sip_mode="dial-in",
num_endpoints=1
) )
) )
) )
@@ -97,8 +95,7 @@ async def _create_daily_room(room_url, callId, callDomain=None, vendor="daily"):
print(f"Joining existing room: {room_url}") print(f"Joining existing room: {room_url}")
room: DailyRoomObject = await daily_helpers["rest"].get_room_from_url(room_url) room: DailyRoomObject = await daily_helpers["rest"].get_room_from_url(room_url)
except Exception: except Exception:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Room not found: {room_url}")
status_code=500, detail=f"Room not found: {room_url}")
print(f"Daily room: {room.url} {room.config.sip_endpoint}") print(f"Daily room: {room.url} {room.config.sip_endpoint}")
@@ -106,8 +103,7 @@ async def _create_daily_room(room_url, callId, callDomain=None, vendor="daily"):
token = await daily_helpers["rest"].get_token(room.url, MAX_SESSION_TIME) token = await daily_helpers["rest"].get_token(room.url, MAX_SESSION_TIME)
if not room or not token: if not room or not token:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Failed to get room or token token")
status_code=500, detail=f"Failed to get room or token token")
# Spawn a new agent, and join the user session # Spawn a new agent, and join the user session
# Note: this is mostly for demonstration purposes (refer to 'deployment' in docs) # Note: this is mostly for demonstration purposes (refer to 'deployment' in docs)
@@ -120,14 +116,10 @@ async def _create_daily_room(room_url, callId, callDomain=None, vendor="daily"):
try: try:
subprocess.Popen( subprocess.Popen(
[bot_proc], [bot_proc], shell=True, bufsize=1, cwd=os.path.dirname(os.path.abspath(__file__))
shell=True,
bufsize=1,
cwd=os.path.dirname(os.path.abspath(__file__))
) )
except Exception as e: except Exception as e:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Failed to start subprocess: {e}")
status_code=500, detail=f"Failed to start subprocess: {e}")
return room return room
@@ -150,11 +142,10 @@ async def twilio_start_bot(request: Request):
pass pass
room_url = os.getenv("DAILY_SAMPLE_ROOM_URL", None) room_url = os.getenv("DAILY_SAMPLE_ROOM_URL", None)
callId = data.get('CallSid') callId = data.get("CallSid")
if not callId: if not callId:
raise HTTPException( raise HTTPException(status_code=500, detail="Missing 'CallSid' in request")
status_code=500, detail="Missing 'CallSid' in request")
print("CallId: %s" % callId) print("CallId: %s" % callId)
@@ -170,7 +161,8 @@ async def twilio_start_bot(request: Request):
# http://com.twilio.music.classical.s3.amazonaws.com/BusyStrings.mp3 # http://com.twilio.music.classical.s3.amazonaws.com/BusyStrings.mp3
resp = VoiceResponse() resp = VoiceResponse()
resp.play( resp.play(
url="http://com.twilio.sounds.music.s3.amazonaws.com/MARKOVICHAMP-Borghestral.mp3", loop=10) url="http://com.twilio.sounds.music.s3.amazonaws.com/MARKOVICHAMP-Borghestral.mp3", loop=10
)
return str(resp) return str(resp)
@@ -192,18 +184,14 @@ async def daily_start_bot(request: Request) -> JSONResponse:
callId = data.get("callId", None) callId = data.get("callId", None)
callDomain = data.get("callDomain", None) callDomain = data.get("callDomain", None)
except Exception: except Exception:
raise HTTPException( raise HTTPException(status_code=500, detail="Missing properties 'callId' or 'callDomain'")
status_code=500,
detail="Missing properties 'callId' or 'callDomain'")
print(f"CallId: {callId}, CallDomain: {callDomain}") print(f"CallId: {callId}, CallDomain: {callDomain}")
room: DailyRoomObject = await _create_daily_room(room_url, callId, callDomain, "daily") room: DailyRoomObject = await _create_daily_room(room_url, callId, callDomain, "daily")
# Grab a token for the user to join with # Grab a token for the user to join with
return JSONResponse({ return JSONResponse({"room_url": room.url, "sipUri": room.config.sip_endpoint})
"room_url": room.url,
"sipUri": room.config.sip_endpoint
})
# ----------------- Main ----------------- # # ----------------- Main ----------------- #
@@ -215,24 +203,18 @@ if __name__ == "__main__":
raise Exception(f"Missing environment variable: {env_var}.") raise Exception(f"Missing environment variable: {env_var}.")
parser = argparse.ArgumentParser(description="Pipecat Bot Runner") parser = argparse.ArgumentParser(description="Pipecat Bot Runner")
parser.add_argument("--host", type=str, parser.add_argument(
default=os.getenv("HOST", "0.0.0.0"), help="Host address") "--host", type=str, default=os.getenv("HOST", "0.0.0.0"), help="Host address"
parser.add_argument("--port", type=int, )
default=os.getenv("PORT", 7860), help="Port number") parser.add_argument("--port", type=int, default=os.getenv("PORT", 7860), help="Port number")
parser.add_argument("--reload", action="store_true", parser.add_argument("--reload", action="store_true", default=True, help="Reload code on change")
default=True, help="Reload code on change")
config = parser.parse_args() config = parser.parse_args()
try: try:
import uvicorn import uvicorn
uvicorn.run( uvicorn.run("bot_runner:app", host=config.host, port=config.port, reload=config.reload)
"bot_runner:app",
host=config.host,
port=config.port,
reload=config.reload
)
except KeyboardInterrupt: except KeyboardInterrupt:
print("Pipecat runner shutting down...") print("Pipecat runner shutting down...")

40
examples/dialin-chatbot/bot_twilio.py
View File

@@ -6,11 +6,11 @@ import argparse
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import LLMAssistantResponseAggregator, LLMUserResponseAggregator from pipecat.processors.aggregators.llm_response import (
from pipecat.frames.frames import ( LLMAssistantResponseAggregator,
LLMMessagesFrame, LLMUserResponseAggregator,
EndFrame
) )
from pipecat.frames.frames import LLMMessagesFrame, EndFrame
from pipecat.services.elevenlabs import ElevenLabsTTSService from pipecat.services.elevenlabs import ElevenLabsTTSService
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.transports.services.daily import DailyParams, DailyTransport
@@ -21,14 +21,15 @@ from twilio.rest import Client
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
logger.add(sys.stderr, level="DEBUG") logger.add(sys.stderr, level="DEBUG")
twilio_account_sid = os.getenv('TWILIO_ACCOUNT_SID') twilio_account_sid = os.getenv("TWILIO_ACCOUNT_SID")
twilio_auth_token = os.getenv('TWILIO_AUTH_TOKEN') twilio_auth_token = os.getenv("TWILIO_AUTH_TOKEN")
twilioclient = Client(twilio_account_sid, twilio_auth_token) twilioclient = Client(twilio_account_sid, twilio_auth_token)
daily_api_key = os.getenv("DAILY_API_KEY", "") daily_api_key = os.getenv("DAILY_API_KEY", "")
@@ -51,7 +52,7 @@ async def main(room_url: str, token: str, callId: str, sipUri: str):
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(), vad_analyzer=SileroVADAnalyzer(),
transcription_enabled=True, transcription_enabled=True,
) ),
) )
tts = ElevenLabsTTSService( tts = ElevenLabsTTSService(
@@ -59,10 +60,7 @@ async def main(room_url: str, token: str, callId: str, sipUri: str):
voice_id=os.getenv("ELEVENLABS_VOICE_ID", ""), voice_id=os.getenv("ELEVENLABS_VOICE_ID", ""),
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o"
)
messages = [ messages = [
{ {
@@ -74,14 +72,16 @@ async def main(room_url: str, token: str, callId: str, sipUri: str):
tma_in = LLMUserResponseAggregator(messages) tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), [
tma_in, transport.input(),
llm, tma_in,
tts, llm,
transport.output(), tts,
tma_out, transport.output(),
]) tma_out,
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
@@ -103,7 +103,7 @@ async def main(room_url: str, token: str, callId: str, sipUri: str):
try: try:
# The TwiML is updated using Twilio's client library # The TwiML is updated using Twilio's client library
call = twilioclient.calls(callId).update( call = twilioclient.calls(callId).update(
twiml=f'<Response><Dial><Sip>{sipUri}</Sip></Dial></Response>' twiml=f"<Response><Dial><Sip>{sipUri}</Sip></Dial></Response>"
) )
except Exception as e: except Exception as e:
raise Exception(f"Failed to forward call: {str(e)}") raise Exception(f"Failed to forward call: {str(e)}")

2
examples/dialin-chatbot/requirements.txt
View File

@@ -1,4 +1,4 @@
pipecat-ai[daily,openai,silero] pipecat-ai[daily,elevenlabs,openai,silero]
fastapi fastapi
uvicorn uvicorn
python-dotenv python-dotenv

15
examples/foundational/01-say-one-thing.py
View File

@@ -9,11 +9,11 @@ import aiohttp
import os import os
import sys import sys
from pipecat.frames.frames import TextFrame from pipecat.frames.frames import EndFrame, TextFrame
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.task import PipelineTask from pipecat.pipeline.task import PipelineTask
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.services.cartesia import CartesiaTTSService from pipecat.services.cartesia import CartesiaHttpTTSService
from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.transports.services.daily import DailyParams, DailyTransport
from runner import configure from runner import configure
@@ -21,6 +21,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -32,9 +33,10 @@ async def main():
(room_url, _) = await configure(session) (room_url, _) = await configure(session)
transport = DailyTransport( transport = DailyTransport(
room_url, None, "Say One Thing", DailyParams(audio_out_enabled=True)) room_url, None, "Say One Thing", DailyParams(audio_out_enabled=True)
)
tts = CartesiaTTSService( tts = CartesiaHttpTTSService(
api_key=os.getenv("CARTESIA_API_KEY"), api_key=os.getenv("CARTESIA_API_KEY"),
voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
) )
@@ -47,10 +49,11 @@ async def main():
# participant joins. # participant joins.
@transport.event_handler("on_participant_joined") @transport.event_handler("on_participant_joined")
async def on_new_participant_joined(transport, participant): async def on_new_participant_joined(transport, participant):
participant_name = participant["info"]["userName"] or '' participant_name = participant["info"]["userName"] or ""
await task.queue_frame(TextFrame(f"Hello there, {participant_name}!")) await task.queue_frames([TextFrame(f"Hello there, {participant_name}!"), EndFrame()])
await runner.run(task) await runner.run(task)
if __name__ == "__main__": if __name__ == "__main__":
asyncio.run(main()) asyncio.run(main())

1
examples/foundational/01a-local-audio.py
View File

@@ -20,6 +20,7 @@ from pipecat.transports.local.audio import LocalAudioTransport
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)

108
examples/foundational/01b-livekit-audio.py Normal file
View File

@@ -0,0 +1,108 @@
import argparse
import asyncio
import os
import sys
import aiohttp
from dotenv import load_dotenv
from livekit import api # pip install livekit-api
from loguru import logger
from pipecat.frames.frames import TextFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask
from pipecat.services.cartesia import CartesiaTTSService
from pipecat.transports.services.livekit import LiveKitParams, LiveKitTransport
load_dotenv(override=True)
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
def generate_token(room_name: str, participant_name: str, api_key: str, api_secret: str) -> str:
token = api.AccessToken(api_key, api_secret)
token.with_identity(participant_name).with_name(participant_name).with_grants(
api.VideoGrants(
room_join=True,
room=room_name,
)
)
return token.to_jwt()
async def configure_livekit():
parser = argparse.ArgumentParser(description="LiveKit AI SDK Bot Sample")
parser.add_argument(
"-r", "--room", type=str, required=False, help="Name of the LiveKit room to join"
)
parser.add_argument("-u", "--url", type=str, required=False, help="URL of the LiveKit server")
args, unknown = parser.parse_known_args()
room_name = args.room or os.getenv("LIVEKIT_ROOM_NAME")
url = args.url or os.getenv("LIVEKIT_URL")
api_key = os.getenv("LIVEKIT_API_KEY")
api_secret = os.getenv("LIVEKIT_API_SECRET")
if not room_name:
raise Exception(
"No LiveKit room specified. Use the -r/--room option from the command line, or set LIVEKIT_ROOM_NAME in your environment."
)
if not url:
raise Exception(
"No LiveKit server URL specified. Use the -u/--url option from the command line, or set LIVEKIT_URL in your environment."
)
if not api_key or not api_secret:
raise Exception(
"LIVEKIT_API_KEY and LIVEKIT_API_SECRET must be set in environment variables."
)
token = generate_token(room_name, "Say One Thing", api_key, api_secret)
user_token = generate_token(room_name, "User", api_key, api_secret)
logger.info(f"User token: {user_token}")
return (url, token, room_name)
async def main():
async with aiohttp.ClientSession() as session:
(url, token, room_name) = await configure_livekit()
transport = LiveKitTransport(
url=url,
token=token,
room_name=room_name,
params=LiveKitParams(audio_out_enabled=True, audio_out_sample_rate=16000),
)
tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY"),
voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
)
runner = PipelineRunner()
task = PipelineTask(Pipeline([tts, transport.output()]))
# Register an event handler so we can play the audio when the
# participant joins.
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant_id):
await asyncio.sleep(1)
await task.queue_frame(
TextFrame(
"Hello there! How are you doing today? Would you like to talk about the weather?"
)
)
await runner.run(task)
if __name__ == "__main__":
asyncio.run(main())

22
examples/foundational/02-llm-say-one-thing.py
View File

@@ -9,11 +9,11 @@ import aiohttp
import os import os
import sys import sys
from pipecat.frames.frames import LLMMessagesFrame from pipecat.frames.frames import EndFrame, LLMMessagesFrame
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask from pipecat.pipeline.task import PipelineTask
from pipecat.services.cartesia import CartesiaTTSService from pipecat.services.cartesia import CartesiaHttpTTSService
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.transports.services.daily import DailyParams, DailyTransport
@@ -22,6 +22,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -33,25 +34,22 @@ async def main():
(room_url, _) = await configure(session) (room_url, _) = await configure(session)
transport = DailyTransport( transport = DailyTransport(
room_url, room_url, None, "Say One Thing From an LLM", DailyParams(audio_out_enabled=True)
None, )
"Say One Thing From an LLM",
DailyParams(audio_out_enabled=True))
tts = CartesiaTTSService( tts = CartesiaHttpTTSService(
api_key=os.getenv("CARTESIA_API_KEY"), api_key=os.getenv("CARTESIA_API_KEY"),
voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
messages = [ messages = [
{ {
"role": "system", "role": "system",
"content": "You are an LLM in a WebRTC session, and this is a 'hello world' demo. Say hello to the world.", "content": "You are an LLM in a WebRTC session, and this is a 'hello world' demo. Say hello to the world.",
}] }
]
runner = PipelineRunner() runner = PipelineRunner()
@@ -59,7 +57,7 @@ async def main():
@transport.event_handler("on_first_participant_joined") @transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant): async def on_first_participant_joined(transport, participant):
await task.queue_frame(LLMMessagesFrame(messages)) await task.queue_frames([LLMMessagesFrame(messages), EndFrame()])
await runner.run(task) await runner.run(task)

11
examples/foundational/03-still-frame.py
View File

@@ -21,6 +21,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -35,17 +36,11 @@ async def main():
room_url, room_url,
None, None,
"Show a still frame image", "Show a still frame image",
DailyParams( DailyParams(camera_out_enabled=True, camera_out_width=1024, camera_out_height=1024),
camera_out_enabled=True,
camera_out_width=1024,
camera_out_height=1024
)
) )
imagegen = FalImageGenService( imagegen = FalImageGenService(
params=FalImageGenService.InputParams( params=FalImageGenService.InputParams(image_size="square_hd"),
image_size="square_hd"
),
aiohttp_session=session, aiohttp_session=session,
key=os.getenv("FAL_KEY"), key=os.getenv("FAL_KEY"),
) )

11
examples/foundational/03a-local-still-frame.py
View File

@@ -22,6 +22,7 @@ from pipecat.transports.local.tk import TkLocalTransport
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -35,15 +36,11 @@ async def main():
transport = TkLocalTransport( transport = TkLocalTransport(
tk_root, tk_root,
TransportParams( TransportParams(camera_out_enabled=True, camera_out_width=1024, camera_out_height=1024),
camera_out_enabled=True, )
camera_out_width=1024,
camera_out_height=1024))
imagegen = FalImageGenService( imagegen = FalImageGenService(
params=FalImageGenService.InputParams( params=FalImageGenService.InputParams(image_size="square_hd"),
image_size="square_hd"
),
aiohttp_session=session, aiohttp_session=session,
key=os.getenv("FAL_KEY"), key=os.getenv("FAL_KEY"),
) )

11
examples/foundational/04-utterance-and-speech.py
View File

@@ -4,6 +4,10 @@
# SPDX-License-Identifier: BSD 2-Clause License # SPDX-License-Identifier: BSD 2-Clause License
# #
#
# This example broken on latest pipecat and needs updating.
#
import aiohttp import aiohttp
import asyncio import asyncio
import os import os
@@ -24,6 +28,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -54,8 +59,7 @@ async def main():
voice_id=os.getenv("ELEVENLABS_VOICE_ID"), voice_id=os.getenv("ELEVENLABS_VOICE_ID"),
) )
messages = [{"role": "system", messages = [{"role": "system", "content": "tell the user a joke about llamas"}]
"content": "tell the user a joke about llamas"}]
# Start a task to run the LLM to create a joke, and convert the LLM # Start a task to run the LLM to create a joke, and convert the LLM
# output to audio frames. This task will run in parallel with generating # output to audio frames. This task will run in parallel with generating
@@ -73,8 +77,7 @@ async def main():
] ]
) )
merge_pipeline = SequentialMergePipeline( merge_pipeline = SequentialMergePipeline([simple_tts_pipeline, llm_pipeline])
[simple_tts_pipeline, llm_pipeline])
await asyncio.gather( await asyncio.gather(
transport.run(merge_pipeline), transport.run(merge_pipeline),

67
examples/foundational/05-sync-speech-and-image.py
View File

@@ -14,21 +14,18 @@ from dataclasses import dataclass
from pipecat.frames.frames import ( from pipecat.frames.frames import (
AppFrame, AppFrame,
Frame, Frame,
ImageRawFrame,
LLMFullResponseStartFrame, LLMFullResponseStartFrame,
LLMMessagesFrame, LLMMessagesFrame,
TextFrame TextFrame,
) )
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.sync_parallel_pipeline import SyncParallelPipeline
from pipecat.pipeline.task import PipelineTask from pipecat.pipeline.task import PipelineTask
from pipecat.pipeline.parallel_task import ParallelTask
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.processors.aggregators.gated import GatedAggregator
from pipecat.processors.aggregators.llm_response import LLMFullResponseAggregator
from pipecat.processors.aggregators.sentence import SentenceAggregator from pipecat.processors.aggregators.sentence import SentenceAggregator
from pipecat.services.cartesia import CartesiaHttpTTSService
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
from pipecat.services.elevenlabs import ElevenLabsTTSService
from pipecat.services.fal import FalImageGenService from pipecat.services.fal import FalImageGenService
from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.transports.services.daily import DailyParams, DailyTransport
@@ -37,6 +34,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -84,47 +82,46 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
camera_out_enabled=True, camera_out_enabled=True,
camera_out_width=1024, camera_out_width=1024,
camera_out_height=1024 camera_out_height=1024,
) ),
) )
tts = ElevenLabsTTSService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("ELEVENLABS_API_KEY"),
voice_id=os.getenv("ELEVENLABS_VOICE_ID"),
)
llm = OpenAILLMService( tts = CartesiaHttpTTSService(
api_key=os.getenv("OPENAI_API_KEY"), api_key=os.getenv("CARTESIA_API_KEY"),
model="gpt-4o") voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
)
imagegen = FalImageGenService( imagegen = FalImageGenService(
params=FalImageGenService.InputParams( params=FalImageGenService.InputParams(image_size="square_hd"),
image_size="square_hd"
),
aiohttp_session=session, aiohttp_session=session,
key=os.getenv("FAL_KEY"), key=os.getenv("FAL_KEY"),
) )
gated_aggregator = GatedAggregator(
gate_open_fn=lambda frame: isinstance(frame, ImageRawFrame),
gate_close_fn=lambda frame: isinstance(frame, LLMFullResponseStartFrame),
start_open=False
)
sentence_aggregator = SentenceAggregator() sentence_aggregator = SentenceAggregator()
month_prepender = MonthPrepender() month_prepender = MonthPrepender()
llm_full_response_aggregator = LLMFullResponseAggregator()
pipeline = Pipeline([ # With `SyncParallelPipeline` we synchronize audio and images by pushing
llm, # LLM # them basically in order (e.g. I1 A1 A1 A1 I2 A2 A2 A2 A2 I3 A3). To do
sentence_aggregator, # Aggregates LLM output into full sentences # that, each pipeline runs concurrently and `SyncParallelPipeline` will
ParallelTask( # Run pipelines in parallel aggregating the result # wait for the input frame to be processed.
[month_prepender, tts], # Create "Month: sentence" and output audio #
[llm_full_response_aggregator, imagegen] # Aggregate full LLM response # Note that `SyncParallelPipeline` requires the last processor in each
), # of the pipelines to be synchronous. In this case, we use
gated_aggregator, # Queues everything until an image is available # `CartesiaHttpTTSService` and `FalImageGenService` which make HTTP
transport.output() # Transport output # requests and wait for the response.
]) pipeline = Pipeline(
[
llm, # LLM
sentence_aggregator, # Aggregates LLM output into full sentences
SyncParallelPipeline( # Run pipelines in parallel aggregating the result
[month_prepender, tts], # Create "Month: sentence" and output audio
[imagegen], # Generate image
),
transport.output(), # Transport output
]
)
frames = [] frames = []
for month in [ for month in [

86
examples/foundational/05a-local-sync-speech-and-image.py
View File

@@ -11,18 +11,25 @@ import sys
import tkinter as tk import tkinter as tk
from pipecat.frames.frames import AudioRawFrame, Frame, URLImageRawFrame, LLMMessagesFrame, TextFrame from pipecat.frames.frames import (
from pipecat.pipeline.parallel_pipeline import ParallelPipeline Frame,
OutputAudioRawFrame,
TTSAudioRawFrame,
URLImageRawFrame,
LLMMessagesFrame,
TextFrame,
)
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.sync_parallel_pipeline import SyncParallelPipeline
from pipecat.pipeline.task import PipelineTask from pipecat.pipeline.task import PipelineTask
from pipecat.processors.aggregators.llm_response import LLMFullResponseAggregator from pipecat.processors.aggregators.sentence import SentenceAggregator
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.services.cartesia import CartesiaHttpTTSService
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
from pipecat.services.elevenlabs import ElevenLabsTTSService
from pipecat.services.fal import FalImageGenService from pipecat.services.fal import FalImageGenService
from pipecat.transports.base_transport import TransportParams from pipecat.transports.base_transport import TransportParams
from pipecat.transports.local.tk import TkLocalTransport from pipecat.transports.local.tk import TkLocalTransport, TkOutputTransport
from loguru import logger from loguru import logger
@@ -42,7 +49,12 @@ async def main():
runner = PipelineRunner() runner = PipelineRunner()
async def get_month_data(month): async def get_month_data(month):
messages = [{"role": "system", "content": f"Describe a nature photograph suitable for use in a calendar, for the month of {month}. Include only the image description with no preamble. Limit the description to one sentence, please.", }] messages = [
{
"role": "system",
"content": f"Describe a nature photograph suitable for use in a calendar, for the month of {month}. Include only the image description with no preamble. Limit the description to one sentence, please.",
}
]
class ImageDescription(FrameProcessor): class ImageDescription(FrameProcessor):
def __init__(self): def __init__(self):
@@ -60,14 +72,17 @@ async def main():
def __init__(self): def __init__(self):
super().__init__() super().__init__()
self.audio = bytearray() self.audio = bytearray()
self.frame = None
async def process_frame(self, frame: Frame, direction: FrameDirection): async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction) await super().process_frame(frame, direction)
if isinstance(frame, AudioRawFrame): if isinstance(frame, TTSAudioRawFrame):
self.audio.extend(frame.audio) self.audio.extend(frame.audio)
self.frame = AudioRawFrame( self.frame = OutputAudioRawFrame(
bytes(self.audio), frame.sample_rate, frame.num_channels) bytes(self.audio), frame.sample_rate, frame.num_channels
)
await self.push_frame(frame, direction)
class ImageGrabber(FrameProcessor): class ImageGrabber(FrameProcessor):
def __init__(self): def __init__(self):
@@ -79,23 +94,22 @@ async def main():
if isinstance(frame, URLImageRawFrame): if isinstance(frame, URLImageRawFrame):
self.frame = frame self.frame = frame
await self.push_frame(frame, direction)
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
tts = ElevenLabsTTSService( tts = CartesiaHttpTTSService(
api_key=os.getenv("ELEVENLABS_API_KEY"), api_key=os.getenv("CARTESIA_API_KEY"),
voice_id=os.getenv("ELEVENLABS_VOICE_ID")) voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
)
imagegen = FalImageGenService( imagegen = FalImageGenService(
params=FalImageGenService.InputParams( params=FalImageGenService.InputParams(image_size="square_hd"),
image_size="square_hd"
),
aiohttp_session=session, aiohttp_session=session,
key=os.getenv("FAL_KEY")) key=os.getenv("FAL_KEY"),
)
aggregator = LLMFullResponseAggregator() sentence_aggregator = SentenceAggregator()
description = ImageDescription() description = ImageDescription()
@@ -103,13 +117,27 @@ async def main():
image_grabber = ImageGrabber() image_grabber = ImageGrabber()
pipeline = Pipeline([ # With `SyncParallelPipeline` we synchronize audio and images by
llm, # pushing them basically in order (e.g. I1 A1 A1 A1 I2 A2 A2 A2 A2
aggregator, # I3 A3). To do that, each pipeline runs concurrently and
description, # `SyncParallelPipeline` will wait for the input frame to be
ParallelPipeline([tts, audio_grabber], # processed.
[imagegen, image_grabber]) #
]) # Note that `SyncParallelPipeline` requires the last processor in
# each of the pipelines to be synchronous. In this case, we use
# `CartesiaHttpTTSService` and `FalImageGenService` which make HTTP
# requests and wait for the response.
pipeline = Pipeline(
[
llm, # LLM
sentence_aggregator, # Aggregates LLM output into full sentences
description, # Store sentence
SyncParallelPipeline(
[tts, audio_grabber], # Generate and store audio for the given sentence
[imagegen, image_grabber], # Generate and storeimage for the given sentence
),
]
)
task = PipelineTask(pipeline) task = PipelineTask(pipeline)
await task.queue_frame(LLMMessagesFrame(messages)) await task.queue_frame(LLMMessagesFrame(messages))
@@ -130,7 +158,9 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
camera_out_enabled=True, camera_out_enabled=True,
camera_out_width=1024, camera_out_width=1024,
camera_out_height=1024)) camera_out_height=1024,
),
)
pipeline = Pipeline([transport.output()]) pipeline = Pipeline([transport.output()])

60
examples/foundational/06-listen-and-respond.py
View File

@@ -10,6 +10,12 @@ import os
import sys import sys
from pipecat.frames.frames import Frame, LLMMessagesFrame, MetricsFrame from pipecat.frames.frames import Frame, LLMMessagesFrame, MetricsFrame
from pipecat.metrics.metrics import (
TTFBMetricsData,
ProcessingMetricsData,
LLMUsageMetricsData,
TTSUsageMetricsData,
)
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -28,6 +34,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -37,8 +44,20 @@ logger.add(sys.stderr, level="DEBUG")
class MetricsLogger(FrameProcessor): class MetricsLogger(FrameProcessor):
async def process_frame(self, frame: Frame, direction: FrameDirection): async def process_frame(self, frame: Frame, direction: FrameDirection):
if isinstance(frame, MetricsFrame): if isinstance(frame, MetricsFrame):
print( for d in frame.data:
f"!!! MetricsFrame: {frame}, ttfb: {frame.ttfb}, processing: {frame.processing}, tokens: {frame.tokens}, characters: {frame.characters}") if isinstance(d, TTFBMetricsData):
print(f"!!! MetricsFrame: {frame}, ttfb: {d.value}")
elif isinstance(d, ProcessingMetricsData):
print(f"!!! MetricsFrame: {frame}, processing: {d.value}")
elif isinstance(d, LLMUsageMetricsData):
tokens = d.value
print(
f"!!! MetricsFrame: {frame}, tokens: {
tokens.prompt_tokens}, characters: {
tokens.completion_tokens}"
)
elif isinstance(d, TTSUsageMetricsData):
print(f"!!! MetricsFrame: {frame}, characters: {d.value}")
await self.push_frame(frame, direction) await self.push_frame(frame, direction)
@@ -54,8 +73,8 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
tts = CartesiaTTSService( tts = CartesiaTTSService(
@@ -63,10 +82,7 @@ async def main():
voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o"
)
ml = MetricsLogger() ml = MetricsLogger()
@@ -79,29 +95,25 @@ async def main():
tma_in = LLMUserResponseAggregator(messages) tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), [
tma_in, transport.input(),
llm, tma_in,
tts, llm,
ml, tts,
transport.output(), ml,
tma_out, transport.output(),
]) tma_out,
]
)
task = PipelineTask(pipeline) task = PipelineTask(pipeline)
task = PipelineTask(pipeline, PipelineParams(
allow_interruptions=True,
enable_metrics=True,
report_only_initial_ttfb=False,
))
@transport.event_handler("on_first_participant_joined") @transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant): async def on_first_participant_joined(transport, participant):
transport.capture_participant_transcription(participant["id"]) transport.capture_participant_transcription(participant["id"])
# Kick off the conversation. # Kick off the conversation.
messages.append( messages.append({"role": "system", "content": "Please introduce yourself to the user."})
{"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)]) await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner() runner = PipelineRunner()

55
examples/foundational/06a-image-sync.py
View File

@@ -11,7 +11,7 @@ import sys
from PIL import Image from PIL import Image
from pipecat.frames.frames import ImageRawFrame, Frame, SystemFrame, TextFrame from pipecat.frames.frames import Frame, OutputImageRawFrame, SystemFrame, TextFrame
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask from pipecat.pipeline.task import PipelineTask
@@ -20,8 +20,8 @@ from pipecat.processors.aggregators.llm_response import (
LLMUserResponseAggregator, LLMUserResponseAggregator,
) )
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.services.cartesia import CartesiaHttpTTSService
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
from pipecat.services.elevenlabs import ElevenLabsTTSService
from pipecat.transports.services.daily import DailyTransport from pipecat.transports.services.daily import DailyTransport
from pipecat.vad.silero import SileroVADAnalyzer from pipecat.vad.silero import SileroVADAnalyzer
@@ -31,6 +31,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -52,9 +53,21 @@ class ImageSyncAggregator(FrameProcessor):
await super().process_frame(frame, direction) await super().process_frame(frame, direction)
if not isinstance(frame, SystemFrame) and direction == FrameDirection.DOWNSTREAM: if not isinstance(frame, SystemFrame) and direction == FrameDirection.DOWNSTREAM:
await self.push_frame(ImageRawFrame(image=self._speaking_image_bytes, size=(1024, 1024), format=self._speaking_image_format)) await self.push_frame(
OutputImageRawFrame(
image=self._speaking_image_bytes,
size=(1024, 1024),
format=self._speaking_image_format,
)
)
await self.push_frame(frame) await self.push_frame(frame)
await self.push_frame(ImageRawFrame(image=self._waiting_image_bytes, size=(1024, 1024), format=self._waiting_image_format)) await self.push_frame(
OutputImageRawFrame(
image=self._waiting_image_bytes,
size=(1024, 1024),
format=self._waiting_image_format,
)
)
else: else:
await self.push_frame(frame) await self.push_frame(frame)
@@ -75,17 +88,15 @@ async def main():
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(), vad_analyzer=SileroVADAnalyzer(),
) ),
) )
tts = ElevenLabsTTSService( tts = CartesiaHttpTTSService(
api_key=os.getenv("ELEVENLABS_API_KEY"), api_key=os.getenv("CARTESIA_API_KEY"),
voice_id=os.getenv("ELEVENLABS_VOICE_ID"), voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
messages = [ messages = [
{ {
@@ -102,21 +113,23 @@ async def main():
os.path.join(os.path.dirname(__file__), "assets", "waiting.png"), os.path.join(os.path.dirname(__file__), "assets", "waiting.png"),
) )
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), [
image_sync_aggregator, transport.input(),
tma_in, image_sync_aggregator,
llm, tma_in,
tts, llm,
transport.output(), tts,
tma_out transport.output(),
]) tma_out,
]
)
task = PipelineTask(pipeline) task = PipelineTask(pipeline)
@transport.event_handler("on_first_participant_joined") @transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant): async def on_first_participant_joined(transport, participant):
participant_name = participant["info"]["userName"] or '' participant_name = participant["info"]["userName"] or ""
transport.capture_participant_transcription(participant["id"]) transport.capture_participant_transcription(participant["id"])
await task.queue_frames([TextFrame(f"Hi there {participant_name}!")]) await task.queue_frames([TextFrame(f"Hi there {participant_name}!")])

49
examples/foundational/07-interruptible.py
View File

@@ -14,7 +14,9 @@ from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import ( from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator, LLMUserResponseAggregator) LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.services.cartesia import CartesiaTTSService from pipecat.services.cartesia import CartesiaTTSService
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.transports.services.daily import DailyParams, DailyTransport
@@ -25,6 +27,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -43,8 +46,8 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
tts = CartesiaTTSService( tts = CartesiaTTSService(
@@ -52,9 +55,7 @@ async def main():
voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
messages = [ messages = [
{ {
@@ -66,28 +67,32 @@ async def main():
tma_in = LLMUserResponseAggregator(messages) tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Transport user input [
tma_in, # User responses transport.input(), # Transport user input
llm, # LLM tma_in, # User responses
tts, # TTS llm, # LLM
transport.output(), # Transport bot output tts, # TTS
tma_out # Assistant spoken responses transport.output(), # Transport bot output
]) tma_out, # Assistant spoken responses
]
)
task = PipelineTask(pipeline, PipelineParams( task = PipelineTask(
allow_interruptions=True, pipeline,
enable_metrics=True, PipelineParams(
enable_usage_metrics=True, allow_interruptions=True,
report_only_initial_ttfb=True, enable_metrics=True,
)) enable_usage_metrics=True,
report_only_initial_ttfb=True,
),
)
@transport.event_handler("on_first_participant_joined") @transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant): async def on_first_participant_joined(transport, participant):
transport.capture_participant_transcription(participant["id"]) transport.capture_participant_transcription(participant["id"])
# Kick off the conversation. # Kick off the conversation.
messages.append( messages.append({"role": "system", "content": "Please introduce yourself to the user."})
{"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)]) await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner() runner = PipelineRunner()

46
examples/foundational/07a-interruptible-anthropic.py
View File

@@ -5,26 +5,24 @@
# #
import asyncio import asyncio
import aiohttp
import os import os
import sys import sys
import aiohttp
from dotenv import load_dotenv
from loguru import logger
from runner import configure
from pipecat.frames.frames import LLMMessagesFrame from pipecat.frames.frames import LLMMessagesFrame
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import ( from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
LLMAssistantResponseAggregator, LLMUserResponseAggregator)
from pipecat.services.cartesia import CartesiaTTSService
from pipecat.services.anthropic import AnthropicLLMService from pipecat.services.anthropic import AnthropicLLMService
from pipecat.services.cartesia import CartesiaTTSService
from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.transports.services.daily import DailyParams, DailyTransport
from pipecat.vad.silero import SileroVADAnalyzer from pipecat.vad.silero import SileroVADAnalyzer
from runner import configure
from loguru import logger
from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -43,8 +41,8 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
tts = CartesiaTTSService( tts = CartesiaTTSService(
@@ -53,8 +51,8 @@ async def main():
) )
llm = AnthropicLLMService( llm = AnthropicLLMService(
api_key=os.getenv("ANTHROPIC_API_KEY"), api_key=os.getenv("ANTHROPIC_API_KEY"), model="claude-3-opus-20240229"
model="claude-3-opus-20240229") )
# todo: think more about how to handle system prompts in a more general way. OpenAI, # todo: think more about how to handle system prompts in a more general way. OpenAI,
# Google, and Anthropic all have slightly different approaches to providing a system # Google, and Anthropic all have slightly different approaches to providing a system
@@ -66,17 +64,19 @@ async def main():
}, },
] ]
tma_in = LLMUserResponseAggregator(messages) context = OpenAILLMContext(messages)
tma_out = LLMAssistantResponseAggregator(messages) context_aggregator = llm.create_context_aggregator(context)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Transport user input [
tma_in, # User responses transport.input(), # Transport user input
llm, # LLM context_aggregator.user(), # User responses
tts, # TTS llm, # LLM
transport.output(), # Transport bot output tts, # TTS
tma_out # Assistant spoken responses transport.output(), # Transport bot output
]) context_aggregator.assistant(), # Assistant spoken responses
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))

38
examples/foundational/07b-interruptible-langchain.py
View File

@@ -15,7 +15,9 @@ from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import ( from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator, LLMUserResponseAggregator) LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.processors.frameworks.langchain import LangchainProcessor from pipecat.processors.frameworks.langchain import LangchainProcessor
from pipecat.services.cartesia import CartesiaTTSService from pipecat.services.cartesia import CartesiaTTSService
from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.transports.services.daily import DailyParams, DailyTransport
@@ -32,6 +34,7 @@ from loguru import logger
from runner import configure from runner import configure
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
@@ -70,19 +73,22 @@ async def main():
prompt = ChatPromptTemplate.from_messages( prompt = ChatPromptTemplate.from_messages(
[ [
("system", (
"Be nice and helpful. Answer very briefly and without special characters like `#` or `*`. " "system",
"Your response will be synthesized to voice and those characters will create unnatural sounds.", "Be nice and helpful. Answer very briefly and without special characters like `#` or `*`. "
), "Your response will be synthesized to voice and those characters will create unnatural sounds.",
),
MessagesPlaceholder("chat_history"), MessagesPlaceholder("chat_history"),
("human", "{input}"), ("human", "{input}"),
]) ]
)
chain = prompt | ChatOpenAI(model="gpt-4o", temperature=0.7) chain = prompt | ChatOpenAI(model="gpt-4o", temperature=0.7)
history_chain = RunnableWithMessageHistory( history_chain = RunnableWithMessageHistory(
chain, chain,
get_session_history, get_session_history,
history_messages_key="chat_history", history_messages_key="chat_history",
input_messages_key="input") input_messages_key="input",
)
lc = LangchainProcessor(history_chain) lc = LangchainProcessor(history_chain)
tma_in = LLMUserResponseAggregator() tma_in = LLMUserResponseAggregator()
@@ -90,12 +96,12 @@ async def main():
pipeline = Pipeline( pipeline = Pipeline(
[ [
transport.input(), # Transport user input transport.input(), # Transport user input
tma_in, # User responses tma_in, # User responses
lc, # Langchain lc, # Langchain
tts, # TTS tts, # TTS
transport.output(), # Transport bot output transport.output(), # Transport bot output
tma_out, # Assistant spoken responses tma_out, # Assistant spoken responses
] ]
) )
@@ -109,11 +115,7 @@ async def main():
# the `LLMMessagesFrame` will be picked up by the LangchainProcessor using # the `LLMMessagesFrame` will be picked up by the LangchainProcessor using
# only the content of the last message to inject it in the prompt defined # only the content of the last message to inject it in the prompt defined
# above. So no role is required here. # above. So no role is required here.
messages = [( messages = [({"content": "Please briefly introduce yourself to the user."})]
{
"content": "Please briefly introduce yourself to the user."
}
)]
await task.queue_frames([LLMMessagesFrame(messages)]) await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner() runner = PipelineRunner()

52
examples/foundational/07c-interruptible-deepgram.py
View File

@@ -5,26 +5,27 @@
# #
import asyncio import asyncio
import aiohttp
import os import os
import sys import sys
import aiohttp
from dotenv import load_dotenv
from loguru import logger
from runner import configure
from pipecat.frames.frames import LLMMessagesFrame from pipecat.frames.frames import LLMMessagesFrame
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import ( from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator, LLMUserResponseAggregator) LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.services.deepgram import DeepgramSTTService, DeepgramTTSService from pipecat.services.deepgram import DeepgramSTTService, DeepgramTTSService
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.transports.services.daily import DailyParams, DailyTransport
from pipecat.vad.silero import SileroVADAnalyzer from pipecat.vad.silero import SileroVADAnalyzer
from runner import configure
from loguru import logger
from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -43,21 +44,15 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(), vad_analyzer=SileroVADAnalyzer(),
vad_audio_passthrough=True vad_audio_passthrough=True,
) ),
) )
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY")) stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
tts = DeepgramTTSService( tts = DeepgramTTSService(api_key=os.getenv("DEEPGRAM_API_KEY"), voice="aura-helios-en")
aiohttp_session=session,
api_key=os.getenv("DEEPGRAM_API_KEY"),
voice="aura-helios-en"
)
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
messages = [ messages = [
{ {
@@ -69,15 +64,17 @@ async def main():
tma_in = LLMUserResponseAggregator(messages) tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Transport user input [
stt, # STT transport.input(), # Transport user input
tma_in, # User responses stt, # STT
llm, # LLM tma_in, # User responses
tts, # TTS llm, # LLM
transport.output(), # Transport bot output tts, # TTS
tma_out # Assistant spoken responses transport.output(), # Transport bot output
]) tma_out, # Assistant spoken responses
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
@@ -85,8 +82,7 @@ async def main():
async def on_first_participant_joined(transport, participant): async def on_first_participant_joined(transport, participant):
transport.capture_participant_transcription(participant["id"]) transport.capture_participant_transcription(participant["id"])
# Kick off the conversation. # Kick off the conversation.
messages.append( messages.append({"role": "system", "content": "Please introduce yourself to the user."})
{"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)]) await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner() runner = PipelineRunner()

102
examples/foundational/07d-interruptible-elevenlabs.py Normal file
View File

@@ -0,0 +1,102 @@
#
# Copyright (c) 2024, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import asyncio
import os
import sys
import aiohttp
from dotenv import load_dotenv
from loguru import logger
from runner import configure
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.services.elevenlabs import ElevenLabsTTSService
from pipecat.services.openai import OpenAILLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport
from pipecat.vad.silero import SileroVADAnalyzer
load_dotenv(override=True)
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
async def main():
async with aiohttp.ClientSession() as session:
(room_url, token) = await configure(session)
transport = DailyTransport(
room_url,
token,
"Respond bot",
DailyParams(
audio_out_enabled=True,
transcription_enabled=True,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
),
)
tts = ElevenLabsTTSService(
api_key=os.getenv("ELEVENLABS_API_KEY", ""),
voice_id=os.getenv("ELEVENLABS_VOICE_ID", ""),
)
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
messages = [
{
"role": "system",
"content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
},
]
tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline(
[
transport.input(), # Transport user input
tma_in, # User responses
llm, # LLM
tts, # TTS
transport.output(), # Transport bot output
tma_out, # Assistant spoken responses
]
)
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
report_only_initial_ttfb=True,
),
)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner()
await runner.run(task)
if __name__ == "__main__":
asyncio.run(main())

46
examples/foundational/07e-interruptible-playht.py
View File

@@ -4,27 +4,28 @@
# SPDX-License-Identifier: BSD 2-Clause License # SPDX-License-Identifier: BSD 2-Clause License
# #
import aiohttp
import asyncio import asyncio
import os import os
import sys import sys
import aiohttp
from dotenv import load_dotenv
from loguru import logger
from runner import configure
from pipecat.frames.frames import LLMMessagesFrame from pipecat.frames.frames import LLMMessagesFrame
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import ( from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator, LLMUserResponseAggregator) LLMAssistantResponseAggregator,
from pipecat.services.playht import PlayHTTTSService LLMUserResponseAggregator,
)
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
from pipecat.services.playht import PlayHTTTSService
from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.transports.services.daily import DailyParams, DailyTransport
from pipecat.vad.silero import SileroVADAnalyzer from pipecat.vad.silero import SileroVADAnalyzer
from runner import configure
from loguru import logger
from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -44,8 +45,8 @@ async def main():
audio_out_sample_rate=16000, audio_out_sample_rate=16000,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
tts = PlayHTTTSService( tts = PlayHTTTSService(
@@ -54,9 +55,7 @@ async def main():
voice_url="s3://voice-cloning-zero-shot/801a663f-efd0-4254-98d0-5c175514c3e8/jennifer/manifest.json", voice_url="s3://voice-cloning-zero-shot/801a663f-efd0-4254-98d0-5c175514c3e8/jennifer/manifest.json",
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
messages = [ messages = [
{ {
@@ -68,14 +67,16 @@ async def main():
tma_in = LLMUserResponseAggregator(messages) tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Transport user input [
tma_in, # User responses transport.input(), # Transport user input
llm, # LLM tma_in, # User responses
tts, # TTS llm, # LLM
transport.output(), # Transport bot output tts, # TTS
tma_out # Assistant spoken responses transport.output(), # Transport bot output
]) tma_out, # Assistant spoken responses
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
@@ -83,8 +84,7 @@ async def main():
async def on_first_participant_joined(transport, participant): async def on_first_participant_joined(transport, participant):
transport.capture_participant_transcription(participant["id"]) transport.capture_participant_transcription(participant["id"])
# Kick off the conversation. # Kick off the conversation.
messages.append( messages.append({"role": "system", "content": "Please introduce yourself to the user."})
{"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)]) await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner() runner = PipelineRunner()

30
examples/foundational/07f-interruptible-azure.py
View File

@@ -14,7 +14,9 @@ from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import ( from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator, LLMUserResponseAggregator) LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.services.azure import AzureLLMService, AzureSTTService, AzureTTSService from pipecat.services.azure import AzureLLMService, AzureSTTService, AzureTTSService
from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.transports.services.daily import DailyParams, DailyTransport
from pipecat.vad.silero import SileroVADAnalyzer from pipecat.vad.silero import SileroVADAnalyzer
@@ -25,6 +27,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -45,7 +48,7 @@ async def main():
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(), vad_analyzer=SileroVADAnalyzer(),
vad_audio_passthrough=True, vad_audio_passthrough=True,
) ),
) )
stt = AzureSTTService( stt = AzureSTTService(
@@ -74,15 +77,17 @@ async def main():
tma_in = LLMUserResponseAggregator(messages) tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Transport user input [
stt, # STT transport.input(), # Transport user input
tma_in, # User responses stt, # STT
llm, # LLM tma_in, # User responses
tts, # TTS llm, # LLM
transport.output(), # Transport bot output tts, # TTS
tma_out # Assistant spoken responses transport.output(), # Transport bot output
]) tma_out, # Assistant spoken responses
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
@@ -90,8 +95,7 @@ async def main():
async def on_first_participant_joined(transport, participant): async def on_first_participant_joined(transport, participant):
transport.capture_participant_transcription(participant["id"]) transport.capture_participant_transcription(participant["id"])
# Kick off the conversation. # Kick off the conversation.
messages.append( messages.append({"role": "system", "content": "Please introduce yourself to the user."})
{"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)]) await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner() runner = PipelineRunner()

52
examples/foundational/07g-interruptible-openai-tts.py
View File

@@ -4,27 +4,27 @@
# SPDX-License-Identifier: BSD 2-Clause License # SPDX-License-Identifier: BSD 2-Clause License
# #
import aiohttp
import asyncio import asyncio
import os import os
import sys import sys
import aiohttp
from dotenv import load_dotenv
from loguru import logger
from runner import configure
from pipecat.frames.frames import LLMMessagesFrame from pipecat.frames.frames import LLMMessagesFrame
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import ( from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator, LLMUserResponseAggregator) LLMAssistantResponseAggregator,
from pipecat.services.openai import OpenAITTSService LLMUserResponseAggregator,
from pipecat.services.openai import OpenAILLMService )
from pipecat.services.openai import OpenAILLMService, OpenAITTSService
from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.transports.services.daily import DailyParams, DailyTransport
from pipecat.vad.silero import SileroVADAnalyzer from pipecat.vad.silero import SileroVADAnalyzer
from runner import configure
from loguru import logger
from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -44,18 +44,13 @@ async def main():
audio_out_sample_rate=24000, audio_out_sample_rate=24000,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
tts = OpenAITTSService( tts = OpenAITTSService(api_key=os.getenv("OPENAI_API_KEY"), voice="alloy")
api_key=os.getenv("OPENAI_API_KEY"),
voice="alloy"
)
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
messages = [ messages = [
{ {
@@ -67,14 +62,16 @@ async def main():
tma_in = LLMUserResponseAggregator(messages) tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Transport user input [
tma_in, # User responses transport.input(), # Transport user input
llm, # LLM tma_in, # User responses
tts, # TTS llm, # LLM
transport.output(), # Transport bot output tts, # TTS
tma_out # Assistant spoken responses transport.output(), # Transport bot output
]) tma_out, # Assistant spoken responses
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
@@ -82,8 +79,7 @@ async def main():
async def on_first_participant_joined(transport, participant): async def on_first_participant_joined(transport, participant):
transport.capture_participant_transcription(participant["id"]) transport.capture_participant_transcription(participant["id"])
# Kick off the conversation. # Kick off the conversation.
messages.append( messages.append({"role": "system", "content": "Please introduce yourself to the user."})
{"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)]) await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner() runner = PipelineRunner()

30
examples/foundational/07h-interruptible-openpipe.py
View File

@@ -28,6 +28,7 @@ from loguru import logger
import time import time
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -46,8 +47,8 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
tts = CartesiaTTSService( tts = CartesiaTTSService(
@@ -60,9 +61,7 @@ async def main():
api_key=os.getenv("OPENAI_API_KEY"), api_key=os.getenv("OPENAI_API_KEY"),
openpipe_api_key=os.getenv("OPENPIPE_API_KEY"), openpipe_api_key=os.getenv("OPENPIPE_API_KEY"),
model="gpt-4o", model="gpt-4o",
tags={ tags={"conversation_id": f"pipecat-{timestamp}"},
"conversation_id": f"pipecat-{timestamp}"
}
) )
messages = [ messages = [
@@ -74,14 +73,16 @@ async def main():
tma_in = LLMUserResponseAggregator(messages) tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Transport user input [
tma_in, # User responses transport.input(), # Transport user input
llm, # LLM tma_in, # User responses
tts, # TTS llm, # LLM
transport.output(), # Transport bot output tts, # TTS
tma_out # Assistant spoken responses transport.output(), # Transport bot output
]) tma_out, # Assistant spoken responses
]
)
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True)) task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
@@ -89,8 +90,7 @@ async def main():
async def on_first_participant_joined(transport, participant): async def on_first_participant_joined(transport, participant):
transport.capture_participant_transcription(participant["id"]) transport.capture_participant_transcription(participant["id"])
# Kick off the conversation. # Kick off the conversation.
messages.append( messages.append({"role": "system", "content": "Please introduce yourself to the user."})
{"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)]) await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner() runner = PipelineRunner()

34
examples/foundational/07i-interruptible-xtts.py
View File

@@ -14,7 +14,9 @@ from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import ( from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator, LLMUserResponseAggregator) LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.services.deepgram import DeepgramSTTService, DeepgramTTSService from pipecat.services.deepgram import DeepgramSTTService, DeepgramTTSService
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
from pipecat.services.xtts import XTTSService from pipecat.services.xtts import XTTSService
@@ -26,6 +28,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -45,19 +48,17 @@ async def main():
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(), vad_analyzer=SileroVADAnalyzer(),
) ),
) )
tts = XTTSService( tts = XTTSService(
aiohttp_session=session, aiohttp_session=session,
voice_id="Claribel Dervla", voice_id="Claribel Dervla",
language="en", language="en",
base_url="http://localhost:8000" base_url="http://localhost:8000",
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
messages = [ messages = [
{ {
@@ -69,14 +70,16 @@ async def main():
tma_in = LLMUserResponseAggregator(messages) tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Transport user input [
tma_in, # User responses transport.input(), # Transport user input
llm, # LLM tma_in, # User responses
tts, # TTS llm, # LLM
transport.output(), # Transport bot output tts, # TTS
tma_out # Assistant spoken responses transport.output(), # Transport bot output
]) tma_out, # Assistant spoken responses
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
@@ -84,8 +87,7 @@ async def main():
async def on_first_participant_joined(transport, participant): async def on_first_participant_joined(transport, participant):
transport.capture_participant_transcription(participant["id"]) transport.capture_participant_transcription(participant["id"])
# Kick off the conversation. # Kick off the conversation.
messages.append( messages.append({"role": "system", "content": "Please introduce yourself to the user."})
{"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)]) await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner() runner = PipelineRunner()

34
examples/foundational/07j-interruptible-gladia.py
View File

@@ -14,7 +14,9 @@ from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import ( from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator, LLMUserResponseAggregator) LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.services.cartesia import CartesiaTTSService from pipecat.services.cartesia import CartesiaTTSService
from pipecat.services.gladia import GladiaSTTService from pipecat.services.gladia import GladiaSTTService
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
@@ -26,6 +28,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -45,7 +48,7 @@ async def main():
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(), vad_analyzer=SileroVADAnalyzer(),
vad_audio_passthrough=True, vad_audio_passthrough=True,
) ),
) )
stt = GladiaSTTService( stt = GladiaSTTService(
@@ -57,9 +60,7 @@ async def main():
voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
messages = [ messages = [
{ {
@@ -71,15 +72,17 @@ async def main():
tma_in = LLMUserResponseAggregator(messages) tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Transport user input [
stt, # STT transport.input(), # Transport user input
tma_in, # User responses stt, # STT
llm, # LLM tma_in, # User responses
tts, # TTS llm, # LLM
transport.output(), # Transport bot output tts, # TTS
tma_out # Assistant spoken responses transport.output(), # Transport bot output
]) tma_out, # Assistant spoken responses
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
@@ -87,8 +90,7 @@ async def main():
async def on_first_participant_joined(transport, participant): async def on_first_participant_joined(transport, participant):
transport.capture_participant_transcription(participant["id"]) transport.capture_participant_transcription(participant["id"])
# Kick off the conversation. # Kick off the conversation.
messages.append( messages.append({"role": "system", "content": "Please introduce yourself to the user."})
{"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)]) await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner() runner = PipelineRunner()

39
examples/foundational/07k-interruptible-lmnt.py
View File

@@ -14,7 +14,9 @@ from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import ( from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator, LLMUserResponseAggregator) LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.services.lmnt import LmntTTSService from pipecat.services.lmnt import LmntTTSService
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.transports.services.daily import DailyParams, DailyTransport
@@ -25,6 +27,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -44,18 +47,13 @@ async def main():
audio_out_sample_rate=24000, audio_out_sample_rate=24000,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
tts = LmntTTSService( tts = LmntTTSService(api_key=os.getenv("LMNT_API_KEY"), voice_id="morgan")
api_key=os.getenv("LMNT_API_KEY"),
voice_id="morgan"
)
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
messages = [ messages = [
{ {
@@ -67,14 +65,16 @@ async def main():
tma_in = LLMUserResponseAggregator(messages) tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Transport user input [
tma_in, # User responses transport.input(), # Transport user input
llm, # LLM tma_in, # User responses
tts, # TTS llm, # LLM
transport.output(), # Transport bot output tts, # TTS
tma_out # Assistant spoken responses transport.output(), # Transport bot output
]) tma_out, # Assistant spoken responses
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
@@ -82,8 +82,7 @@ async def main():
async def on_first_participant_joined(transport, participant): async def on_first_participant_joined(transport, participant):
transport.capture_participant_transcription(participant["id"]) transport.capture_participant_transcription(participant["id"])
# Kick off the conversation. # Kick off the conversation.
messages.append( messages.append({"role": "system", "content": "Please introduce yourself to the user."})
{"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)]) await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner() runner = PipelineRunner()

109
examples/foundational/07l-interruptible-together.py Normal file
View File

@@ -0,0 +1,109 @@
#
# Copyright (c) 2024, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import asyncio
import os
import sys
import aiohttp
from dotenv import load_dotenv
from loguru import logger
from runner import configure
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.services.ai_services import OpenAILLMContext
from pipecat.services.cartesia import CartesiaTTSService
from pipecat.services.together import TogetherLLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport
from pipecat.vad.silero import SileroVADAnalyzer
load_dotenv(override=True)
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
async def main():
async with aiohttp.ClientSession() as session:
(room_url, token) = await configure(session)
transport = DailyTransport(
room_url,
token,
"Respond bot",
DailyParams(
audio_out_enabled=True,
transcription_enabled=True,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
),
)
tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY"),
voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
)
llm = TogetherLLMService(
api_key=os.getenv("TOGETHER_API_KEY"),
model=os.getenv("TOGETHER_MODEL"),
params=TogetherLLMService.InputParams(
temperature=1.0,
top_p=0.9,
top_k=40,
extra={
"frequency_penalty": 2.0,
"presence_penalty": 0.0,
},
),
)
messages = [
{
"role": "system",
"content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond in plain language. Respond to what the user said in a creative and helpful way.",
},
]
context = OpenAILLMContext(messages)
context_aggregator = llm.create_context_aggregator(context)
user_aggregator = context_aggregator.user()
assistant_aggregator = context_aggregator.assistant()
pipeline = Pipeline(
[
transport.input(), # Transport user input
user_aggregator, # User responses
llm, # LLM
tts, # TTS
transport.output(), # Transport bot output
assistant_aggregator, # Assistant spoken responses
]
)
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True, enable_metrics=True, enable_usage_metrics=True
),
)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner()
await runner.run(task)
if __name__ == "__main__":
asyncio.run(main())

102
examples/foundational/07m-interruptible-aws.py Normal file
View File

@@ -0,0 +1,102 @@
#
# Copyright (c) 2024, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import asyncio
import os
import sys
import aiohttp
from dotenv import load_dotenv
from loguru import logger
from runner import configure
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.services.aws import AWSTTSService
from pipecat.services.deepgram import DeepgramSTTService
from pipecat.services.openai import OpenAILLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport
from pipecat.vad.silero import SileroVADAnalyzer
load_dotenv(override=True)
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
async def main():
async with aiohttp.ClientSession() as session:
(room_url, token) = await configure(session)
transport = DailyTransport(
room_url,
token,
"Respond bot",
DailyParams(
audio_out_enabled=True,
audio_out_sample_rate=16000,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
vad_audio_passthrough=True,
),
)
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
tts = AWSTTSService(
api_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
region=os.getenv("AWS_REGION"),
voice_id="Amy",
params=AWSTTSService.InputParams(engine="neural", language="en-GB", rate="1.05"),
)
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
messages = [
{
"role": "system",
"content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
},
]
tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline(
[
transport.input(), # Transport user input
stt, # STT
tma_in, # User responses
llm, # LLM
tts, # TTS
transport.output(), # Transport bot output
tma_out, # Assistant spoken responses
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner()
await runner.run(task)
if __name__ == "__main__":
asyncio.run(main())

64
examples/foundational/07d-interruptible-cartesia.py → examples/foundational/07n-interruptible-google.py
View File

@@ -4,28 +4,29 @@
# SPDX-License-Identifier: BSD 2-Clause License # SPDX-License-Identifier: BSD 2-Clause License
# #
import aiohttp
import asyncio import asyncio
import os import os
import sys import sys
import aiohttp
from dotenv import load_dotenv
from loguru import logger
from runner import configure
from pipecat.frames.frames import LLMMessagesFrame from pipecat.frames.frames import LLMMessagesFrame
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import ( from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator, LLMUserResponseAggregator) LLMAssistantResponseAggregator,
from pipecat.services.cartesia import CartesiaTTSService LLMUserResponseAggregator,
)
from pipecat.services.deepgram import DeepgramSTTService
from pipecat.services.google import GoogleTTSService
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.transports.services.daily import DailyParams, DailyTransport
from pipecat.vad.silero import SileroVADAnalyzer from pipecat.vad.silero import SileroVADAnalyzer
from runner import configure
from loguru import logger
from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -41,23 +42,22 @@ async def main():
token, token,
"Respond bot", "Respond bot",
DailyParams( DailyParams(
audio_out_sample_rate=44100,
audio_out_enabled=True, audio_out_enabled=True,
transcription_enabled=True, audio_out_sample_rate=24000,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) vad_audio_passthrough=True,
),
) )
tts = CartesiaTTSService( stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
api_key=os.getenv("CARTESIA_API_KEY"),
voice_id="a0e99841-438c-4a64-b679-ae501e7d6091", # Barbershop Man tts = GoogleTTSService(
sample_rate=44100, voice_id="en-US-Neural2-J",
params=GoogleTTSService.InputParams(language="en-US", rate="1.05"),
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
messages = [ messages = [
{ {
@@ -69,23 +69,25 @@ async def main():
tma_in = LLMUserResponseAggregator(messages) tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Transport user input [
tma_in, # User responses transport.input(), # Transport user input
llm, # LLM stt, # STT
tts, # TTS tma_in, # User responses
tma_out, # Goes before the transport because cartesia has word-level timestamps! llm, # LLM
transport.output(), # Transport bot output tts, # TTS
]) transport.output(), # Transport bot output
tma_out, # Assistant spoken responses
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True, enable_metrics=True)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
@transport.event_handler("on_first_participant_joined") @transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant): async def on_first_participant_joined(transport, participant):
transport.capture_participant_transcription(participant["id"]) transport.capture_participant_transcription(participant["id"])
# Kick off the conversation. # Kick off the conversation.
messages.append( messages.append({"role": "system", "content": "Please introduce yourself to the user."})
{"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)]) await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner() runner = PipelineRunner()

23
examples/foundational/08-bots-arguing.py
View File

@@ -3,18 +3,19 @@ import aiohttp
import asyncio import asyncio
import logging import logging
import os import os
from pipecat.pipeline.aggregators import SentenceAggregator from pipecat.processors.aggregators import SentenceAggregator
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.transports.daily_transport import DailyTransport from pipecat.transports.services.daily import DailyTransport
from pipecat.services.azure_ai_services import AzureLLMService, AzureTTSService from pipecat.services.azure import AzureLLMService, AzureTTSService
from pipecat.services.elevenlabs_ai_services import ElevenLabsTTSService from pipecat.services.elevenlabs import ElevenLabsTTSService
from pipecat.services.fal_ai_services import FalImageGenService from pipecat.services.fal import FalImageGenService
from pipecat.pipeline.frames import AudioFrame, EndFrame, ImageFrame, LLMMessagesFrame, TextFrame from pipecat.frames.frames import AudioFrame, EndFrame, ImageFrame, LLMMessagesFrame, TextFrame
from runner import configure from runner import configure
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logging.basicConfig(format=f"%(levelno)s %(asctime)s %(message)s") logging.basicConfig(format=f"%(levelno)s %(asctime)s %(message)s")
@@ -53,9 +54,7 @@ async def main():
voice_id="jBpfuIE2acCO8z3wKNLl", voice_id="jBpfuIE2acCO8z3wKNLl",
) )
dalle = FalImageGenService( dalle = FalImageGenService(
params=FalImageGenService.InputParams( params=FalImageGenService.InputParams(image_size="1024x1024"),
image_size="1024x1024"
),
aiohttp_session=session, aiohttp_session=session,
key=os.getenv("FAL_KEY"), key=os.getenv("FAL_KEY"),
) )
@@ -75,13 +74,11 @@ async def main():
async def get_text_and_audio(messages) -> Tuple[str, bytearray]: async def get_text_and_audio(messages) -> Tuple[str, bytearray]:
"""This function streams text from the LLM and uses the TTS service to convert """This function streams text from the LLM and uses the TTS service to convert
that text to speech as it's received. """ that text to speech as it's received."""
source_queue = asyncio.Queue() source_queue = asyncio.Queue()
sink_queue = asyncio.Queue() sink_queue = asyncio.Queue()
sentence_aggregator = SentenceAggregator() sentence_aggregator = SentenceAggregator()
pipeline = Pipeline( pipeline = Pipeline([llm, sentence_aggregator, tts1], source_queue, sink_queue)
[llm, sentence_aggregator, tts1], source_queue, sink_queue
)
await source_queue.put(LLMMessagesFrame(messages)) await source_queue.put(LLMMessagesFrame(messages))
await source_queue.put(EndFrame()) await source_queue.put(EndFrame())

39
examples/foundational/09-mirror.py
View File

@@ -8,9 +8,17 @@ import aiohttp
import asyncio import asyncio
import sys import sys
from pipecat.frames.frames import (
Frame,
InputAudioRawFrame,
InputImageRawFrame,
OutputAudioRawFrame,
OutputImageRawFrame,
)
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask from pipecat.pipeline.task import PipelineTask
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.transports.services.daily import DailyTransport, DailyParams from pipecat.transports.services.daily import DailyTransport, DailyParams
from runner import configure from runner import configure
@@ -18,33 +26,56 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
logger.add(sys.stderr, level="DEBUG") logger.add(sys.stderr, level="DEBUG")
class MirrorProcessor(FrameProcessor):
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
if isinstance(frame, InputAudioRawFrame):
await self.push_frame(
OutputAudioRawFrame(
audio=frame.audio,
sample_rate=frame.sample_rate,
num_channels=frame.num_channels,
)
)
elif isinstance(frame, InputImageRawFrame):
await self.push_frame(
OutputImageRawFrame(image=frame.image, size=frame.size, format=frame.format)
)
else:
await self.push_frame(frame, direction)
async def main(): async def main():
async with aiohttp.ClientSession() as session: async with aiohttp.ClientSession() as session:
(room_url, token) = await configure(session) (room_url, token) = await configure(session)
transport = DailyTransport( transport = DailyTransport(
room_url, token, "Test", room_url,
token,
"Test",
DailyParams( DailyParams(
audio_in_enabled=True, audio_in_enabled=True,
audio_out_enabled=True, audio_out_enabled=True,
camera_out_enabled=True, camera_out_enabled=True,
camera_out_is_live=True, camera_out_is_live=True,
camera_out_width=1280, camera_out_width=1280,
camera_out_height=720 camera_out_height=720,
) ),
) )
@transport.event_handler("on_first_participant_joined") @transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant): async def on_first_participant_joined(transport, participant):
transport.capture_participant_video(participant["id"]) transport.capture_participant_video(participant["id"])
pipeline = Pipeline([transport.input(), transport.output()]) pipeline = Pipeline([transport.input(), MirrorProcessor(), transport.output()])
runner = PipelineRunner() runner = PipelineRunner()

39
examples/foundational/09a-local-mirror.py
View File

@@ -10,9 +10,17 @@ import sys
import tkinter as tk import tkinter as tk
from pipecat.frames.frames import (
Frame,
InputAudioRawFrame,
InputImageRawFrame,
OutputAudioRawFrame,
OutputImageRawFrame,
)
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask from pipecat.pipeline.task import PipelineTask
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.transports.base_transport import TransportParams from pipecat.transports.base_transport import TransportParams
from pipecat.transports.local.tk import TkLocalTransport from pipecat.transports.local.tk import TkLocalTransport
from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.transports.services.daily import DailyParams, DailyTransport
@@ -22,12 +30,33 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
logger.add(sys.stderr, level="DEBUG") logger.add(sys.stderr, level="DEBUG")
class MirrorProcessor(FrameProcessor):
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
if isinstance(frame, InputAudioRawFrame):
await self.push_frame(
OutputAudioRawFrame(
audio=frame.audio,
sample_rate=frame.sample_rate,
num_channels=frame.num_channels,
)
)
elif isinstance(frame, InputImageRawFrame):
await self.push_frame(
OutputImageRawFrame(image=frame.image, size=frame.size, format=frame.format)
)
else:
await self.push_frame(frame, direction)
async def main(): async def main():
async with aiohttp.ClientSession() as session: async with aiohttp.ClientSession() as session:
(room_url, token) = await configure(session) (room_url, token) = await configure(session)
@@ -36,8 +65,8 @@ async def main():
tk_root.title("Local Mirror") tk_root.title("Local Mirror")
daily_transport = DailyTransport( daily_transport = DailyTransport(
room_url, token, "Test", DailyParams( room_url, token, "Test", DailyParams(audio_in_enabled=True)
audio_in_enabled=True)) )
tk_transport = TkLocalTransport( tk_transport = TkLocalTransport(
tk_root, tk_root,
@@ -46,13 +75,15 @@ async def main():
camera_out_enabled=True, camera_out_enabled=True,
camera_out_is_live=True, camera_out_is_live=True,
camera_out_width=1280, camera_out_width=1280,
camera_out_height=720)) camera_out_height=720,
),
)
@daily_transport.event_handler("on_first_participant_joined") @daily_transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant): async def on_first_participant_joined(transport, participant):
transport.capture_participant_video(participant["id"]) transport.capture_participant_video(participant["id"])
pipeline = Pipeline([daily_transport.input(), tk_transport.output()]) pipeline = Pipeline([daily_transport.input(), MirrorProcessor(), tk_transport.output()])
task = PipelineTask(pipeline) task = PipelineTask(pipeline)

33
examples/foundational/10-wake-phrase.py
View File

@@ -14,7 +14,9 @@ from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import ( from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator, LLMUserResponseAggregator) LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.services.cartesia import CartesiaTTSService from pipecat.services.cartesia import CartesiaTTSService
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.transports.services.daily import DailyParams, DailyTransport
@@ -25,6 +27,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -43,8 +46,8 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
tts = CartesiaTTSService( tts = CartesiaTTSService(
@@ -52,9 +55,7 @@ async def main():
voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
messages = [ messages = [
{ {
@@ -67,15 +68,17 @@ async def main():
tma_in = LLMUserResponseAggregator(messages) tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Transport user input [
hey_robot_filter, # Filter out speech not directed at the robot transport.input(), # Transport user input
tma_in, # User responses hey_robot_filter, # Filter out speech not directed at the robot
llm, # LLM tma_in, # User responses
tts, # TTS llm, # LLM
transport.output(), # Transport bot output tts, # TTS
tma_out # Assistant spoken responses transport.output(), # Transport bot output
]) tma_out, # Assistant spoken responses
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))

52
examples/foundational/11-sound-effects.py
View File

@@ -12,9 +12,9 @@ import wave
from pipecat.frames.frames import ( from pipecat.frames.frames import (
Frame, Frame,
AudioRawFrame,
LLMFullResponseEndFrame, LLMFullResponseEndFrame,
LLMMessagesFrame, LLMMessagesFrame,
OutputAudioRawFrame,
) )
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
@@ -25,7 +25,7 @@ from pipecat.processors.aggregators.llm_response import (
) )
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.processors.logger import FrameLogger from pipecat.processors.logger import FrameLogger
from pipecat.services.elevenlabs import ElevenLabsTTSService from pipecat.services.cartesia import CartesiaHttpTTSService
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.transports.services.daily import DailyParams, DailyTransport
from pipecat.vad.silero import SileroVADAnalyzer from pipecat.vad.silero import SileroVADAnalyzer
@@ -35,6 +35,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -53,12 +54,12 @@ for file in sound_files:
filename = os.path.splitext(os.path.basename(full_path))[0] filename = os.path.splitext(os.path.basename(full_path))[0]
# Open the image and convert it to bytes # Open the image and convert it to bytes
with wave.open(full_path) as audio_file: with wave.open(full_path) as audio_file:
sounds[file] = AudioRawFrame(audio_file.readframes(-1), sounds[file] = OutputAudioRawFrame(
audio_file.getframerate(), audio_file.getnchannels()) audio_file.readframes(-1), audio_file.getframerate(), audio_file.getnchannels()
)
class OutboundSoundEffectWrapper(FrameProcessor): class OutboundSoundEffectWrapper(FrameProcessor):
async def process_frame(self, frame: Frame, direction: FrameDirection): async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction) await super().process_frame(frame, direction)
@@ -71,7 +72,6 @@ class OutboundSoundEffectWrapper(FrameProcessor):
class InboundSoundEffectWrapper(FrameProcessor): class InboundSoundEffectWrapper(FrameProcessor):
async def process_frame(self, frame: Frame, direction: FrameDirection): async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction) await super().process_frame(frame, direction)
@@ -95,17 +95,15 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
tts = ElevenLabsTTSService( tts = CartesiaHttpTTSService(
api_key=os.getenv("ELEVENLABS_API_KEY"), api_key=os.getenv("CARTESIA_API_KEY"),
voice_id="ErXwobaYiN019PkySvjV", voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
) )
messages = [ messages = [
@@ -122,18 +120,20 @@ async def main():
fl = FrameLogger("LLM Out") fl = FrameLogger("LLM Out")
fl2 = FrameLogger("Transcription In") fl2 = FrameLogger("Transcription In")
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), [
tma_in, transport.input(),
in_sound, tma_in,
fl2, in_sound,
llm, fl2,
fl, llm,
tts, fl,
out_sound, tts,
transport.output(), out_sound,
tma_out transport.output(),
]) tma_out,
]
)
@transport.event_handler("on_first_participant_joined") @transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant): async def on_first_participant_joined(transport, participant):

31
examples/foundational/12-describe-video.py
View File

@@ -26,6 +26,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -33,7 +34,6 @@ logger.add(sys.stderr, level="DEBUG")
class UserImageRequester(FrameProcessor): class UserImageRequester(FrameProcessor):
def __init__(self, participant_id: str | None = None): def __init__(self, participant_id: str | None = None):
super().__init__() super().__init__()
self._participant_id = participant_id self._participant_id = participant_id
@@ -45,7 +45,9 @@ class UserImageRequester(FrameProcessor):
await super().process_frame(frame, direction) await super().process_frame(frame, direction)
if self._participant_id and isinstance(frame, TextFrame): if self._participant_id and isinstance(frame, TextFrame):
await self.push_frame(UserImageRequestFrame(self._participant_id), FrameDirection.UPSTREAM) await self.push_frame(
UserImageRequestFrame(self._participant_id), FrameDirection.UPSTREAM
)
await self.push_frame(frame, direction) await self.push_frame(frame, direction)
@@ -61,8 +63,8 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
user_response = UserResponseAggregator() user_response = UserResponseAggregator()
@@ -86,15 +88,17 @@ async def main():
transport.capture_participant_transcription(participant["id"]) transport.capture_participant_transcription(participant["id"])
image_requester.set_participant_id(participant["id"]) image_requester.set_participant_id(participant["id"])
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), [
user_response, transport.input(),
image_requester, user_response,
vision_aggregator, image_requester,
moondream, vision_aggregator,
tts, moondream,
transport.output() tts,
]) transport.output(),
]
)
task = PipelineTask(pipeline) task = PipelineTask(pipeline)
@@ -102,5 +106,6 @@ async def main():
await runner.run(task) await runner.run(task)
if __name__ == "__main__": if __name__ == "__main__":
asyncio.run(main()) asyncio.run(main())

35
examples/foundational/12a-describe-video-gemini-flash.py
View File

@@ -26,6 +26,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -33,7 +34,6 @@ logger.add(sys.stderr, level="DEBUG")
class UserImageRequester(FrameProcessor): class UserImageRequester(FrameProcessor):
def __init__(self, participant_id: str | None = None): def __init__(self, participant_id: str | None = None):
super().__init__() super().__init__()
self._participant_id = participant_id self._participant_id = participant_id
@@ -45,7 +45,9 @@ class UserImageRequester(FrameProcessor):
await super().process_frame(frame, direction) await super().process_frame(frame, direction)
if self._participant_id and isinstance(frame, TextFrame): if self._participant_id and isinstance(frame, TextFrame):
await self.push_frame(UserImageRequestFrame(self._participant_id), FrameDirection.UPSTREAM) await self.push_frame(
UserImageRequestFrame(self._participant_id), FrameDirection.UPSTREAM
)
await self.push_frame(frame, direction) await self.push_frame(frame, direction)
@@ -62,8 +64,8 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
user_response = UserResponseAggregator() user_response = UserResponseAggregator()
@@ -73,8 +75,8 @@ async def main():
vision_aggregator = VisionImageFrameAggregator() vision_aggregator = VisionImageFrameAggregator()
google = GoogleLLMService( google = GoogleLLMService(
model="gemini-1.5-flash-latest", model="gemini-1.5-flash-latest", api_key=os.getenv("GOOGLE_API_KEY")
api_key=os.getenv("GOOGLE_API_KEY")) )
tts = CartesiaTTSService( tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY"), api_key=os.getenv("CARTESIA_API_KEY"),
@@ -88,15 +90,17 @@ async def main():
transport.capture_participant_transcription(participant["id"]) transport.capture_participant_transcription(participant["id"])
image_requester.set_participant_id(participant["id"]) image_requester.set_participant_id(participant["id"])
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), [
user_response, transport.input(),
image_requester, user_response,
vision_aggregator, image_requester,
google, vision_aggregator,
tts, google,
transport.output() tts,
]) transport.output(),
]
)
task = PipelineTask(pipeline) task = PipelineTask(pipeline)
@@ -104,5 +108,6 @@ async def main():
await runner.run(task) await runner.run(task)
if __name__ == "__main__": if __name__ == "__main__":
asyncio.run(main()) asyncio.run(main())

36
examples/foundational/12b-describe-video-gpt-4o.py
View File

@@ -26,6 +26,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -33,7 +34,6 @@ logger.add(sys.stderr, level="DEBUG")
class UserImageRequester(FrameProcessor): class UserImageRequester(FrameProcessor):
def __init__(self, participant_id: str | None = None): def __init__(self, participant_id: str | None = None):
super().__init__() super().__init__()
self._participant_id = participant_id self._participant_id = participant_id
@@ -45,7 +45,9 @@ class UserImageRequester(FrameProcessor):
await super().process_frame(frame, direction) await super().process_frame(frame, direction)
if self._participant_id and isinstance(frame, TextFrame): if self._participant_id and isinstance(frame, TextFrame):
await self.push_frame(UserImageRequestFrame(self._participant_id), FrameDirection.UPSTREAM) await self.push_frame(
UserImageRequestFrame(self._participant_id), FrameDirection.UPSTREAM
)
await self.push_frame(frame, direction) await self.push_frame(frame, direction)
@@ -61,8 +63,8 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
user_response = UserResponseAggregator() user_response = UserResponseAggregator()
@@ -71,10 +73,7 @@ async def main():
vision_aggregator = VisionImageFrameAggregator() vision_aggregator = VisionImageFrameAggregator()
openai = OpenAILLMService( openai = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o"
)
tts = CartesiaTTSService( tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY"), api_key=os.getenv("CARTESIA_API_KEY"),
@@ -88,15 +87,17 @@ async def main():
transport.capture_participant_transcription(participant["id"]) transport.capture_participant_transcription(participant["id"])
image_requester.set_participant_id(participant["id"]) image_requester.set_participant_id(participant["id"])
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), [
user_response, transport.input(),
image_requester, user_response,
vision_aggregator, image_requester,
openai, vision_aggregator,
tts, openai,
transport.output() tts,
]) transport.output(),
]
)
task = PipelineTask(pipeline) task = PipelineTask(pipeline)
@@ -104,5 +105,6 @@ async def main():
await runner.run(task) await runner.run(task)
if __name__ == "__main__": if __name__ == "__main__":
asyncio.run(main()) asyncio.run(main())

39
examples/foundational/12c-describe-video-anthropic.py
View File

@@ -26,6 +26,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -33,7 +34,6 @@ logger.add(sys.stderr, level="DEBUG")
class UserImageRequester(FrameProcessor): class UserImageRequester(FrameProcessor):
def __init__(self, participant_id: str | None = None): def __init__(self, participant_id: str | None = None):
super().__init__() super().__init__()
self._participant_id = participant_id self._participant_id = participant_id
@@ -45,7 +45,9 @@ class UserImageRequester(FrameProcessor):
await super().process_frame(frame, direction) await super().process_frame(frame, direction)
if self._participant_id and isinstance(frame, TextFrame): if self._participant_id and isinstance(frame, TextFrame):
await self.push_frame(UserImageRequestFrame(self._participant_id), FrameDirection.UPSTREAM) await self.push_frame(
UserImageRequestFrame(self._participant_id), FrameDirection.UPSTREAM
)
await self.push_frame(frame, direction) await self.push_frame(frame, direction)
@@ -61,8 +63,8 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
user_response = UserResponseAggregator() user_response = UserResponseAggregator()
@@ -71,14 +73,14 @@ async def main():
vision_aggregator = VisionImageFrameAggregator() vision_aggregator = VisionImageFrameAggregator()
anthropic = AnthropicLLMService( anthropic = AnthropicLLMService(api_key=os.getenv("ANTHROPIC_API_KEY"))
api_key=os.getenv("ANTHROPIC_API_KEY")
)
tts = CartesiaTTSService( tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY"), api_key=os.getenv("CARTESIA_API_KEY"),
voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
sample_rate=16000, params=CartesiaTTSService.InputParams(
sample_rate=16000,
),
) )
@transport.event_handler("on_first_participant_joined") @transport.event_handler("on_first_participant_joined")
@@ -88,15 +90,17 @@ async def main():
transport.capture_participant_transcription(participant["id"]) transport.capture_participant_transcription(participant["id"])
image_requester.set_participant_id(participant["id"]) image_requester.set_participant_id(participant["id"])
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), [
user_response, transport.input(),
image_requester, user_response,
vision_aggregator, image_requester,
anthropic, vision_aggregator,
tts, anthropic,
transport.output() tts,
]) transport.output(),
]
)
task = PipelineTask(pipeline) task = PipelineTask(pipeline)
@@ -104,5 +108,6 @@ async def main():
await runner.run(task) await runner.run(task)
if __name__ == "__main__": if __name__ == "__main__":
asyncio.run(main()) asyncio.run(main())

7
examples/foundational/13-whisper-transcription.py
View File

@@ -21,6 +21,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -28,7 +29,6 @@ logger.add(sys.stderr, level="DEBUG")
class TranscriptionLogger(FrameProcessor): class TranscriptionLogger(FrameProcessor):
async def process_frame(self, frame: Frame, direction: FrameDirection): async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction) await super().process_frame(frame, direction)
@@ -40,8 +40,9 @@ async def main():
async with aiohttp.ClientSession() as session: async with aiohttp.ClientSession() as session:
(room_url, _) = await configure(session) (room_url, _) = await configure(session)
transport = DailyTransport(room_url, None, "Transcription bot", transport = DailyTransport(
DailyParams(audio_in_enabled=True)) room_url, None, "Transcription bot", DailyParams(audio_in_enabled=True)
)
stt = WhisperSTTService() stt = WhisperSTTService()

2
examples/foundational/13a-whisper-local.py
View File

@@ -19,6 +19,7 @@ from pipecat.transports.local.audio import LocalAudioTransport
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -26,7 +27,6 @@ logger.add(sys.stderr, level="DEBUG")
class TranscriptionLogger(FrameProcessor): class TranscriptionLogger(FrameProcessor):
async def process_frame(self, frame: Frame, direction: FrameDirection): async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction) await super().process_frame(frame, direction)

7
examples/foundational/13b-deepgram-transcription.py
View File

@@ -22,6 +22,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -29,7 +30,6 @@ logger.add(sys.stderr, level="DEBUG")
class TranscriptionLogger(FrameProcessor): class TranscriptionLogger(FrameProcessor):
async def process_frame(self, frame: Frame, direction: FrameDirection): async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction) await super().process_frame(frame, direction)
@@ -41,8 +41,9 @@ async def main():
async with aiohttp.ClientSession() as session: async with aiohttp.ClientSession() as session:
(room_url, _) = await configure(session) (room_url, _) = await configure(session)
transport = DailyTransport(room_url, None, "Transcription bot", transport = DailyTransport(
DailyParams(audio_in_enabled=True)) room_url, None, "Transcription bot", DailyParams(audio_in_enabled=True)
)
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY")) stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))

59
examples/foundational/14-function-calling.py
View File

@@ -9,11 +9,9 @@ import aiohttp
import os import os
import sys import sys
from pipecat.frames.frames import TextFrame
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask from pipecat.pipeline.task import PipelineTask
from pipecat.processors.logger import FrameLogger
from pipecat.services.cartesia import CartesiaTTSService from pipecat.services.cartesia import CartesiaTTSService
from pipecat.services.openai import OpenAILLMContext, OpenAILLMService from pipecat.services.openai import OpenAILLMContext, OpenAILLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.transports.services.daily import DailyParams, DailyTransport
@@ -26,6 +24,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -33,7 +32,12 @@ logger.add(sys.stderr, level="DEBUG")
async def start_fetch_weather(function_name, llm, context): async def start_fetch_weather(function_name, llm, context):
await llm.push_frame(TextFrame("Let me check on that.")) # note: we can't push a frame to the LLM here. the bot
# can interrupt itself and/or cause audio overlapping glitches.
# possible question for Aleix and Chad about what the right way
# to trigger speech is, now, with the new queues/async/sync refactors.
# await llm.push_frame(TextFrame("Let me check on that."))
logger.debug(f"Starting fetch_weather_from_api with function_name: {function_name}")
async def fetch_weather_from_api(function_name, tool_call_id, args, llm, context, result_callback): async def fetch_weather_from_api(function_name, tool_call_id, args, llm, context, result_callback):
@@ -52,8 +56,8 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
tts = CartesiaTTSService( tts = CartesiaTTSService(
@@ -61,18 +65,10 @@ async def main():
voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
# Register a function_name of None to get all functions # Register a function_name of None to get all functions
# sent to the same callback with an additional function_name parameter. # sent to the same callback with an additional function_name parameter.
llm.register_function( llm.register_function(None, fetch_weather_from_api, start_callback=start_fetch_weather)
None,
fetch_weather_from_api,
start_callback=start_fetch_weather)
fl_in = FrameLogger("Inner")
fl_out = FrameLogger("Outer")
tools = [ tools = [
ChatCompletionToolParam( ChatCompletionToolParam(
@@ -89,17 +85,15 @@ async def main():
}, },
"format": { "format": {
"type": "string", "type": "string",
"enum": [ "enum": ["celsius", "fahrenheit"],
"celsius",
"fahrenheit"],
"description": "The temperature unit to use. Infer this from the users location.", "description": "The temperature unit to use. Infer this from the users location.",
}, },
}, },
"required": [ "required": ["location", "format"],
"location",
"format"],
}, },
})] },
)
]
messages = [ messages = [
{ {
"role": "system", "role": "system",
@@ -110,16 +104,16 @@ async def main():
context = OpenAILLMContext(messages, tools) context = OpenAILLMContext(messages, tools)
context_aggregator = llm.create_context_aggregator(context) context_aggregator = llm.create_context_aggregator(context)
pipeline = Pipeline([ pipeline = Pipeline(
fl_in, [
transport.input(), transport.input(),
context_aggregator.user(), context_aggregator.user(),
llm, llm,
fl_out, tts,
tts, transport.output(),
transport.output(), context_aggregator.assistant(),
context_aggregator.assistant(), ]
]) )
task = PipelineTask(pipeline) task = PipelineTask(pipeline)
@@ -133,5 +127,6 @@ async def main():
await runner.run(task) await runner.run(task)
if __name__ == "__main__": if __name__ == "__main__":
asyncio.run(main()) asyncio.run(main())

28
examples/foundational/19a-tools-anthropic.py → examples/foundational/14a-function-calling-anthropic.py
View File

@@ -23,6 +23,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -46,8 +47,8 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
tts = CartesiaTTSService( tts = CartesiaTTSService(
@@ -56,8 +57,7 @@ async def main():
) )
llm = AnthropicLLMService( llm = AnthropicLLMService(
api_key=os.getenv("ANTHROPIC_API_KEY"), api_key=os.getenv("ANTHROPIC_API_KEY"), model="claude-3-5-sonnet-20240620"
model="claude-3-5-sonnet-20240620"
) )
llm.register_function("get_weather", get_weather) llm.register_function("get_weather", get_weather)
@@ -90,18 +90,20 @@ async def main():
context = OpenAILLMContext(messages, tools) context = OpenAILLMContext(messages, tools)
context_aggregator = llm.create_context_aggregator(context) context_aggregator = llm.create_context_aggregator(context)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Transport user input [
context_aggregator.user(), # User spoken responses transport.input(), # Transport user input
llm, # LLM context_aggregator.user(), # User spoken responses
tts, # TTS llm, # LLM
transport.output(), # Transport bot output tts, # TTS
context_aggregator.assistant(), # Assistant spoken responses and tool context transport.output(), # Transport bot output
]) context_aggregator.assistant(), # Assistant spoken responses and tool context
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True, enable_metrics=True)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True, enable_metrics=True))
@ transport.event_handler("on_first_participant_joined") @transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant): async def on_first_participant_joined(transport, participant):
transport.capture_participant_transcription(participant["id"]) transport.capture_participant_transcription(participant["id"])
# Kick off the conversation. # Kick off the conversation.

37
examples/foundational/19b-tools-video-anthropic.py → examples/foundational/14b-function-calling-anthropic-video.py
View File

@@ -23,6 +23,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -55,8 +56,8 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
tts = CartesiaTTSService( tts = CartesiaTTSService(
@@ -67,7 +68,7 @@ async def main():
llm = AnthropicLLMService( llm = AnthropicLLMService(
api_key=os.getenv("ANTHROPIC_API_KEY"), api_key=os.getenv("ANTHROPIC_API_KEY"),
model="claude-3-5-sonnet-20240620", model="claude-3-5-sonnet-20240620",
enable_prompt_caching_beta=True enable_prompt_caching_beta=True,
) )
llm.register_function("get_weather", get_weather) llm.register_function("get_weather", get_weather)
llm.register_function("get_image", get_image) llm.register_function("get_image", get_image)
@@ -100,7 +101,7 @@ async def main():
}, },
"required": ["question"], "required": ["question"],
}, },
} },
] ]
# todo: test with very short initial user message # todo: test with very short initial user message
@@ -134,28 +135,28 @@ If you need to use a tool, simply use the tool. Do not tell the user the tool yo
"type": "text", "type": "text",
"text": system_prompt, "text": system_prompt,
} }
] ],
}, },
{ {"role": "user", "content": "Start the conversation by introducing yourself."},
"role": "user", ]
"content": "Start the conversation by introducing yourself."
}]
context = OpenAILLMContext(messages, tools) context = OpenAILLMContext(messages, tools)
context_aggregator = llm.create_context_aggregator(context) context_aggregator = llm.create_context_aggregator(context)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Transport user input [
context_aggregator.user(), # User speech to text transport.input(), # Transport user input
llm, # LLM context_aggregator.user(), # User speech to text
tts, # TTS llm, # LLM
transport.output(), # Transport bot output tts, # TTS
context_aggregator.assistant(), # Assistant spoken responses and tool context transport.output(), # Transport bot output
]) context_aggregator.assistant(), # Assistant spoken responses and tool context
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True, enable_metrics=True)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True, enable_metrics=True))
@ transport.event_handler("on_first_participant_joined") @transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant): async def on_first_participant_joined(transport, participant):
global video_participant_id global video_participant_id
video_participant_id = participant["id"] video_participant_id = participant["id"]

136
examples/foundational/14c-function-calling-together.py Normal file
View File

@@ -0,0 +1,136 @@
#
# Copyright (c) 2024, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import asyncio
import aiohttp
import os
import sys
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask
from pipecat.services.cartesia import CartesiaTTSService
from pipecat.services.openai import OpenAILLMContext
from pipecat.services.together import TogetherLLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport
from pipecat.vad.silero import SileroVADAnalyzer
from openai.types.chat import ChatCompletionToolParam
from runner import configure
from loguru import logger
from dotenv import load_dotenv
load_dotenv(override=True)
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
async def start_fetch_weather(function_name, llm, context):
# note: we can't push a frame to the LLM here. the bot
# can interrupt itself and/or cause audio overlapping glitches.
# possible question for Aleix and Chad about what the right way
# to trigger speech is, now, with the new queues/async/sync refactors.
# await llm.push_frame(TextFrame("Let me check on that."))
logger.debug(f"Starting fetch_weather_from_api with function_name: {function_name}")
async def fetch_weather_from_api(function_name, tool_call_id, args, llm, context, result_callback):
await result_callback({"conditions": "nice", "temperature": "75"})
async def main():
async with aiohttp.ClientSession() as session:
(room_url, token) = await configure(session)
transport = DailyTransport(
room_url,
token,
"Respond bot",
DailyParams(
audio_out_enabled=True,
transcription_enabled=True,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
),
)
tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY"),
voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
)
llm = TogetherLLMService(
api_key=os.getenv("TOGETHER_API_KEY"),
model="meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo",
)
# Register a function_name of None to get all functions
# sent to the same callback with an additional function_name parameter.
llm.register_function(None, fetch_weather_from_api, start_callback=start_fetch_weather)
tools = [
ChatCompletionToolParam(
type="function",
function={
"name": "get_current_weather",
"description": "Get the current weather",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA",
},
"format": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "The temperature unit to use. Infer this from the users location.",
},
},
"required": ["location", "format"],
},
},
)
]
messages = [
{
"role": "system",
"content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
},
]
context = OpenAILLMContext(messages, tools)
context_aggregator = llm.create_context_aggregator(context)
pipeline = Pipeline(
[
transport.input(),
context_aggregator.user(),
llm,
tts,
transport.output(),
context_aggregator.assistant(),
]
)
task = PipelineTask(pipeline)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
# await tts.say("Hi! Ask me about the weather in San Francisco.")
runner = PipelineRunner()
await runner.run(task)
if __name__ == "__main__":
asyncio.run(main())

167
examples/foundational/14d-function-calling-video.py Normal file
View File

@@ -0,0 +1,167 @@
#
# Copyright (c) 2024, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import asyncio
import aiohttp
import os
import sys
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask
from pipecat.services.cartesia import CartesiaTTSService
from pipecat.services.openai import OpenAILLMContext, OpenAILLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport
from pipecat.vad.silero import SileroVADAnalyzer
from openai.types.chat import ChatCompletionToolParam
from runner import configure
from loguru import logger
from dotenv import load_dotenv
load_dotenv(override=True)
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
video_participant_id = None
async def get_weather(function_name, tool_call_id, arguments, llm, context, result_callback):
location = arguments["location"]
await result_callback(f"The weather in {location} is currently 72 degrees and sunny.")
async def get_image(function_name, tool_call_id, arguments, llm, context, result_callback):
logger.debug(f"!!! IN get_image {video_participant_id}, {arguments}")
question = arguments["question"]
await llm.request_image_frame(user_id=video_participant_id, text_content=question)
async def main():
async with aiohttp.ClientSession() as session:
(room_url, token) = await configure(session)
transport = DailyTransport(
room_url,
token,
"Respond bot",
DailyParams(
audio_out_enabled=True,
transcription_enabled=True,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
),
)
tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY"),
voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
)
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
llm.register_function("get_weather", get_weather)
llm.register_function("get_image", get_image)
tools = [
ChatCompletionToolParam(
type="function",
function={
"name": "get_weather",
"description": "Get the current weather",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA",
},
"format": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "The temperature unit to use. Infer this from the users location.",
},
},
"required": ["location", "format"],
},
},
),
ChatCompletionToolParam(
type="function",
function={
"name": "get_image",
"description": "Get an image from the video stream.",
"parameters": {
"type": "object",
"properties": {
"question": {
"type": "string",
"description": "The question to ask the AI to generate an image of",
},
},
"required": ["question"],
},
},
),
]
system_prompt = """\
You are a helpful assistant who converses with a user and answers questions. Respond concisely to general questions.
Your response will be turned into speech so use only simple words and punctuation.
You have access to two tools: get_weather and get_image.
You can respond to questions about the weather using the get_weather tool.
You can answer questions about the user's video stream using the get_image tool. Some examples of phrases that \
indicate you should use the get_image tool are:
- What do you see?
- What's in the video?
- Can you describe the video?
- Tell me about what you see.
- Tell me something interesting about what you see.
- What's happening in the video?
"""
messages = [
{"role": "system", "content": system_prompt},
]
context = OpenAILLMContext(messages, tools)
context_aggregator = llm.create_context_aggregator(context)
pipeline = Pipeline(
[
transport.input(),
context_aggregator.user(),
llm,
tts,
transport.output(),
context_aggregator.assistant(),
]
)
task = PipelineTask(pipeline)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
global video_participant_id
video_participant_id = participant["id"]
transport.capture_participant_transcription(participant["id"])
transport.capture_participant_video(video_participant_id, framerate=0)
# Kick off the conversation.
await tts.say("Hi! Ask me about the weather in San Francisco.")
runner = PipelineRunner()
await runner.run(task)
if __name__ == "__main__":
asyncio.run(main())

49
examples/foundational/15-switch-voices.py
View File

@@ -28,6 +28,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -39,7 +40,11 @@ current_voice = "News Lady"
async def switch_voice(function_name, tool_call_id, args, llm, context, result_callback): async def switch_voice(function_name, tool_call_id, args, llm, context, result_callback):
global current_voice global current_voice
current_voice = args["voice"] current_voice = args["voice"]
await result_callback({"voice": f"You are now using your {current_voice} voice. Your responses should now be as if you were a {current_voice}."}) await result_callback(
{
"voice": f"You are now using your {current_voice} voice. Your responses should now be as if you were a {current_voice}."
}
)
async def news_lady_filter(frame) -> bool: async def news_lady_filter(frame) -> bool:
@@ -66,8 +71,8 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
news_lady = CartesiaTTSService( news_lady = CartesiaTTSService(
@@ -85,9 +90,7 @@ async def main():
voice_id="a0e99841-438c-4a64-b679-ae501e7d6091", # Barbershop Man voice_id="a0e99841-438c-4a64-b679-ae501e7d6091", # Barbershop Man
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
llm.register_function("switch_voice", switch_voice) llm.register_function("switch_voice", switch_voice)
tools = [ tools = [
@@ -106,7 +109,9 @@ async def main():
}, },
"required": ["voice"], "required": ["voice"],
}, },
})] },
)
]
messages = [ messages = [
{ {
"role": "system", "role": "system",
@@ -117,18 +122,20 @@ async def main():
context = OpenAILLMContext(messages, tools) context = OpenAILLMContext(messages, tools)
context_aggregator = llm.create_context_aggregator(context) context_aggregator = llm.create_context_aggregator(context)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Transport user input [
context_aggregator.user(), # User responses transport.input(), # Transport user input
llm, # LLM context_aggregator.user(), # User responses
ParallelPipeline( # TTS (one of the following vocies) llm, # LLM
[FunctionFilter(news_lady_filter), news_lady], # News Lady voice ParallelPipeline( # TTS (one of the following vocies)
[FunctionFilter(british_lady_filter), british_lady], # British Lady voice [FunctionFilter(news_lady_filter), news_lady], # News Lady voice
[FunctionFilter(barbershop_man_filter), barbershop_man], # Barbershop Man voice [FunctionFilter(british_lady_filter), british_lady], # British Lady voice
), [FunctionFilter(barbershop_man_filter), barbershop_man], # Barbershop Man voice
transport.output(), # Transport bot output ),
context_aggregator.assistant(), # Assistant spoken responses transport.output(), # Transport bot output
]) context_aggregator.assistant(), # Assistant spoken responses
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
@@ -139,7 +146,9 @@ async def main():
messages.append( messages.append(
{ {
"role": "system", "role": "system",
"content": f"Please introduce yourself to the user and let them know the voices you can do. Your initial responses should be as if you were a {current_voice}."}) "content": f"Please introduce yourself to the user and let them know the voices you can do. Your initial responses should be as if you were a {current_voice}.",
}
)
await task.queue_frames([LLMMessagesFrame(messages)]) await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner() runner = PipelineRunner()

43
examples/foundational/15a-switch-languages.py
View File

@@ -29,6 +29,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -64,8 +65,8 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(), vad_analyzer=SileroVADAnalyzer(),
vad_audio_passthrough=True vad_audio_passthrough=True,
) ),
) )
stt = WhisperSTTService(model=Model.LARGE) stt = WhisperSTTService(model=Model.LARGE)
@@ -80,9 +81,7 @@ async def main():
voice_id="846d6cb0-2301-48b6-9683-48f5618ea2f6", # Spanish-speaking Lady voice_id="846d6cb0-2301-48b6-9683-48f5618ea2f6", # Spanish-speaking Lady
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
llm.register_function("switch_language", switch_language) llm.register_function("switch_language", switch_language)
tools = [ tools = [
@@ -101,7 +100,9 @@ async def main():
}, },
"required": ["language"], "required": ["language"],
}, },
})] },
)
]
messages = [ messages = [
{ {
"role": "system", "role": "system",
@@ -112,18 +113,20 @@ async def main():
context = OpenAILLMContext(messages, tools) context = OpenAILLMContext(messages, tools)
context_aggregator = llm.create_context_aggregator(context) context_aggregator = llm.create_context_aggregator(context)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Transport user input [
stt, # STT transport.input(), # Transport user input
context_aggregator.user(), # User responses stt, # STT
llm, # LLM context_aggregator.user(), # User responses
ParallelPipeline( # TTS (bot will speak the chosen language) llm, # LLM
[FunctionFilter(english_filter), english_tts], # English ParallelPipeline( # TTS (bot will speak the chosen language)
[FunctionFilter(spanish_filter), spanish_tts], # Spanish [FunctionFilter(english_filter), english_tts], # English
), [FunctionFilter(spanish_filter), spanish_tts], # Spanish
transport.output(), # Transport bot output ),
context_aggregator.assistant() # Assistant spoken responses transport.output(), # Transport bot output
]) context_aggregator.assistant(), # Assistant spoken responses
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
@@ -134,7 +137,9 @@ async def main():
messages.append( messages.append(
{ {
"role": "system", "role": "system",
"content": f"Please introduce yourself to the user and let them know the languages you speak. Your initial responses should be in {current_language}."}) "content": f"Please introduce yourself to the user and let them know the languages you speak. Your initial responses should be in {current_language}.",
}
)
await task.queue_frames([LLMMessagesFrame(messages)]) await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner() runner = PipelineRunner()

62
examples/foundational/16-gpu-container-local-bot.py
View File

@@ -5,26 +5,31 @@
# #
import asyncio import asyncio
import aiohttp
import os import os
import sys import sys
import aiohttp
from dotenv import load_dotenv
from loguru import logger
from runner import configure
from pipecat.frames.frames import LLMMessagesFrame from pipecat.frames.frames import LLMMessagesFrame
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import ( from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator, LLMUserResponseAggregator) LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.services.deepgram import DeepgramTTSService from pipecat.services.deepgram import DeepgramTTSService
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport, DailyTransportMessageFrame from pipecat.transports.services.daily import (
DailyParams,
DailyTransport,
DailyTransportMessageFrame,
)
from pipecat.vad.silero import SileroVADAnalyzer from pipecat.vad.silero import SileroVADAnalyzer
from runner import configure
from loguru import logger
from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -43,15 +48,15 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
tts = DeepgramTTSService( tts = DeepgramTTSService(
aiohttp_session=session, aiohttp_session=session,
api_key=os.getenv("DEEPGRAM_API_KEY"), api_key=os.getenv("DEEPGRAM_API_KEY"),
voice="aura-asteria-en", voice="aura-asteria-en",
base_url="http://0.0.0.0:8080/v1/speak" base_url="http://0.0.0.0:8080/v1/speak",
) )
llm = OpenAILLMService( llm = OpenAILLMService(
@@ -60,7 +65,7 @@ async def main():
# model="gpt-4o" # model="gpt-4o"
# Or, to use a local vLLM (or similar) api server # Or, to use a local vLLM (or similar) api server
model="meta-llama/Meta-Llama-3-8B-Instruct", model="meta-llama/Meta-Llama-3-8B-Instruct",
base_url="http://0.0.0.0:8000/v1" base_url="http://0.0.0.0:8000/v1",
) )
messages = [ messages = [
@@ -73,14 +78,16 @@ async def main():
tma_in = LLMUserResponseAggregator(messages) tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Transport user input [
tma_in, # User responses transport.input(), # Transport user input
llm, # LLM tma_in, # User responses
tts, # TTS llm, # LLM
transport.output(), # Transport bot output tts, # TTS
tma_out # Assistant spoken responses transport.output(), # Transport bot output
]) tma_out, # Assistant spoken responses
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True, enable_metrics=True)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True, enable_metrics=True))
@@ -93,8 +100,7 @@ async def main():
# When the first participant joins, the bot should introduce itself. # When the first participant joins, the bot should introduce itself.
@transport.event_handler("on_first_participant_joined") @transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant): async def on_first_participant_joined(transport, participant):
messages.append( messages.append({"role": "system", "content": "Please introduce yourself to the user."})
{"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)]) await task.queue_frames([LLMMessagesFrame(messages)])
# Handle "latency-ping" messages. The client will send app messages that look like # Handle "latency-ping" messages. The client will send app messages that look like
@@ -111,14 +117,18 @@ async def main():
logger.debug(f"Received latency ping app message: {message}") logger.debug(f"Received latency ping app message: {message}")
ts = message["latency-ping"]["ts"] ts = message["latency-ping"]["ts"]
# Send immediately # Send immediately
transport.output().send_message(DailyTransportMessageFrame( transport.output().send_message(
message={"latency-pong-msg-handler": {"ts": ts}}, DailyTransportMessageFrame(
participant_id=sender)) message={"latency-pong-msg-handler": {"ts": ts}}, participant_id=sender
)
)
# And push to the pipeline for the Daily transport.output to send # And push to the pipeline for the Daily transport.output to send
await tma_in.push_frame( await tma_in.push_frame(
DailyTransportMessageFrame( DailyTransportMessageFrame(
message={"latency-pong-pipeline-delivery": {"ts": ts}}, message={"latency-pong-pipeline-delivery": {"ts": ts}},
participant_id=sender)) participant_id=sender,
)
)
except Exception as e: except Exception as e:
logger.debug(f"message handling error: {e} - {message}") logger.debug(f"message handling error: {e} - {message}")

57
examples/foundational/17-detect-user-idle.py
View File

@@ -14,7 +14,9 @@ from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import ( from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator, LLMUserResponseAggregator) LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.processors.user_idle_processor import UserIdleProcessor from pipecat.processors.user_idle_processor import UserIdleProcessor
from pipecat.services.cartesia import CartesiaTTSService from pipecat.services.cartesia import CartesiaTTSService
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
@@ -26,6 +28,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -44,8 +47,8 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
tts = CartesiaTTSService( tts = CartesiaTTSService(
@@ -53,9 +56,7 @@ async def main():
voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
messages = [ messages = [
{ {
@@ -69,33 +70,41 @@ async def main():
async def user_idle_callback(user_idle: UserIdleProcessor): async def user_idle_callback(user_idle: UserIdleProcessor):
messages.append( messages.append(
{"role": "system", "content": "Ask the user if they are still there and try to prompt for some input, but be short."}) {
await user_idle.queue_frame(LLMMessagesFrame(messages)) "role": "system",
"content": "Ask the user if they are still there and try to prompt for some input, but be short.",
}
)
await user_idle.push_frame(LLMMessagesFrame(messages))
user_idle = UserIdleProcessor(callback=user_idle_callback, timeout=5.0) user_idle = UserIdleProcessor(callback=user_idle_callback, timeout=5.0)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Transport user input [
user_idle, # Idle user check-in transport.input(), # Transport user input
tma_in, # User responses user_idle, # Idle user check-in
llm, # LLM tma_in, # User responses
tts, # TTS llm, # LLM
transport.output(), # Transport bot output tts, # TTS
tma_out # Assistant spoken responses transport.output(), # Transport bot output
]) tma_out, # Assistant spoken responses
]
)
task = PipelineTask(pipeline, PipelineParams( task = PipelineTask(
allow_interruptions=True, pipeline,
enable_metrics=True, PipelineParams(
report_only_initial_ttfb=True, allow_interruptions=True,
)) enable_metrics=True,
report_only_initial_ttfb=True,
),
)
@transport.event_handler("on_first_participant_joined") @transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant): async def on_first_participant_joined(transport, participant):
transport.capture_participant_transcription(participant["id"]) transport.capture_participant_transcription(participant["id"])
# Kick off the conversation. # Kick off the conversation.
messages.append( messages.append({"role": "system", "content": "Please introduce yourself to the user."})
{"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)]) await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner() runner = PipelineRunner()

22
examples/foundational/18-gstreamer-filesrc.py
View File

@@ -20,6 +20,7 @@ from runner import configure_with_args
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -29,12 +30,7 @@ logger.add(sys.stderr, level="DEBUG")
async def main(): async def main():
async with aiohttp.ClientSession() as session: async with aiohttp.ClientSession() as session:
parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample") parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample")
parser.add_argument( parser.add_argument("-i", "--input", type=str, required=True, help="Input video file")
"-i",
"--input",
type=str,
required=True,
help="Input video file")
(room_url, _, args) = await configure_with_args(session, parser) (room_url, _, args) = await configure_with_args(session, parser)
@@ -49,7 +45,7 @@ async def main():
camera_out_width=1280, camera_out_width=1280,
camera_out_height=720, camera_out_height=720,
camera_out_is_live=True, camera_out_is_live=True,
) ),
) )
gst = GStreamerPipelineSource( gst = GStreamerPipelineSource(
@@ -59,13 +55,15 @@ async def main():
video_height=720, video_height=720,
audio_sample_rate=16000, audio_sample_rate=16000,
audio_channels=1, audio_channels=1,
) ),
) )
pipeline = Pipeline([ pipeline = Pipeline(
gst, # GStreamer file source [
transport.output(), # Transport bot output gst, # GStreamer file source
]) transport.output(), # Transport bot output
]
)
task = PipelineTask(pipeline) task = PipelineTask(pipeline)

21
examples/foundational/18a-gstreamer-videotestsrc.py
View File

@@ -19,6 +19,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -38,20 +39,22 @@ async def main():
camera_out_width=1280, camera_out_width=1280,
camera_out_height=720, camera_out_height=720,
camera_out_is_live=True, camera_out_is_live=True,
) ),
) )
gst = GStreamerPipelineSource( gst = GStreamerPipelineSource(
pipeline="videotestsrc ! capsfilter caps=\"video/x-raw,width=1280,height=720,framerate=30/1\"", pipeline='videotestsrc ! capsfilter caps="video/x-raw,width=1280,height=720,framerate=30/1"',
out_params=GStreamerPipelineSource.OutputParams( out_params=GStreamerPipelineSource.OutputParams(
video_width=1280, video_width=1280, video_height=720, clock_sync=False
video_height=720, ),
clock_sync=False)) )
pipeline = Pipeline([ pipeline = Pipeline(
gst, # GStreamer file source [
transport.output(), # Transport bot output gst, # GStreamer file source
]) transport.output(), # Transport bot output
]
)
task = PipelineTask(pipeline) task = PipelineTask(pipeline)

138
examples/foundational/19c-tools-togetherai.py
View File

@@ -1,138 +0,0 @@
#
# Copyright (c) 2024, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import asyncio
import aiohttp
import os
import sys
import json
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.services.cartesia import CartesiaTTSService
from pipecat.services.together import TogetherLLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport
from pipecat.vad.silero import SileroVADAnalyzer
from runner import configure
from loguru import logger
from dotenv import load_dotenv
load_dotenv(override=True)
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
async def get_current_weather(
function_name,
tool_call_id,
arguments,
llm,
context,
result_callback):
logger.debug("IN get_current_weather")
location = arguments["location"]
await result_callback(f"The weather in {location} is currently 72 degrees and sunny.")
async def main():
async with aiohttp.ClientSession() as session:
(room_url, token) = await configure(session)
transport = DailyTransport(
room_url,
token,
"Respond bot",
DailyParams(
audio_out_enabled=True,
transcription_enabled=True,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer()
)
)
tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY"),
voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
)
llm = TogetherLLMService(
api_key=os.getenv("TOGETHER_API_KEY"),
model=os.getenv("TOGETHER_MODEL"),
)
llm.register_function("get_current_weather", get_current_weather)
weatherTool = {
"name": "get_current_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA",
},
},
"required": ["location"],
},
}
system_prompt = f"""\
You have access to the following functions:
Use the function '{weatherTool["name"]}' to '{weatherTool["description"]}':
{json.dumps(weatherTool)}
If you choose to call a function ONLY reply in the following format with no prefix or suffix:
<function=example_function_name>{{\"example_name\": \"example_value\"}}</function>
Reminder:
- Function calls MUST follow the specified format, start with <function= and end with </function>
- Required parameters MUST be specified
- Only call one function at a time
- Put the entire function call reply on one line
- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls
"""
messages = [{"role": "system",
"content": system_prompt},
{"role": "user",
"content": "Wait for the user to say something."}]
context = OpenAILLMContext(messages)
context_aggregator = llm.create_context_aggregator(context)
pipeline = Pipeline([
transport.input(), # Transport user input
context_aggregator.user(), # User speech to text
llm, # LLM
tts, # TTS
transport.output(), # Transport bot output
context_aggregator.assistant(), # Assistant spoken responses and tool context
])
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True, enable_metrics=True))
@ transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner()
await runner.run(task)
if __name__ == "__main__":
asyncio.run(main())

21
examples/foundational/runner.py
View File

@@ -17,16 +17,13 @@ async def configure(aiohttp_session: aiohttp.ClientSession):
async def configure_with_args( async def configure_with_args(
aiohttp_session: aiohttp.ClientSession, aiohttp_session: aiohttp.ClientSession, parser: argparse.ArgumentParser | None = None
parser: argparse.ArgumentParser | None = None): ):
if not parser: if not parser:
parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample") parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample")
parser.add_argument( parser.add_argument(
"-u", "-u", "--url", type=str, required=False, help="URL of the Daily room to join"
"--url", )
type=str,
required=False,
help="URL of the Daily room to join")
parser.add_argument( parser.add_argument(
"-k", "-k",
"--apikey", "--apikey",
@@ -42,15 +39,19 @@ async def configure_with_args(
if not url: if not url:
raise Exception( raise Exception(
"No Daily room specified. use the -u/--url option from the command line, or set DAILY_SAMPLE_ROOM_URL in your environment to specify a Daily room URL.") "No Daily room specified. use the -u/--url option from the command line, or set DAILY_SAMPLE_ROOM_URL in your environment to specify a Daily room URL."
)
if not key: if not key:
raise Exception("No Daily API key specified. use the -k/--apikey option from the command line, or set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.") raise Exception(
"No Daily API key specified. use the -k/--apikey option from the command line, or set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers."
)
daily_rest_helper = DailyRESTHelper( daily_rest_helper = DailyRESTHelper(
daily_api_key=key, daily_api_key=key,
daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"), daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
aiohttp_session=aiohttp_session) aiohttp_session=aiohttp_session,
)
# Create a meeting token for the given room with an expiration 1 hour in # Create a meeting token for the given room with an expiration 1 hour in
# the future. # the future.

42
examples/moondream-chatbot/bot.py
View File

@@ -13,10 +13,11 @@ from PIL import Image
from pipecat.frames.frames import ( from pipecat.frames.frames import (
ImageRawFrame, ImageRawFrame,
OutputImageRawFrame,
SpriteFrame, SpriteFrame,
Frame, Frame,
LLMMessagesFrame, LLMMessagesFrame,
AudioRawFrame, TTSAudioRawFrame,
TTSStoppedFrame, TTSStoppedFrame,
TextFrame, TextFrame,
UserImageRawFrame, UserImageRawFrame,
@@ -42,6 +43,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -59,7 +61,7 @@ for i in range(1, 26):
# Get the filename without the extension to use as the dictionary key # Get the filename without the extension to use as the dictionary key
# Open the image and convert it to bytes # Open the image and convert it to bytes
with Image.open(full_path) as img: with Image.open(full_path) as img:
sprites.append(ImageRawFrame(image=img.tobytes(), size=img.size, format=img.format)) sprites.append(OutputImageRawFrame(image=img.tobytes(), size=img.size, format=img.format))
flipped = sprites[::-1] flipped = sprites[::-1]
sprites.extend(flipped) sprites.extend(flipped)
@@ -82,7 +84,7 @@ class TalkingAnimation(FrameProcessor):
async def process_frame(self, frame: Frame, direction: FrameDirection): async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction) await super().process_frame(frame, direction)
if isinstance(frame, AudioRawFrame): if isinstance(frame, TTSAudioRawFrame):
if not self._is_talking: if not self._is_talking:
await self.push_frame(talking_frame) await self.push_frame(talking_frame)
self._is_talking = True self._is_talking = True
@@ -105,7 +107,9 @@ class UserImageRequester(FrameProcessor):
if self.participant_id and isinstance(frame, TextFrame): if self.participant_id and isinstance(frame, TextFrame):
if frame.text == user_request_answer: if frame.text == user_request_answer:
await self.push_frame(UserImageRequestFrame(self.participant_id), FrameDirection.UPSTREAM) await self.push_frame(
UserImageRequestFrame(self.participant_id), FrameDirection.UPSTREAM
)
await self.push_frame(TextFrame("Describe the image in a short sentence.")) await self.push_frame(TextFrame("Describe the image in a short sentence."))
elif isinstance(frame, UserImageRawFrame): elif isinstance(frame, UserImageRawFrame):
await self.push_frame(frame) await self.push_frame(frame)
@@ -149,8 +153,8 @@ async def main():
camera_out_height=576, camera_out_height=576,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
tts = CartesiaTTSService( tts = CartesiaTTSService(
@@ -158,9 +162,7 @@ async def main():
voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
ta = TalkingAnimation() ta = TalkingAnimation()
@@ -183,17 +185,17 @@ async def main():
ura = LLMUserResponseAggregator(messages) ura = LLMUserResponseAggregator(messages)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), [
ura, transport.input(),
llm, ura,
ParallelPipeline( llm,
[sa, ir, va, moondream], ParallelPipeline([sa, ir, va, moondream], [tf, imgf]),
[tf, imgf]), tts,
tts, ta,
ta, transport.output(),
transport.output() ]
]) )
task = PipelineTask(pipeline) task = PipelineTask(pipeline)
await task.queue_frame(quiet_frame) await task.queue_frame(quiet_frame)

2
examples/moondream-chatbot/requirements.txt
View File

@@ -1,4 +1,4 @@
python-dotenv python-dotenv
fastapi[all] fastapi[all]
uvicorn uvicorn
pipecat-ai[daily,moondream,openai,silero] pipecat-ai[daily,cartesia,moondream,openai,silero]

16
examples/moondream-chatbot/runner.py
View File

@@ -14,11 +14,8 @@ from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper
async def configure(aiohttp_session: aiohttp.ClientSession): async def configure(aiohttp_session: aiohttp.ClientSession):
parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample") parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample")
parser.add_argument( parser.add_argument(
"-u", "-u", "--url", type=str, required=False, help="URL of the Daily room to join"
"--url", )
type=str,
required=False,
help="URL of the Daily room to join")
parser.add_argument( parser.add_argument(
"-k", "-k",
"--apikey", "--apikey",
@@ -34,15 +31,18 @@ async def configure(aiohttp_session: aiohttp.ClientSession):
if not url: if not url:
raise Exception( raise Exception(
"No Daily room specified. use the -u/--url option from the command line, or set DAILY_SAMPLE_ROOM_URL in your environment to specify a Daily room URL.") "No Daily room specified. use the -u/--url option from the command line, or set DAILY_SAMPLE_ROOM_URL in your environment to specify a Daily room URL."
)
if not key: if not key:
raise Exception("No Daily API key specified. use the -k/--apikey option from the command line, or set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.") raise Exception(
"No Daily API key specified. use the -k/--apikey option from the command line, or set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers."
)
daily_rest_helper = DailyRESTHelper( daily_rest_helper = DailyRESTHelper(
daily_api_key=key, daily_api_key=key,
daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"), daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
aiohttp_session=aiohttp_session aiohttp_session=aiohttp_session,
) )
# Create a meeting token for the given room with an expiration 1 hour in # Create a meeting token for the given room with an expiration 1 hour in

41
examples/moondream-chatbot/server.py
View File

@@ -38,13 +38,14 @@ async def lifespan(app: FastAPI):
aiohttp_session = aiohttp.ClientSession() aiohttp_session = aiohttp.ClientSession()
daily_helpers["rest"] = DailyRESTHelper( daily_helpers["rest"] = DailyRESTHelper(
daily_api_key=os.getenv("DAILY_API_KEY", ""), daily_api_key=os.getenv("DAILY_API_KEY", ""),
daily_api_url=os.getenv("DAILY_API_URL", 'https://api.daily.co/v1'), daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
aiohttp_session=aiohttp_session aiohttp_session=aiohttp_session,
) )
yield yield
await aiohttp_session.close() await aiohttp_session.close()
cleanup() cleanup()
app = FastAPI(lifespan=lifespan) app = FastAPI(lifespan=lifespan)
app.add_middleware( app.add_middleware(
@@ -65,37 +66,34 @@ async def start_agent(request: Request):
if not room.url: if not room.url:
raise HTTPException( raise HTTPException(
status_code=500, status_code=500,
detail="Missing 'room' property in request data. Cannot start agent without a target room!") detail="Missing 'room' property in request data. Cannot start agent without a target room!",
)
# Check if there is already an existing process running in this room # Check if there is already an existing process running in this room
num_bots_in_room = sum( num_bots_in_room = sum(
1 for proc in bot_procs.values() if proc[1] == room.url and proc[0].poll() is None) 1 for proc in bot_procs.values() if proc[1] == room.url and proc[0].poll() is None
)
if num_bots_in_room >= MAX_BOTS_PER_ROOM: if num_bots_in_room >= MAX_BOTS_PER_ROOM:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Max bot limited reach for room: {room.url}")
status_code=500, detail=f"Max bot limited reach for room: {room.url}")
# Get the token for the room # Get the token for the room
token = await daily_helpers["rest"].get_token(room.url) token = await daily_helpers["rest"].get_token(room.url)
if not token: if not token:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Failed to get token for room: {room.url}")
status_code=500, detail=f"Failed to get token for room: {room.url}")
# Spawn a new agent, and join the user session # Spawn a new agent, and join the user session
# Note: this is mostly for demonstration purposes (refer to 'deployment' in README) # Note: this is mostly for demonstration purposes (refer to 'deployment' in README)
try: try:
proc = subprocess.Popen( proc = subprocess.Popen(
[ [f"python3 -m bot -u {room.url} -t {token}"],
f"python3 -m bot -u {room.url} -t {token}"
],
shell=True, shell=True,
bufsize=1, bufsize=1,
cwd=os.path.dirname(os.path.abspath(__file__)) cwd=os.path.dirname(os.path.abspath(__file__)),
) )
bot_procs[proc.pid] = (proc, room.url) bot_procs[proc.pid] = (proc, room.url)
except Exception as e: except Exception as e:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Failed to start subprocess: {e}")
status_code=500, detail=f"Failed to start subprocess: {e}")
return RedirectResponse(room.url) return RedirectResponse(room.url)
@@ -107,8 +105,7 @@ def get_status(pid: int):
# If the subprocess doesn't exist, return an error # If the subprocess doesn't exist, return an error
if not proc: if not proc:
raise HTTPException( raise HTTPException(status_code=404, detail=f"Bot with process id: {pid} not found")
status_code=404, detail=f"Bot with process id: {pid} not found")
# Check the status of the subprocess # Check the status of the subprocess
if proc[0].poll() is None: if proc[0].poll() is None:
@@ -125,14 +122,10 @@ if __name__ == "__main__":
default_host = os.getenv("HOST", "0.0.0.0") default_host = os.getenv("HOST", "0.0.0.0")
default_port = int(os.getenv("FAST_API_PORT", "7860")) default_port = int(os.getenv("FAST_API_PORT", "7860"))
parser = argparse.ArgumentParser( parser = argparse.ArgumentParser(description="Daily Moondream FastAPI server")
description="Daily Moondream FastAPI server") parser.add_argument("--host", type=str, default=default_host, help="Host address")
parser.add_argument("--host", type=str, parser.add_argument("--port", type=int, default=default_port, help="Port number")
default=default_host, help="Host address") parser.add_argument("--reload", action="store_true", help="Reload code on change")
parser.add_argument("--port", type=int,
default=default_port, help="Port number")
parser.add_argument("--reload", action="store_true",
help="Reload code on change")
config = parser.parse_args() config = parser.parse_args()

174
examples/patient-intake/bot.py
View File

@@ -10,7 +10,7 @@ import os
import sys import sys
import wave import wave
from pipecat.frames.frames import AudioRawFrame from pipecat.frames.frames import OutputAudioRawFrame
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -26,6 +26,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -49,40 +50,44 @@ for file in sound_files:
filename = os.path.splitext(os.path.basename(full_path))[0] filename = os.path.splitext(os.path.basename(full_path))[0]
# Open the sound and convert it to bytes # Open the sound and convert it to bytes
with wave.open(full_path) as audio_file: with wave.open(full_path) as audio_file:
sounds[file] = AudioRawFrame(audio_file.readframes(-1), sounds[file] = OutputAudioRawFrame(
audio_file.getframerate(), audio_file.getnchannels()) audio_file.readframes(-1), audio_file.getframerate(), audio_file.getnchannels()
)
class IntakeProcessor: class IntakeProcessor:
def __init__(self, context: OpenAILLMContext): def __init__(self, context: OpenAILLMContext):
print(f"Initializing context from IntakeProcessor") print(f"Initializing context from IntakeProcessor")
context.add_message({"role": "system", "content": "You are Jessica, an agent for a company called Tri-County Health Services. Your job is to collect important information from the user before their doctor visit. You're talking to Chad Bailey. You should address the user by their first name and be polite and professional. You're not a medical professional, so you shouldn't provide any advice. Keep your responses short. Your job is to collect information to give to a doctor. Don't make assumptions about what values to plug into functions. Ask for clarification if a user response is ambiguous. Start by introducing yourself. Then, ask the user to confirm their identity by telling you their birthday, including the year. When they answer with their birthday, call the verify_birthday function."}) context.add_message(
context.set_tools([
{ {
"type": "function", "role": "system",
"function": { "content": "You are Jessica, an agent for a company called Tri-County Health Services. Your job is to collect important information from the user before their doctor visit. You're talking to Chad Bailey. You should address the user by their first name and be polite and professional. You're not a medical professional, so you shouldn't provide any advice. Keep your responses short. Your job is to collect information to give to a doctor. Don't make assumptions about what values to plug into functions. Ask for clarification if a user response is ambiguous. Start by introducing yourself. Then, ask the user to confirm their identity by telling you their birthday, including the year. When they answer with their birthday, call the verify_birthday function.",
"name": "verify_birthday", }
"description": "Use this function to verify the user has provided their correct birthday.", )
"parameters": { context.set_tools(
"type": "object", [
"properties": { {
"birthday": { "type": "function",
"type": "string", "function": {
"description": "The user's birthdate, including the year. The user can provide it in any format, but convert it to YYYY-MM-DD format to call this function.", "name": "verify_birthday",
}}, "description": "Use this function to verify the user has provided their correct birthday.",
"parameters": {
"type": "object",
"properties": {
"birthday": {
"type": "string",
"description": "The user's birthdate, including the year. The user can provide it in any format, but convert it to YYYY-MM-DD format to call this function.",
}
},
},
}, },
}, }
}]) ]
)
async def verify_birthday( async def verify_birthday(
self, self, function_name, tool_call_id, args, llm, context, result_callback
function_name, ):
tool_call_id,
args,
llm,
context,
result_callback):
if args["birthday"] == "1983-01-01": if args["birthday"] == "1983-01-01":
context.set_tools( context.set_tools(
[ [
@@ -109,18 +114,35 @@ class IntakeProcessor:
}, },
}, },
}, },
}}, }
},
}, },
}, },
}]) }
]
)
# It's a bit weird to push this to the LLM, but it gets it into the pipeline # It's a bit weird to push this to the LLM, but it gets it into the pipeline
# await llm.push_frame(sounds["ding2.wav"], FrameDirection.DOWNSTREAM) # await llm.push_frame(sounds["ding2.wav"], FrameDirection.DOWNSTREAM)
# We don't need the function call in the context, so just return a new # We don't need the function call in the context, so just return a new
# system message and let the framework re-prompt # system message and let the framework re-prompt
await result_callback([{"role": "system", "content": "Next, thank the user for confirming their identity, then ask the user to list their current prescriptions. Each prescription needs to have a medication name and a dosage. Do not call the list_prescriptions function with any unknown dosages."}]) await result_callback(
[
{
"role": "system",
"content": "Next, thank the user for confirming their identity, then ask the user to list their current prescriptions. Each prescription needs to have a medication name and a dosage. Do not call the list_prescriptions function with any unknown dosages.",
}
]
)
else: else:
# The user provided an incorrect birthday; ask them to try again # The user provided an incorrect birthday; ask them to try again
await result_callback([{"role": "system", "content": "The user provided an incorrect birthday. Ask them for their birthday again. When they answer, call the verify_birthday function."}]) await result_callback(
[
{
"role": "system",
"content": "The user provided an incorrect birthday. Ask them for their birthday again. When they answer, call the verify_birthday function.",
}
]
)
async def start_prescriptions(self, function_name, llm, context): async def start_prescriptions(self, function_name, llm, context):
print(f"!!! doing start prescriptions") print(f"!!! doing start prescriptions")
@@ -143,16 +165,22 @@ class IntakeProcessor:
"name": { "name": {
"type": "string", "type": "string",
"description": "What the user is allergic to", "description": "What the user is allergic to",
}}, }
},
}, },
}}, }
},
}, },
}, },
}]) }
]
)
context.add_message( context.add_message(
{ {
"role": "system", "role": "system",
"content": "Next, ask the user if they have any allergies. Once they have listed their allergies or confirmed they don't have any, call the list_allergies function."}) "content": "Next, ask the user if they have any allergies. Once they have listed their allergies or confirmed they don't have any, call the list_allergies function.",
}
)
print(f"!!! about to await llm process frame in start prescrpitions") print(f"!!! about to await llm process frame in start prescrpitions")
await llm.process_frame(OpenAILLMContextFrame(context), FrameDirection.DOWNSTREAM) await llm.process_frame(OpenAILLMContextFrame(context), FrameDirection.DOWNSTREAM)
print(f"!!! past await process frame in start prescriptions") print(f"!!! past await process frame in start prescriptions")
@@ -178,17 +206,22 @@ class IntakeProcessor:
"name": { "name": {
"type": "string", "type": "string",
"description": "The user's medical condition", "description": "The user's medical condition",
}}, }
},
}, },
}}, }
},
}, },
}, },
}, },
]) ]
)
context.add_message( context.add_message(
{ {
"role": "system", "role": "system",
"content": "Now ask the user if they have any medical conditions the doctor should know about. Once they've answered the question, call the list_conditions function."}) "content": "Now ask the user if they have any medical conditions the doctor should know about. Once they've answered the question, call the list_conditions function.",
}
)
await llm.process_frame(OpenAILLMContextFrame(context), FrameDirection.DOWNSTREAM) await llm.process_frame(OpenAILLMContextFrame(context), FrameDirection.DOWNSTREAM)
async def start_conditions(self, function_name, llm, context): async def start_conditions(self, function_name, llm, context):
@@ -212,24 +245,31 @@ class IntakeProcessor:
"name": { "name": {
"type": "string", "type": "string",
"description": "The user's reason for visiting the doctor", "description": "The user's reason for visiting the doctor",
}}, }
},
}, },
}}, }
},
}, },
}, },
}]) }
]
)
context.add_message( context.add_message(
{ {
"role": "system", "role": "system",
"content": "Finally, ask the user the reason for their doctor visit today. Once they answer, call the list_visit_reasons function."}) "content": "Finally, ask the user the reason for their doctor visit today. Once they answer, call the list_visit_reasons function.",
}
)
await llm.process_frame(OpenAILLMContextFrame(context), FrameDirection.DOWNSTREAM) await llm.process_frame(OpenAILLMContextFrame(context), FrameDirection.DOWNSTREAM)
async def start_visit_reasons(self, function_name, llm, context): async def start_visit_reasons(self, function_name, llm, context):
print("!!! doing start visit reasons") print("!!! doing start visit reasons")
# move to finish call # move to finish call
context.set_tools([]) context.set_tools([])
context.add_message({"role": "system", context.add_message(
"content": "Now, thank the user and end the conversation."}) {"role": "system", "content": "Now, thank the user and end the conversation."}
)
await llm.process_frame(OpenAILLMContextFrame(context), FrameDirection.DOWNSTREAM) await llm.process_frame(OpenAILLMContextFrame(context), FrameDirection.DOWNSTREAM)
async def save_data(self, function_name, tool_call_id, args, llm, context, result_callback): async def save_data(self, function_name, tool_call_id, args, llm, context, result_callback):
@@ -260,7 +300,7 @@ async def main():
# tier="nova", # tier="nova",
# model="2-general" # model="2-general"
# ) # )
) ),
) )
tts = CartesiaTTSService( tts = CartesiaTTSService(
@@ -273,9 +313,7 @@ async def main():
# voice_id="846d6cb0-2301-48b6-9683-48f5618ea2f6", # Spanish-speaking Lady # voice_id="846d6cb0-2301-48b6-9683-48f5618ea2f6", # Spanish-speaking Lady
# ) # )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
messages = [] messages = []
context = OpenAILLMContext(messages=messages) context = OpenAILLMContext(messages=messages)
@@ -284,33 +322,31 @@ async def main():
intake = IntakeProcessor(context) intake = IntakeProcessor(context)
llm.register_function("verify_birthday", intake.verify_birthday) llm.register_function("verify_birthday", intake.verify_birthday)
llm.register_function( llm.register_function(
"list_prescriptions", "list_prescriptions", intake.save_data, start_callback=intake.start_prescriptions
intake.save_data, )
start_callback=intake.start_prescriptions)
llm.register_function( llm.register_function(
"list_allergies", "list_allergies", intake.save_data, start_callback=intake.start_allergies
intake.save_data, )
start_callback=intake.start_allergies)
llm.register_function( llm.register_function(
"list_conditions", "list_conditions", intake.save_data, start_callback=intake.start_conditions
intake.save_data, )
start_callback=intake.start_conditions)
llm.register_function( llm.register_function(
"list_visit_reasons", "list_visit_reasons", intake.save_data, start_callback=intake.start_visit_reasons
intake.save_data, )
start_callback=intake.start_visit_reasons)
fl = FrameLogger("LLM Output") fl = FrameLogger("LLM Output")
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Transport input [
context_aggregator.user(), # User responses transport.input(), # Transport input
llm, # LLM context_aggregator.user(), # User responses
fl, # Frame logger llm, # LLM
tts, # TTS fl, # Frame logger
transport.output(), # Transport output tts, # TTS
context_aggregator.assistant(), # Assistant responses transport.output(), # Transport output
]) context_aggregator.assistant(), # Assistant responses
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=False)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=False))

2
examples/patient-intake/env.example
View File

@@ -1,4 +1,4 @@
DAILY_SAMPLE_ROOM_URL=https://yourdomain.daily.co/yourroom # (for joining the bot to the same room repeatedly for local dev) DAILY_SAMPLE_ROOM_URL=https://yourdomain.daily.co/yourroom # (for joining the bot to the same room repeatedly for local dev)
DAILY_API_KEY=7df... DAILY_API_KEY=7df...
OPENAI_API_KEY=sk-PL... OPENAI_API_KEY=sk-PL...
ELEVENLABS_API_KEY=aeb... CARTESIA_API_KEY=your_cartesia_api_key_here

2
examples/patient-intake/requirements.txt
View File

@@ -1,4 +1,4 @@
python-dotenv python-dotenv
fastapi[all] fastapi[all]
uvicorn uvicorn
pipecat-ai[daily,openai,silero] pipecat-ai[daily,cartesia,openai,silero]

17
examples/patient-intake/runner.py
View File

@@ -14,11 +14,8 @@ from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper
async def configure(aiohttp_session: aiohttp.ClientSession): async def configure(aiohttp_session: aiohttp.ClientSession):
parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample") parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample")
parser.add_argument( parser.add_argument(
"-u", "-u", "--url", type=str, required=False, help="URL of the Daily room to join"
"--url", )
type=str,
required=False,
help="URL of the Daily room to join")
parser.add_argument( parser.add_argument(
"-k", "-k",
"--apikey", "--apikey",
@@ -34,15 +31,19 @@ async def configure(aiohttp_session: aiohttp.ClientSession):
if not url: if not url:
raise Exception( raise Exception(
"No Daily room specified. use the -u/--url option from the command line, or set DAILY_SAMPLE_ROOM_URL in your environment to specify a Daily room URL.") "No Daily room specified. use the -u/--url option from the command line, or set DAILY_SAMPLE_ROOM_URL in your environment to specify a Daily room URL."
)
if not key: if not key:
raise Exception("No Daily API key specified. use the -k/--apikey option from the command line, or set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.") raise Exception(
"No Daily API key specified. use the -k/--apikey option from the command line, or set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers."
)
daily_rest_helper = DailyRESTHelper( daily_rest_helper = DailyRESTHelper(
daily_api_key=key, daily_api_key=key,
daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"), daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
aiohttp_session=aiohttp_session) aiohttp_session=aiohttp_session,
)
# Create a meeting token for the given room with an expiration 1 hour in # Create a meeting token for the given room with an expiration 1 hour in
# the future. # the future.

41
examples/patient-intake/server.py
View File

@@ -38,13 +38,14 @@ async def lifespan(app: FastAPI):
aiohttp_session = aiohttp.ClientSession() aiohttp_session = aiohttp.ClientSession()
daily_helpers["rest"] = DailyRESTHelper( daily_helpers["rest"] = DailyRESTHelper(
daily_api_key=os.getenv("DAILY_API_KEY", ""), daily_api_key=os.getenv("DAILY_API_KEY", ""),
daily_api_url=os.getenv("DAILY_API_URL", 'https://api.daily.co/v1'), daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
aiohttp_session=aiohttp_session aiohttp_session=aiohttp_session,
) )
yield yield
await aiohttp_session.close() await aiohttp_session.close()
cleanup() cleanup()
app = FastAPI(lifespan=lifespan) app = FastAPI(lifespan=lifespan)
app.add_middleware( app.add_middleware(
@@ -65,37 +66,34 @@ async def start_agent(request: Request):
if not room.url: if not room.url:
raise HTTPException( raise HTTPException(
status_code=500, status_code=500,
detail="Missing 'room' property in request data. Cannot start agent without a target room!") detail="Missing 'room' property in request data. Cannot start agent without a target room!",
)
# Check if there is already an existing process running in this room # Check if there is already an existing process running in this room
num_bots_in_room = sum( num_bots_in_room = sum(
1 for proc in bot_procs.values() if proc[1] == room.url and proc[0].poll() is None) 1 for proc in bot_procs.values() if proc[1] == room.url and proc[0].poll() is None
)
if num_bots_in_room >= MAX_BOTS_PER_ROOM: if num_bots_in_room >= MAX_BOTS_PER_ROOM:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Max bot limited reach for room: {room.url}")
status_code=500, detail=f"Max bot limited reach for room: {room.url}")
# Get the token for the room # Get the token for the room
token = await daily_helpers["rest"].get_token(room.url) token = await daily_helpers["rest"].get_token(room.url)
if not token: if not token:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Failed to get token for room: {room.url}")
status_code=500, detail=f"Failed to get token for room: {room.url}")
# Spawn a new agent, and join the user session # Spawn a new agent, and join the user session
# Note: this is mostly for demonstration purposes (refer to 'deployment' in README) # Note: this is mostly for demonstration purposes (refer to 'deployment' in README)
try: try:
proc = subprocess.Popen( proc = subprocess.Popen(
[ [f"python3 -m bot -u {room.url} -t {token}"],
f"python3 -m bot -u {room.url} -t {token}"
],
shell=True, shell=True,
bufsize=1, bufsize=1,
cwd=os.path.dirname(os.path.abspath(__file__)) cwd=os.path.dirname(os.path.abspath(__file__)),
) )
bot_procs[proc.pid] = (proc, room.url) bot_procs[proc.pid] = (proc, room.url)
except Exception as e: except Exception as e:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Failed to start subprocess: {e}")
status_code=500, detail=f"Failed to start subprocess: {e}")
return RedirectResponse(room.url) return RedirectResponse(room.url)
@@ -107,8 +105,7 @@ def get_status(pid: int):
# If the subprocess doesn't exist, return an error # If the subprocess doesn't exist, return an error
if not proc: if not proc:
raise HTTPException( raise HTTPException(status_code=404, detail=f"Bot with process id: {pid} not found")
status_code=404, detail=f"Bot with process id: {pid} not found")
# Check the status of the subprocess # Check the status of the subprocess
if proc[0].poll() is None: if proc[0].poll() is None:
@@ -125,14 +122,10 @@ if __name__ == "__main__":
default_host = os.getenv("HOST", "0.0.0.0") default_host = os.getenv("HOST", "0.0.0.0")
default_port = int(os.getenv("FAST_API_PORT", "7860")) default_port = int(os.getenv("FAST_API_PORT", "7860"))
parser = argparse.ArgumentParser( parser = argparse.ArgumentParser(description="Daily Storyteller FastAPI server")
description="Daily Storyteller FastAPI server") parser.add_argument("--host", type=str, default=default_host, help="Host address")
parser.add_argument("--host", type=str, parser.add_argument("--port", type=int, default=default_port, help="Port number")
default=default_host, help="Host address") parser.add_argument("--reload", action="store_true", help="Reload code on change")
parser.add_argument("--port", type=int,
default=default_port, help="Port number")
parser.add_argument("--reload", action="store_true",
help="Reload code on change")
config = parser.parse_args() config = parser.parse_args()
print(f"to join a test room, visit http://localhost:{config.port}/start") print(f"to join a test room, visit http://localhost:{config.port}/start")

44
examples/simple-chatbot/bot.py
View File

@@ -14,14 +14,17 @@ from PIL import Image
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import LLMAssistantResponseAggregator, LLMUserResponseAggregator from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.frames.frames import ( from pipecat.frames.frames import (
AudioRawFrame, OutputImageRawFrame,
ImageRawFrame,
SpriteFrame, SpriteFrame,
Frame, Frame,
LLMMessagesFrame, LLMMessagesFrame,
TTSStoppedFrame TTSAudioRawFrame,
TTSStoppedFrame,
) )
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.services.elevenlabs import ElevenLabsTTSService from pipecat.services.elevenlabs import ElevenLabsTTSService
@@ -34,6 +37,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -49,7 +53,7 @@ for i in range(1, 26):
# Get the filename without the extension to use as the dictionary key # Get the filename without the extension to use as the dictionary key
# Open the image and convert it to bytes # Open the image and convert it to bytes
with Image.open(full_path) as img: with Image.open(full_path) as img:
sprites.append(ImageRawFrame(image=img.tobytes(), size=img.size, format=img.format)) sprites.append(OutputImageRawFrame(image=img.tobytes(), size=img.size, format=img.format))
flipped = sprites[::-1] flipped = sprites[::-1]
sprites.extend(flipped) sprites.extend(flipped)
@@ -72,7 +76,7 @@ class TalkingAnimation(FrameProcessor):
async def process_frame(self, frame: Frame, direction: FrameDirection): async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction) await super().process_frame(frame, direction)
if isinstance(frame, AudioRawFrame): if isinstance(frame, TTSAudioRawFrame):
if not self._is_talking: if not self._is_talking:
await self.push_frame(talking_frame) await self.push_frame(talking_frame)
self._is_talking = True self._is_talking = True
@@ -107,7 +111,7 @@ async def main():
# tier="nova", # tier="nova",
# model="2-general" # model="2-general"
# ) # )
) ),
) )
tts = ElevenLabsTTSService( tts = ElevenLabsTTSService(
@@ -116,7 +120,6 @@ async def main():
# English # English
# #
voice_id="pNInz6obpgDQGcFmaJgB", voice_id="pNInz6obpgDQGcFmaJgB",
# #
# Spanish # Spanish
# #
@@ -124,9 +127,7 @@ async def main():
# voice_id="gD1IexrzCvsXPHUuT0s3", # voice_id="gD1IexrzCvsXPHUuT0s3",
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
messages = [ messages = [
{ {
@@ -135,7 +136,6 @@ async def main():
# English # English
# #
"content": "You are Chatbot, a friendly, helpful robot. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by introducing yourself.", "content": "You are Chatbot, a friendly, helpful robot. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by introducing yourself.",
# #
# Spanish # Spanish
# #
@@ -148,15 +148,17 @@ async def main():
ta = TalkingAnimation() ta = TalkingAnimation()
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), [
user_response, transport.input(),
llm, user_response,
tts, llm,
ta, tts,
transport.output(), ta,
assistant_response, transport.output(),
]) assistant_response,
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
await task.queue_frame(quiet_frame) await task.queue_frame(quiet_frame)

2
examples/simple-chatbot/requirements.txt
View File

@@ -1,4 +1,4 @@
python-dotenv python-dotenv
fastapi[all] fastapi[all]
uvicorn uvicorn
pipecat-ai[daily,openai,silero,elevenlabs] pipecat-ai[daily,elevenlabs,openai,silero]

16
examples/simple-chatbot/runner.py
View File

@@ -14,11 +14,8 @@ from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper
async def configure(aiohttp_session: aiohttp.ClientSession): async def configure(aiohttp_session: aiohttp.ClientSession):
parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample") parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample")
parser.add_argument( parser.add_argument(
"-u", "-u", "--url", type=str, required=False, help="URL of the Daily room to join"
"--url", )
type=str,
required=False,
help="URL of the Daily room to join")
parser.add_argument( parser.add_argument(
"-k", "-k",
"--apikey", "--apikey",
@@ -34,15 +31,18 @@ async def configure(aiohttp_session: aiohttp.ClientSession):
if not url: if not url:
raise Exception( raise Exception(
"No Daily room specified. use the -u/--url option from the command line, or set DAILY_SAMPLE_ROOM_URL in your environment to specify a Daily room URL.") "No Daily room specified. use the -u/--url option from the command line, or set DAILY_SAMPLE_ROOM_URL in your environment to specify a Daily room URL."
)
if not key: if not key:
raise Exception("No Daily API key specified. use the -k/--apikey option from the command line, or set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.") raise Exception(
"No Daily API key specified. use the -k/--apikey option from the command line, or set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers."
)
daily_rest_helper = DailyRESTHelper( daily_rest_helper = DailyRESTHelper(
daily_api_key=key, daily_api_key=key,
daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"), daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
aiohttp_session=aiohttp_session aiohttp_session=aiohttp_session,
) )
# Create a meeting token for the given room with an expiration 1 hour in # Create a meeting token for the given room with an expiration 1 hour in

41
examples/simple-chatbot/server.py
View File

@@ -38,13 +38,14 @@ async def lifespan(app: FastAPI):
aiohttp_session = aiohttp.ClientSession() aiohttp_session = aiohttp.ClientSession()
daily_helpers["rest"] = DailyRESTHelper( daily_helpers["rest"] = DailyRESTHelper(
daily_api_key=os.getenv("DAILY_API_KEY", ""), daily_api_key=os.getenv("DAILY_API_KEY", ""),
daily_api_url=os.getenv("DAILY_API_URL", 'https://api.daily.co/v1'), daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
aiohttp_session=aiohttp_session aiohttp_session=aiohttp_session,
) )
yield yield
await aiohttp_session.close() await aiohttp_session.close()
cleanup() cleanup()
app = FastAPI(lifespan=lifespan) app = FastAPI(lifespan=lifespan)
app.add_middleware( app.add_middleware(
@@ -65,37 +66,34 @@ async def start_agent(request: Request):
if not room.url: if not room.url:
raise HTTPException( raise HTTPException(
status_code=500, status_code=500,
detail="Missing 'room' property in request data. Cannot start agent without a target room!") detail="Missing 'room' property in request data. Cannot start agent without a target room!",
)
# Check if there is already an existing process running in this room # Check if there is already an existing process running in this room
num_bots_in_room = sum( num_bots_in_room = sum(
1 for proc in bot_procs.values() if proc[1] == room.url and proc[0].poll() is None) 1 for proc in bot_procs.values() if proc[1] == room.url and proc[0].poll() is None
)
if num_bots_in_room >= MAX_BOTS_PER_ROOM: if num_bots_in_room >= MAX_BOTS_PER_ROOM:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Max bot limited reach for room: {room.url}")
status_code=500, detail=f"Max bot limited reach for room: {room.url}")
# Get the token for the room # Get the token for the room
token = await daily_helpers["rest"].get_token(room.url) token = await daily_helpers["rest"].get_token(room.url)
if not token: if not token:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Failed to get token for room: {room.url}")
status_code=500, detail=f"Failed to get token for room: {room.url}")
# Spawn a new agent, and join the user session # Spawn a new agent, and join the user session
# Note: this is mostly for demonstration purposes (refer to 'deployment' in README) # Note: this is mostly for demonstration purposes (refer to 'deployment' in README)
try: try:
proc = subprocess.Popen( proc = subprocess.Popen(
[ [f"python3 -m bot -u {room.url} -t {token}"],
f"python3 -m bot -u {room.url} -t {token}"
],
shell=True, shell=True,
bufsize=1, bufsize=1,
cwd=os.path.dirname(os.path.abspath(__file__)) cwd=os.path.dirname(os.path.abspath(__file__)),
) )
bot_procs[proc.pid] = (proc, room.url) bot_procs[proc.pid] = (proc, room.url)
except Exception as e: except Exception as e:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Failed to start subprocess: {e}")
status_code=500, detail=f"Failed to start subprocess: {e}")
return RedirectResponse(room.url) return RedirectResponse(room.url)
@@ -107,8 +105,7 @@ def get_status(pid: int):
# If the subprocess doesn't exist, return an error # If the subprocess doesn't exist, return an error
if not proc: if not proc:
raise HTTPException( raise HTTPException(status_code=404, detail=f"Bot with process id: {pid} not found")
status_code=404, detail=f"Bot with process id: {pid} not found")
# Check the status of the subprocess # Check the status of the subprocess
if proc[0].poll() is None: if proc[0].poll() is None:
@@ -125,14 +122,10 @@ if __name__ == "__main__":
default_host = os.getenv("HOST", "0.0.0.0") default_host = os.getenv("HOST", "0.0.0.0")
default_port = int(os.getenv("FAST_API_PORT", "7860")) default_port = int(os.getenv("FAST_API_PORT", "7860"))
parser = argparse.ArgumentParser( parser = argparse.ArgumentParser(description="Daily Storyteller FastAPI server")
description="Daily Storyteller FastAPI server") parser.add_argument("--host", type=str, default=default_host, help="Host address")
parser.add_argument("--host", type=str, parser.add_argument("--port", type=int, default=default_port, help="Port number")
default=default_host, help="Host address") parser.add_argument("--reload", action="store_true", help="Reload code on change")
parser.add_argument("--port", type=int,
default=default_port, help="Port number")
parser.add_argument("--reload", action="store_true",
help="Reload code on change")
config = parser.parse_args() config = parser.parse_args()

2497
examples/storytelling-chatbot/frontend/package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

32
examples/storytelling-chatbot/frontend/package.json
View File

@@ -11,28 +11,28 @@
"dependencies": { "dependencies": {
"@daily-co/daily-js": "^0.62.0", "@daily-co/daily-js": "^0.62.0",
"@daily-co/daily-react": "^0.18.0", "@daily-co/daily-react": "^0.18.0",
"@radix-ui/react-select": "^2.0.0", "@radix-ui/react-select": "^2.1.2",
"@radix-ui/react-slot": "^1.0.2", "@radix-ui/react-slot": "^1.0.2",
"@tabler/icons-react": "^3.1.0", "@tabler/icons-react": "^3.19.0",
"class-variance-authority": "^0.7.0", "class-variance-authority": "^0.7.0",
"clsx": "^2.1.0", "clsx": "^2.1.1",
"framer-motion": "^11.0.27", "framer-motion": "^11.9.0",
"next": "14.1.4", "next": "^14.2.14",
"react": "^18", "react": "^18.3.1",
"react-dom": "^18", "react-dom": "^18.3.1",
"recoil": "^0.7.7", "recoil": "^0.7.7",
"tailwind-merge": "^2.2.2", "tailwind-merge": "^2.5.2",
"tailwindcss-animate": "^1.0.7" "tailwindcss-animate": "^1.0.7"
}, },
"devDependencies": { "devDependencies": {
"@types/node": "^20", "@types/node": "^20.16.10",
"@types/react": "^18", "@types/react": "^18.3.11",
"@types/react-dom": "^18", "@types/react-dom": "^18.3.0",
"autoprefixer": "^10.0.1", "autoprefixer": "^10.4.20",
"eslint": "^8", "eslint": "^8.57.1",
"eslint-config-next": "14.1.4", "eslint-config-next": "14.1.4",
"postcss": "^8", "postcss": "^8.4.47",
"tailwindcss": "^3.4.3", "tailwindcss": "^3.4.13",
"typescript": "^5" "typescript": "^5.6.2"
} }
} }

3208
examples/storytelling-chatbot/frontend/yarn.lock
View File

File diff suppressed because it is too large Load Diff

2
examples/storytelling-chatbot/requirements.txt
View File

@@ -2,4 +2,4 @@ async_timeout
fastapi fastapi
uvicorn uvicorn
python-dotenv python-dotenv
pipecat-ai[daily,openai,fal] pipecat-ai[daily,elevenlabs,openai,fal]

56
examples/storytelling-chatbot/src/bot.py
View File

@@ -9,11 +9,18 @@ from pipecat.frames.frames import LLMMessagesFrame, StopTaskFrame, EndFrame
from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask from pipecat.pipeline.task import PipelineTask
from pipecat.processors.aggregators.llm_response import LLMAssistantResponseAggregator, LLMUserResponseAggregator from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.services.elevenlabs import ElevenLabsTTSService from pipecat.services.elevenlabs import ElevenLabsTTSService
from pipecat.services.fal import FalImageGenService from pipecat.services.fal import FalImageGenService
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport, DailyTransportMessageFrame from pipecat.transports.services.daily import (
DailyParams,
DailyTransport,
DailyTransportMessageFrame,
)
from processors import StoryProcessor, StoryImageProcessor from processors import StoryProcessor, StoryImageProcessor
from prompts import LLM_BASE_PROMPT, LLM_INTRO_PROMPT, CUE_USER_TURN from prompts import LLM_BASE_PROMPT, LLM_INTRO_PROMPT, CUE_USER_TURN
@@ -22,6 +29,7 @@ from utils.helpers import load_sounds, load_images
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -33,7 +41,6 @@ images = load_images(["book1.png", "book2.png"])
async def main(room_url, token=None): async def main(room_url, token=None):
async with aiohttp.ClientSession() as session: async with aiohttp.ClientSession() as session:
# -------------- Transport --------------- # # -------------- Transport --------------- #
transport = DailyTransport( transport = DailyTransport(
@@ -47,17 +54,14 @@ async def main(room_url, token=None):
camera_out_height=768, camera_out_height=768,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
) ),
) )
logger.debug("Transport created for room:" + room_url) logger.debug("Transport created for room:" + room_url)
# -------------- Services --------------- # # -------------- Services --------------- #
llm_service = OpenAILLMService( llm_service = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o"
)
tts_service = ElevenLabsTTSService( tts_service = ElevenLabsTTSService(
api_key=os.getenv("ELEVENLABS_API_KEY"), api_key=os.getenv("ELEVENLABS_API_KEY"),
@@ -65,10 +69,7 @@ async def main(room_url, token=None):
) )
fal_service_params = FalImageGenService.InputParams( fal_service_params = FalImageGenService.InputParams(
image_size={ image_size={"width": 768, "height": 768}
"width": 768,
"height": 768
}
) )
fal_service = FalImageGenService( fal_service = FalImageGenService(
@@ -110,12 +111,12 @@ async def main(room_url, token=None):
transport.capture_participant_transcription(participant["id"]) transport.capture_participant_transcription(participant["id"])
await intro_task.queue_frames( await intro_task.queue_frames(
[ [
images['book1'], images["book1"],
LLMMessagesFrame([LLM_INTRO_PROMPT]), LLMMessagesFrame([LLM_INTRO_PROMPT]),
DailyTransportMessageFrame(CUE_USER_TURN), DailyTransportMessageFrame(CUE_USER_TURN),
sounds["listening"], sounds["listening"],
images['book2'], images["book2"],
StopTaskFrame() StopTaskFrame(),
] ]
) )
@@ -125,22 +126,24 @@ async def main(room_url, token=None):
# The main story pipeline is used to continue the story based on user # The main story pipeline is used to continue the story based on user
# input. # input.
main_pipeline = Pipeline([ main_pipeline = Pipeline(
transport.input(), [
user_responses, transport.input(),
llm_service, user_responses,
story_processor, llm_service,
image_processor, story_processor,
tts_service, image_processor,
transport.output(), tts_service,
llm_responses transport.output(),
]) llm_responses,
]
)
main_task = PipelineTask(main_pipeline) main_task = PipelineTask(main_pipeline)
@transport.event_handler("on_participant_left") @transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason): async def on_participant_left(transport, participant, reason):
intro_task.queue_frame(EndFrame()) await intro_task.queue_frame(EndFrame())
await main_task.queue_frame(EndFrame()) await main_task.queue_frame(EndFrame())
@transport.event_handler("on_call_state_updated") @transport.event_handler("on_call_state_updated")
@@ -150,6 +153,7 @@ async def main(room_url, token=None):
await runner.run(main_task) await runner.run(main_task)
if __name__ == "__main__": if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Daily Storyteller Bot") parser = argparse.ArgumentParser(description="Daily Storyteller Bot")
parser.add_argument("-u", type=str, help="Room URL") parser.add_argument("-u", type=str, help="Room URL")

110
examples/storytelling-chatbot/src/bot_runner.py
View File

@@ -20,10 +20,15 @@ from fastapi.staticfiles import StaticFiles
from fastapi.responses import FileResponse, JSONResponse from fastapi.responses import FileResponse, JSONResponse
from pipecat.transports.services.helpers.daily_rest import ( from pipecat.transports.services.helpers.daily_rest import (
DailyRESTHelper, DailyRoomObject, DailyRoomProperties, DailyRoomParams) DailyRESTHelper,
DailyRoomObject,
DailyRoomProperties,
DailyRoomParams,
)
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
# ------------ Fast API Config ------------ # # ------------ Fast API Config ------------ #
@@ -38,12 +43,13 @@ async def lifespan(app: FastAPI):
aiohttp_session = aiohttp.ClientSession() aiohttp_session = aiohttp.ClientSession()
daily_helpers["rest"] = DailyRESTHelper( daily_helpers["rest"] = DailyRESTHelper(
daily_api_key=os.getenv("DAILY_API_KEY", ""), daily_api_key=os.getenv("DAILY_API_KEY", ""),
daily_api_url=os.getenv("DAILY_API_URL", 'https://api.daily.co/v1'), daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
aiohttp_session=aiohttp_session aiohttp_session=aiohttp_session,
) )
yield yield
await aiohttp_session.close() await aiohttp_session.close()
app = FastAPI(lifespan=lifespan) app = FastAPI(lifespan=lifespan)
app.add_middleware( app.add_middleware(
@@ -85,55 +91,50 @@ async def start_bot(request: Request) -> JSONResponse:
room_url = os.getenv("DAILY_SAMPLE_ROOM_URL", "") room_url = os.getenv("DAILY_SAMPLE_ROOM_URL", "")
if not room_url: if not room_url:
params = DailyRoomParams( params = DailyRoomParams(properties=DailyRoomProperties())
properties=DailyRoomProperties()
)
try: try:
room: DailyRoomObject = await daily_helpers["rest"].create_room(params=params) room: DailyRoomObject = await daily_helpers["rest"].create_room(params=params)
except Exception as e: except Exception as e:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Unable to provision room {e}")
status_code=500,
detail=f"Unable to provision room {e}")
else: else:
# Check passed room URL exists, we should assume that it already has a sip set up # Check passed room URL exists, we should assume that it already has a sip set up
try: try:
room: DailyRoomObject = await daily_helpers["rest"].get_room_from_url(room_url) room: DailyRoomObject = await daily_helpers["rest"].get_room_from_url(room_url)
except Exception: except Exception:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Room not found: {room_url}")
status_code=500, detail=f"Room not found: {room_url}")
# Give the agent a token to join the session # Give the agent a token to join the session
token = await daily_helpers["rest"].get_token(room.url, MAX_SESSION_TIME) token = await daily_helpers["rest"].get_token(room.url, MAX_SESSION_TIME)
if not room or not token: if not room or not token:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Failed to get token for room: {room_url}")
status_code=500, detail=f"Failed to get token for room: {room_url}")
# Launch a new VM, or run as a shell process (not recommended) # Launch a new VM, or run as a shell process (not recommended)
if os.getenv("RUN_AS_VM", False): if os.getenv("RUN_AS_VM", False):
try: try:
await virtualize_bot(room.url, token) await virtualize_bot(room.url, token)
except Exception as e: except Exception as e:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Failed to spawn VM: {e}")
status_code=500, detail=f"Failed to spawn VM: {e}")
else: else:
try: try:
subprocess.Popen( subprocess.Popen(
[f"python3 -m bot -u {room.url} -t {token}"], [f"python3 -m bot -u {room.url} -t {token}"],
shell=True, shell=True,
bufsize=1, bufsize=1,
cwd=os.path.dirname(os.path.abspath(__file__))) cwd=os.path.dirname(os.path.abspath(__file__)),
)
except Exception as e: except Exception as e:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Failed to start subprocess: {e}")
status_code=500, detail=f"Failed to start subprocess: {e}")
# Grab a token for the user to join with # Grab a token for the user to join with
user_token = await daily_helpers["rest"].get_token(room.url, MAX_SESSION_TIME) user_token = await daily_helpers["rest"].get_token(room.url, MAX_SESSION_TIME)
return JSONResponse({ return JSONResponse(
"room_url": room.url, {
"token": user_token, "room_url": room.url,
}) "token": user_token,
}
)
@app.get("/{path_name:path}", response_class=FileResponse) @app.get("/{path_name:path}", response_class=FileResponse)
@@ -155,6 +156,7 @@ async def catch_all(path_name: Optional[str] = ""):
# ------------ Virtualization ------------ # # ------------ Virtualization ------------ #
async def virtualize_bot(room_url: str, token: str): async def virtualize_bot(room_url: str, token: str):
""" """
This is an example of how to virtualize the bot using Fly.io This is an example of how to virtualize the bot using Fly.io
@@ -163,20 +165,19 @@ async def virtualize_bot(room_url: str, token: str):
FLY_API_HOST = os.getenv("FLY_API_HOST", "https://api.machines.dev/v1") FLY_API_HOST = os.getenv("FLY_API_HOST", "https://api.machines.dev/v1")
FLY_APP_NAME = os.getenv("FLY_APP_NAME", "storytelling-chatbot") FLY_APP_NAME = os.getenv("FLY_APP_NAME", "storytelling-chatbot")
FLY_API_KEY = os.getenv("FLY_API_KEY", "") FLY_API_KEY = os.getenv("FLY_API_KEY", "")
FLY_HEADERS = { FLY_HEADERS = {"Authorization": f"Bearer {FLY_API_KEY}", "Content-Type": "application/json"}
'Authorization': f"Bearer {FLY_API_KEY}",
'Content-Type': 'application/json'
}
async with aiohttp.ClientSession() as session: async with aiohttp.ClientSession() as session:
# Use the same image as the bot runner # Use the same image as the bot runner
async with session.get(f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines", headers=FLY_HEADERS) as r: async with session.get(
f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines", headers=FLY_HEADERS
) as r:
if r.status != 200: if r.status != 200:
text = await r.text() text = await r.text()
raise Exception(f"Unable to get machine info from Fly: {text}") raise Exception(f"Unable to get machine info from Fly: {text}")
data = await r.json() data = await r.json()
image = data[0]['config']['image'] image = data[0]["config"]["image"]
# Machine configuration # Machine configuration
cmd = f"python3 src/bot.py -u {room_url} -t {token}" cmd = f"python3 src/bot.py -u {room_url} -t {token}"
@@ -185,31 +186,28 @@ async def virtualize_bot(room_url: str, token: str):
"config": { "config": {
"image": image, "image": image,
"auto_destroy": True, "auto_destroy": True,
"init": { "init": {"cmd": cmd},
"cmd": cmd "restart": {"policy": "no"},
}, "guest": {"cpu_kind": "shared", "cpus": 1, "memory_mb": 512},
"restart": {
"policy": "no"
},
"guest": {
"cpu_kind": "shared",
"cpus": 1,
"memory_mb": 512
}
}, },
} }
# Spawn a new machine instance # Spawn a new machine instance
async with session.post(f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines", headers=FLY_HEADERS, json=worker_props) as r: async with session.post(
f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines", headers=FLY_HEADERS, json=worker_props
) as r:
if r.status != 200: if r.status != 200:
text = await r.text() text = await r.text()
raise Exception(f"Problem starting a bot worker: {text}") raise Exception(f"Problem starting a bot worker: {text}")
data = await r.json() data = await r.json()
# Wait for the machine to enter the started state # Wait for the machine to enter the started state
vm_id = data['id'] vm_id = data["id"]
async with session.get(f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines/{vm_id}/wait?state=started", headers=FLY_HEADERS) as r: async with session.get(
f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines/{vm_id}/wait?state=started",
headers=FLY_HEADERS,
) as r:
if r.status != 200: if r.status != 200:
text = await r.text() text = await r.text()
raise Exception(f"Bot was unable to enter started state: {text}") raise Exception(f"Bot was unable to enter started state: {text}")
@@ -221,8 +219,13 @@ async def virtualize_bot(room_url: str, token: str):
if __name__ == "__main__": if __name__ == "__main__":
# Check environment variables # Check environment variables
required_env_vars = ['OPENAI_API_KEY', 'DAILY_API_KEY', required_env_vars = [
'FAL_KEY', 'ELEVENLABS_VOICE_ID', 'ELEVENLABS_API_KEY'] "OPENAI_API_KEY",
"DAILY_API_KEY",
"FAL_KEY",
"ELEVENLABS_VOICE_ID",
"ELEVENLABS_API_KEY",
]
for env_var in required_env_vars: for env_var in required_env_vars:
if env_var not in os.environ: if env_var not in os.environ:
raise Exception(f"Missing environment variable: {env_var}.") raise Exception(f"Missing environment variable: {env_var}.")
@@ -232,20 +235,11 @@ if __name__ == "__main__":
default_host = os.getenv("HOST", "0.0.0.0") default_host = os.getenv("HOST", "0.0.0.0")
default_port = int(os.getenv("FAST_API_PORT", "7860")) default_port = int(os.getenv("FAST_API_PORT", "7860"))
parser = argparse.ArgumentParser( parser = argparse.ArgumentParser(description="Daily Storyteller FastAPI server")
description="Daily Storyteller FastAPI server") parser.add_argument("--host", type=str, default=default_host, help="Host address")
parser.add_argument("--host", type=str, parser.add_argument("--port", type=int, default=default_port, help="Port number")
default=default_host, help="Host address") parser.add_argument("--reload", action="store_true", help="Reload code on change")
parser.add_argument("--port", type=int,
default=default_port, help="Port number")
parser.add_argument("--reload", action="store_true",
help="Reload code on change")
config = parser.parse_args() config = parser.parse_args()
uvicorn.run( uvicorn.run("bot_runner:app", host=config.host, port=config.port, reload=config.reload)
"bot_runner:app",
host=config.host,
port=config.port,
reload=config.reload
)

9
examples/storytelling-chatbot/src/processors.py
View File

@@ -6,7 +6,8 @@ from pipecat.frames.frames import (
Frame, Frame,
LLMFullResponseEndFrame, LLMFullResponseEndFrame,
TextFrame, TextFrame,
UserStoppedSpeakingFrame) UserStoppedSpeakingFrame,
)
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.transports.services.daily import DailyTransportMessageFrame from pipecat.transports.services.daily import DailyTransportMessageFrame
@@ -35,6 +36,7 @@ class StoryPromptFrame(TextFrame):
# ------------ Frame Processors ----------- # # ------------ Frame Processors ----------- #
class StoryImageProcessor(FrameProcessor): class StoryImageProcessor(FrameProcessor):
""" """
Processor for image prompt frames that will be sent to the FAL service. Processor for image prompt frames that will be sent to the FAL service.
@@ -113,7 +115,7 @@ class StoryProcessor(FrameProcessor):
# Extract the image prompt from the text using regex # Extract the image prompt from the text using regex
image_prompt = re.search(r"<(.*?)>", self._text).group(1) image_prompt = re.search(r"<(.*?)>", self._text).group(1)
# Remove the image prompt from the text # Remove the image prompt from the text
self._text = re.sub(r"<.*?>", '', self._text, count=1) self._text = re.sub(r"<.*?>", "", self._text, count=1)
# Process the image prompt frame # Process the image prompt frame
await self.push_frame(StoryImageFrame(image_prompt)) await self.push_frame(StoryImageFrame(image_prompt))
@@ -124,8 +126,7 @@ class StoryProcessor(FrameProcessor):
if re.search(r".*\[[bB]reak\].*", self._text): if re.search(r".*\[[bB]reak\].*", self._text):
# Remove the [break] token from the text # Remove the [break] token from the text
# so it isn't spoken out loud by the TTS # so it isn't spoken out loud by the TTS
self._text = re.sub(r'\[[bB]reak\]', '', self._text = re.sub(r"\[[bB]reak\]", "", self._text, flags=re.IGNORECASE)
self._text, flags=re.IGNORECASE)
self._text = self._text.replace("\n", " ") self._text = self._text.replace("\n", " ")
if len(self._text) > 2: if len(self._text) > 2:
# Append the sentence to the story # Append the sentence to the story

4
examples/storytelling-chatbot/src/prompts.py
View File

@@ -3,7 +3,7 @@ LLM_INTRO_PROMPT = {
"content": "You are a creative storyteller who loves to tell whimsical, fantastical stories. \ "content": "You are a creative storyteller who loves to tell whimsical, fantastical stories. \
Your goal is to craft an engaging and fun story. \ Your goal is to craft an engaging and fun story. \
Start by asking the user what kind of story they'd like to hear. Don't provide any examples. \ Start by asking the user what kind of story they'd like to hear. Don't provide any examples. \
Keep your response to only a few sentences." Keep your response to only a few sentences.",
} }
@@ -25,7 +25,7 @@ LLM_BASE_PROMPT = {
Responses should use the format: <...> story sentence [break] <...> story sentence [break] ... \ Responses should use the format: <...> story sentence [break] <...> story sentence [break] ... \
After each response, ask me how I'd like the story to continue and wait for my input. \ After each response, ask me how I'd like the story to continue and wait for my input. \
Please ensure your responses are less than 3-4 sentences long. \ Please ensure your responses are less than 3-4 sentences long. \
Please refrain from using any explicit language or content. Do not tell scary stories." Please refrain from using any explicit language or content. Do not tell scary stories.",
} }

14
examples/storytelling-chatbot/src/utils/helpers.py
View File

@@ -2,7 +2,7 @@ import os
import wave import wave
from PIL import Image from PIL import Image
from pipecat.frames.frames import AudioRawFrame, ImageRawFrame from pipecat.frames.frames import OutputAudioRawFrame, OutputImageRawFrame
script_dir = os.path.dirname(__file__) script_dir = os.path.dirname(__file__)
@@ -16,7 +16,9 @@ def load_images(image_files):
filename = os.path.splitext(os.path.basename(full_path))[0] filename = os.path.splitext(os.path.basename(full_path))[0]
# Open the image and convert it to bytes # Open the image and convert it to bytes
with Image.open(full_path) as img: with Image.open(full_path) as img:
images[filename] = ImageRawFrame(image=img.tobytes(), size=img.size, format=img.format) images[filename] = OutputImageRawFrame(
image=img.tobytes(), size=img.size, format=img.format
)
return images return images
@@ -30,8 +32,10 @@ def load_sounds(sound_files):
filename = os.path.splitext(os.path.basename(full_path))[0] filename = os.path.splitext(os.path.basename(full_path))[0]
# Open the sound and convert it to bytes # Open the sound and convert it to bytes
with wave.open(full_path) as audio_file: with wave.open(full_path) as audio_file:
sounds[filename] = AudioRawFrame(audio=audio_file.readframes(-1), sounds[filename] = OutputAudioRawFrame(
sample_rate=audio_file.getframerate(), audio=audio_file.readframes(-1),
num_channels=audio_file.getnchannels()) sample_rate=audio_file.getframerate(),
num_channels=audio_file.getnchannels(),
)
return sounds return sounds

21
examples/studypal/runner.py
View File

@@ -17,16 +17,13 @@ async def configure(aiohttp_session: aiohttp.ClientSession):
async def configure_with_args( async def configure_with_args(
aiohttp_session: aiohttp.ClientSession, aiohttp_session: aiohttp.ClientSession, parser: argparse.ArgumentParser | None = None
parser: argparse.ArgumentParser | None = None): ):
if not parser: if not parser:
parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample") parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample")
parser.add_argument( parser.add_argument(
"-u", "-u", "--url", type=str, required=False, help="URL of the Daily room to join"
"--url", )
type=str,
required=False,
help="URL of the Daily room to join")
parser.add_argument( parser.add_argument(
"-k", "-k",
"--apikey", "--apikey",
@@ -42,15 +39,19 @@ async def configure_with_args(
if not url: if not url:
raise Exception( raise Exception(
"No Daily room specified. use the -u/--url option from the command line, or set DAILY_SAMPLE_ROOM_URL in your environment to specify a Daily room URL.") "No Daily room specified. use the -u/--url option from the command line, or set DAILY_SAMPLE_ROOM_URL in your environment to specify a Daily room URL."
)
if not key: if not key:
raise Exception("No Daily API key specified. use the -k/--apikey option from the command line, or set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.") raise Exception(
"No Daily API key specified. use the -k/--apikey option from the command line, or set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers."
)
daily_rest_helper = DailyRESTHelper( daily_rest_helper = DailyRESTHelper(
daily_api_key=key, daily_api_key=key,
daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"), daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
aiohttp_session=aiohttp_session) aiohttp_session=aiohttp_session,
)
# Create a meeting token for the given room with an expiration 1 hour in # Create a meeting token for the given room with an expiration 1 hour in
# the future. # the future.

68
examples/studypal/studypal.py
View File

@@ -13,7 +13,9 @@ from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import ( from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator, LLMUserResponseAggregator) LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.services.cartesia import CartesiaTTSService from pipecat.services.cartesia import CartesiaTTSService
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.transports.services.daily import DailyParams, DailyTransport
@@ -24,6 +26,7 @@ from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
# Run this script directly from your command line. # Run this script directly from your command line.
@@ -45,15 +48,17 @@ def truncate_content(content, model_name):
return encoding.decode(truncated_tokens) return encoding.decode(truncated_tokens)
return content return content
# Main function to extract content from url # Main function to extract content from url
async def get_article_content(url: str, aiohttp_session: aiohttp.ClientSession): async def get_article_content(url: str, aiohttp_session: aiohttp.ClientSession):
if 'arxiv.org' in url: if "arxiv.org" in url:
return await get_arxiv_content(url, aiohttp_session) return await get_arxiv_content(url, aiohttp_session)
else: else:
return await get_wikipedia_content(url, aiohttp_session) return await get_wikipedia_content(url, aiohttp_session)
# Helper function to extract content from Wikipedia url (this is # Helper function to extract content from Wikipedia url (this is
# technically agnostic to URL type but will work best with Wikipedia # technically agnostic to URL type but will work best with Wikipedia
# articles) # articles)
@@ -65,23 +70,24 @@ async def get_wikipedia_content(url: str, aiohttp_session: aiohttp.ClientSession
return "Failed to download Wikipedia article." return "Failed to download Wikipedia article."
text = await response.text() text = await response.text()
soup = BeautifulSoup(text, 'html.parser') soup = BeautifulSoup(text, "html.parser")
content = soup.find('div', {'class': 'mw-parser-output'}) content = soup.find("div", {"class": "mw-parser-output"})
if content: if content:
return content.get_text() return content.get_text()
else: else:
return "Failed to extract Wikipedia article content." return "Failed to extract Wikipedia article content."
# Helper function to extract content from arXiv url # Helper function to extract content from arXiv url
async def get_arxiv_content(url: str, aiohttp_session: aiohttp.ClientSession): async def get_arxiv_content(url: str, aiohttp_session: aiohttp.ClientSession):
if '/abs/' in url: if "/abs/" in url:
url = url.replace('/abs/', '/pdf/') url = url.replace("/abs/", "/pdf/")
if not url.endswith('.pdf'): if not url.endswith(".pdf"):
url += '.pdf' url += ".pdf"
async with aiohttp_session.get(url) as response: async with aiohttp_session.get(url) as response:
if response.status != 200: if response.status != 200:
@@ -95,6 +101,7 @@ async def get_arxiv_content(url: str, aiohttp_session: aiohttp.ClientSession):
text += page.extract_text() text += page.extract_text()
return text return text
# This is the main function that handles STT -> LLM -> TTS # This is the main function that handles STT -> LLM -> TTS
@@ -116,40 +123,46 @@ async def main():
audio_out_enabled=True, audio_out_enabled=True,
transcription_enabled=True, transcription_enabled=True,
vad_enabled=True, vad_enabled=True,
vad_analyzer=SileroVADAnalyzer() vad_analyzer=SileroVADAnalyzer(),
) ),
) )
tts = CartesiaTTSService( tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY"), api_key=os.getenv("CARTESIA_API_KEY"),
voice_id=os.getenv("CARTESIA_VOICE_ID", "4d2fd738-3b3d-4368-957a-bb4805275bd9"), voice_id=os.getenv("CARTESIA_VOICE_ID", "4d2fd738-3b3d-4368-957a-bb4805275bd9"),
# British Narration Lady: 4d2fd738-3b3d-4368-957a-bb4805275bd9 # British Narration Lady: 4d2fd738-3b3d-4368-957a-bb4805275bd9
sample_rate=44100, params=CartesiaTTSService.InputParams(
sample_rate=44100,
),
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o-mini")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o-mini")
messages = [{ messages = [
"role": "system", "content": f"""You are an AI study partner. You have been given the following article content: {
"role": "system",
"content": f"""You are an AI study partner. You have been given the following article content:
{article_content} {article_content}
Your task is to help the user understand and learn from this article in 2 sentences. THESE RESPONSES SHOULD BE ONLY MAX 2 SENTENCES. THIS INSTRUCTION IS VERY IMPORTANT. RESPONSES SHOULDN'T BE LONG. Your task is to help the user understand and learn from this article in 2 sentences. THESE RESPONSES SHOULD BE ONLY MAX 2 SENTENCES. THIS INSTRUCTION IS VERY IMPORTANT. RESPONSES SHOULDN'T BE LONG.
""", }, ] """,
},
]
tma_in = LLMUserResponseAggregator(messages) tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), [
tma_in, transport.input(),
llm, tma_in,
tts, llm,
tma_out, tts,
transport.output(), transport.output(),
]) tma_out,
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True, enable_metrics=True)) task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True, enable_metrics=True))
@@ -159,12 +172,15 @@ Your task is to help the user understand and learn from this article in 2 senten
messages.append( messages.append(
{ {
"role": "system", "role": "system",
"content": "Hello! I'm ready to discuss the article with you. What would you like to learn about?"}) "content": "Hello! I'm ready to discuss the article with you. What would you like to learn about?",
}
)
await task.queue_frames([LLMMessagesFrame(messages)]) await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner() runner = PipelineRunner()
await runner.run(task) await runner.run(task)
if __name__ == "__main__": if __name__ == "__main__":
asyncio.run(main()) asyncio.run(main())

32
examples/translation-chatbot/bot.py
View File

@@ -22,13 +22,15 @@ from pipecat.transports.services.daily import (
DailyParams, DailyParams,
DailyTranscriptionSettings, DailyTranscriptionSettings,
DailyTransport, DailyTransport,
DailyTransportMessageFrame) DailyTransportMessageFrame,
)
from runner import configure from runner import configure
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -44,7 +46,6 @@ It also isn't saving what the user or bot says into the context object for use i
# We need to use a custom service here to yield LLM frames without saving # We need to use a custom service here to yield LLM frames without saving
# any context # any context
class TranslationProcessor(FrameProcessor): class TranslationProcessor(FrameProcessor):
def __init__(self, language): def __init__(self, language):
super().__init__() super().__init__()
self._language = language self._language = language
@@ -80,10 +81,7 @@ class TranslationSubtitles(FrameProcessor):
await super().process_frame(frame, direction) await super().process_frame(frame, direction)
if isinstance(frame, TextFrame): if isinstance(frame, TextFrame):
message = { message = {"language": self._language, "text": frame.text}
"language": self._language,
"text": frame.text
}
await self.push_frame(DailyTransportMessageFrame(message)) await self.push_frame(DailyTransportMessageFrame(message))
await self.push_frame(frame) await self.push_frame(frame)
@@ -100,10 +98,8 @@ async def main():
DailyParams( DailyParams(
audio_out_enabled=True, audio_out_enabled=True,
transcription_enabled=True, transcription_enabled=True,
transcription_settings=DailyTranscriptionSettings(extra={ transcription_settings=DailyTranscriptionSettings(extra={"interim_results": False}),
"interim_results": False ),
})
)
) )
tts = AzureTTSService( tts = AzureTTSService(
@@ -112,26 +108,14 @@ async def main():
voice="es-ES-AlvaroNeural", voice="es-ES-AlvaroNeural",
) )
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o"
)
sa = SentenceAggregator() sa = SentenceAggregator()
tp = TranslationProcessor("Spanish") tp = TranslationProcessor("Spanish")
lfra = LLMFullResponseAggregator() lfra = LLMFullResponseAggregator()
ts = TranslationSubtitles("spanish") ts = TranslationSubtitles("spanish")
pipeline = Pipeline([ pipeline = Pipeline([transport.input(), sa, tp, llm, lfra, ts, tts, transport.output()])
transport.input(),
sa,
tp,
llm,
lfra,
ts,
tts,
transport.output()
])
task = PipelineTask(pipeline) task = PipelineTask(pipeline)

16
examples/translation-chatbot/runner.py
View File

@@ -15,11 +15,8 @@ from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper
async def configure(aiohttp_session: aiohttp.ClientSession): async def configure(aiohttp_session: aiohttp.ClientSession):
parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample") parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample")
parser.add_argument( parser.add_argument(
"-u", "-u", "--url", type=str, required=False, help="URL of the Daily room to join"
"--url", )
type=str,
required=False,
help="URL of the Daily room to join")
parser.add_argument( parser.add_argument(
"-k", "-k",
"--apikey", "--apikey",
@@ -35,15 +32,18 @@ async def configure(aiohttp_session: aiohttp.ClientSession):
if not url: if not url:
raise Exception( raise Exception(
"No Daily room specified. use the -u/--url option from the command line, or set DAILY_SAMPLE_ROOM_URL in your environment to specify a Daily room URL.") "No Daily room specified. use the -u/--url option from the command line, or set DAILY_SAMPLE_ROOM_URL in your environment to specify a Daily room URL."
)
if not key: if not key:
raise Exception("No Daily API key specified. use the -k/--apikey option from the command line, or set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.") raise Exception(
"No Daily API key specified. use the -k/--apikey option from the command line, or set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers."
)
daily_rest_helper = DailyRESTHelper( daily_rest_helper = DailyRESTHelper(
daily_api_key=key, daily_api_key=key,
daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"), daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
aiohttp_session=aiohttp_session aiohttp_session=aiohttp_session,
) )
# Create a meeting token for the given room with an expiration 1 hour in # Create a meeting token for the given room with an expiration 1 hour in

41
examples/translation-chatbot/server.py
View File

@@ -38,13 +38,14 @@ async def lifespan(app: FastAPI):
aiohttp_session = aiohttp.ClientSession() aiohttp_session = aiohttp.ClientSession()
daily_helpers["rest"] = DailyRESTHelper( daily_helpers["rest"] = DailyRESTHelper(
daily_api_key=os.getenv("DAILY_API_KEY", ""), daily_api_key=os.getenv("DAILY_API_KEY", ""),
daily_api_url=os.getenv("DAILY_API_URL", 'https://api.daily.co/v1'), daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
aiohttp_session=aiohttp_session aiohttp_session=aiohttp_session,
) )
yield yield
await aiohttp_session.close() await aiohttp_session.close()
cleanup() cleanup()
app = FastAPI(lifespan=lifespan) app = FastAPI(lifespan=lifespan)
app.add_middleware( app.add_middleware(
@@ -65,37 +66,34 @@ async def start_agent(request: Request):
if not room.url: if not room.url:
raise HTTPException( raise HTTPException(
status_code=500, status_code=500,
detail="Missing 'room' property in request data. Cannot start agent without a target room!") detail="Missing 'room' property in request data. Cannot start agent without a target room!",
)
# Check if there is already an existing process running in this room # Check if there is already an existing process running in this room
num_bots_in_room = sum( num_bots_in_room = sum(
1 for proc in bot_procs.values() if proc[1] == room.url and proc[0].poll() is None) 1 for proc in bot_procs.values() if proc[1] == room.url and proc[0].poll() is None
)
if num_bots_in_room >= MAX_BOTS_PER_ROOM: if num_bots_in_room >= MAX_BOTS_PER_ROOM:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Max bot limited reach for room: {room.url}")
status_code=500, detail=f"Max bot limited reach for room: {room.url}")
# Get the token for the room # Get the token for the room
token = await daily_helpers["rest"].get_token(room.url) token = await daily_helpers["rest"].get_token(room.url)
if not token: if not token:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Failed to get token for room: {room.url}")
status_code=500, detail=f"Failed to get token for room: {room.url}")
# Spawn a new agent, and join the user session # Spawn a new agent, and join the user session
# Note: this is mostly for demonstration purposes (refer to 'deployment' in README) # Note: this is mostly for demonstration purposes (refer to 'deployment' in README)
try: try:
proc = subprocess.Popen( proc = subprocess.Popen(
[ [f"python3 -m bot -u {room.url} -t {token}"],
f"python3 -m bot -u {room.url} -t {token}"
],
shell=True, shell=True,
bufsize=1, bufsize=1,
cwd=os.path.dirname(os.path.abspath(__file__)) cwd=os.path.dirname(os.path.abspath(__file__)),
) )
bot_procs[proc.pid] = (proc, room.url) bot_procs[proc.pid] = (proc, room.url)
except Exception as e: except Exception as e:
raise HTTPException( raise HTTPException(status_code=500, detail=f"Failed to start subprocess: {e}")
status_code=500, detail=f"Failed to start subprocess: {e}")
return RedirectResponse(room.url) return RedirectResponse(room.url)
@@ -107,8 +105,7 @@ def get_status(pid: int):
# If the subprocess doesn't exist, return an error # If the subprocess doesn't exist, return an error
if not proc: if not proc:
raise HTTPException( raise HTTPException(status_code=404, detail=f"Bot with process id: {pid} not found")
status_code=404, detail=f"Bot with process id: {pid} not found")
# Check the status of the subprocess # Check the status of the subprocess
if proc[0].poll() is None: if proc[0].poll() is None:
@@ -125,14 +122,10 @@ if __name__ == "__main__":
default_host = os.getenv("HOST", "0.0.0.0") default_host = os.getenv("HOST", "0.0.0.0")
default_port = int(os.getenv("FAST_API_PORT", "7860")) default_port = int(os.getenv("FAST_API_PORT", "7860"))
parser = argparse.ArgumentParser( parser = argparse.ArgumentParser(description="Daily Storyteller FastAPI server")
description="Daily Storyteller FastAPI server") parser.add_argument("--host", type=str, default=default_host, help="Host address")
parser.add_argument("--host", type=str, parser.add_argument("--port", type=int, default=default_port, help="Port number")
default=default_host, help="Host address") parser.add_argument("--reload", action="store_true", help="Reload code on change")
parser.add_argument("--port", type=int,
default=default_port, help="Port number")
parser.add_argument("--reload", action="store_true",
help="Reload code on change")
config = parser.parse_args() config = parser.parse_args()

2
examples/twilio-chatbot/README.md
View File

@@ -55,7 +55,7 @@ This project is a FastAPI-based chatbot that integrates with Twilio to handle We
2. **Update the Twilio Webhook**: 2. **Update the Twilio Webhook**:
Copy the ngrok URL and update your Twilio phone number webhook URL to `http://<ngrok_url>/start_call`. Copy the ngrok URL and update your Twilio phone number webhook URL to `http://<ngrok_url>/start_call`.
3. **Update the streams.xml**: 3. **Update streams.xml**:
Copy the ngrok URL and update templates/streams.xml with `wss://<ngrok_url>/ws`. Copy the ngrok URL and update templates/streams.xml with `wss://<ngrok_url>/ws`.
## Running the Application ## Running the Application

103
examples/twilio-chatbot/bot.py
View File

@@ -1,4 +1,3 @@
import aiohttp
import os import os
import sys import sys
@@ -8,18 +7,22 @@ from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import ( from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator, LLMAssistantResponseAggregator,
LLMUserResponseAggregator LLMUserResponseAggregator,
) )
from pipecat.services.cartesia import CartesiaTTSService from pipecat.services.cartesia import CartesiaTTSService
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
from pipecat.services.deepgram import DeepgramSTTService from pipecat.services.deepgram import DeepgramSTTService
from pipecat.transports.network.fastapi_websocket import FastAPIWebsocketTransport, FastAPIWebsocketParams from pipecat.transports.network.fastapi_websocket import (
FastAPIWebsocketTransport,
FastAPIWebsocketParams,
)
from pipecat.vad.silero import SileroVADAnalyzer from pipecat.vad.silero import SileroVADAnalyzer
from pipecat.serializers.twilio import TwilioFrameSerializer from pipecat.serializers.twilio import TwilioFrameSerializer
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -27,63 +30,61 @@ logger.add(sys.stderr, level="DEBUG")
async def run_bot(websocket_client, stream_sid): async def run_bot(websocket_client, stream_sid):
async with aiohttp.ClientSession() as session: transport = FastAPIWebsocketTransport(
transport = FastAPIWebsocketTransport( websocket=websocket_client,
websocket=websocket_client, params=FastAPIWebsocketParams(
params=FastAPIWebsocketParams( audio_out_enabled=True,
audio_out_enabled=True, add_wav_header=False,
add_wav_header=False, vad_enabled=True,
vad_enabled=True, vad_analyzer=SileroVADAnalyzer(),
vad_analyzer=SileroVADAnalyzer(), vad_audio_passthrough=True,
vad_audio_passthrough=True, serializer=TwilioFrameSerializer(stream_sid),
serializer=TwilioFrameSerializer(stream_sid) ),
) )
)
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
stt = DeepgramSTTService(api_key=os.getenv('DEEPGRAM_API_KEY')) stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
tts = CartesiaTTSService( tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY"), api_key=os.getenv("CARTESIA_API_KEY"),
voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
) )
messages = [ messages = [
{ {
"role": "system", "role": "system",
"content": "You are a helpful LLM in an audio call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.", "content": "You are a helpful LLM in an audio call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
}, },
] ]
tma_in = LLMUserResponseAggregator(messages) tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Websocket input from client [
stt, # Speech-To-Text transport.input(), # Websocket input from client
tma_in, # User responses stt, # Speech-To-Text
llm, # LLM tma_in, # User responses
tts, # Text-To-Speech llm, # LLM
tts, # Text-To-Speech
transport.output(), # Websocket output to client transport.output(), # Websocket output to client
tma_out # LLM responses tma_out, # LLM responses
]) ]
)
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True)) task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
@transport.event_handler("on_client_connected") @transport.event_handler("on_client_connected")
async def on_client_connected(transport, client): async def on_client_connected(transport, client):
# Kick off the conversation. # Kick off the conversation.
messages.append( messages.append({"role": "system", "content": "Please introduce yourself to the user."})
{"role": "system", "content": "Please introduce yourself to the user."}) await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([LLMMessagesFrame(messages)])
@transport.event_handler("on_client_disconnected") @transport.event_handler("on_client_disconnected")
async def on_client_disconnected(transport, client): async def on_client_disconnected(transport, client):
await task.queue_frames([EndFrame()]) await task.queue_frames([EndFrame()])
runner = PipelineRunner(handle_sigint=False) runner = PipelineRunner(handle_sigint=False)
await runner.run(task) await runner.run(task)

2
examples/twilio-chatbot/requirements.txt
View File

@@ -1,4 +1,4 @@
pipecat-ai[daily,openai,silero,deepgram] pipecat-ai[daily,cartesia,openai,silero,deepgram]
fastapi fastapi
uvicorn uvicorn
python-dotenv python-dotenv

4
examples/twilio-chatbot/server.py
View File

@@ -19,7 +19,7 @@ app.add_middleware(
) )
@app.post('/start_call') @app.post("/start_call")
async def start_call(): async def start_call():
print("POST TwiML") print("POST TwiML")
return HTMLResponse(content=open("templates/streams.xml").read(), media_type="application/xml") return HTMLResponse(content=open("templates/streams.xml").read(), media_type="application/xml")
@@ -32,7 +32,7 @@ async def websocket_endpoint(websocket: WebSocket):
await start_data.__anext__() await start_data.__anext__()
call_data = json.loads(await start_data.__anext__()) call_data = json.loads(await start_data.__anext__())
print(call_data, flush=True) print(call_data, flush=True)
stream_sid = call_data['start']['streamSid'] stream_sid = call_data["start"]["streamSid"]
print("WebSocket connection accepted") print("WebSocket connection accepted")
await run_bot(websocket, stream_sid) await run_bot(websocket, stream_sid)

92
examples/websocket-server/bot.py
View File

@@ -4,7 +4,6 @@
# SPDX-License-Identifier: BSD 2-Clause License # SPDX-License-Identifier: BSD 2-Clause License
# #
import aiohttp
import asyncio import asyncio
import os import os
import sys import sys
@@ -15,17 +14,21 @@ from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask from pipecat.pipeline.task import PipelineTask
from pipecat.processors.aggregators.llm_response import ( from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator, LLMAssistantResponseAggregator,
LLMUserResponseAggregator LLMUserResponseAggregator,
) )
from pipecat.services.cartesia import CartesiaTTSService from pipecat.services.cartesia import CartesiaTTSService
from pipecat.services.deepgram import DeepgramSTTService from pipecat.services.deepgram import DeepgramSTTService
from pipecat.services.openai import OpenAILLMService from pipecat.services.openai import OpenAILLMService
from pipecat.transports.network.websocket_server import WebsocketServerParams, WebsocketServerTransport from pipecat.transports.network.websocket_server import (
WebsocketServerParams,
WebsocketServerTransport,
)
from pipecat.vad.silero import SileroVADAnalyzer from pipecat.vad.silero import SileroVADAnalyzer
from loguru import logger from loguru import logger
from dotenv import load_dotenv from dotenv import load_dotenv
load_dotenv(override=True) load_dotenv(override=True)
logger.remove(0) logger.remove(0)
@@ -33,60 +36,59 @@ logger.add(sys.stderr, level="DEBUG")
async def main(): async def main():
async with aiohttp.ClientSession() as session: transport = WebsocketServerTransport(
transport = WebsocketServerTransport( params=WebsocketServerParams(
params=WebsocketServerParams( audio_out_enabled=True,
audio_out_enabled=True, add_wav_header=True,
add_wav_header=True, vad_enabled=True,
vad_enabled=True, vad_analyzer=SileroVADAnalyzer(),
vad_analyzer=SileroVADAnalyzer(), vad_audio_passthrough=True,
vad_audio_passthrough=True
)
) )
)
llm = OpenAILLMService( llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4o")
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY")) stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
tts = CartesiaTTSService( tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY"), api_key=os.getenv("CARTESIA_API_KEY"),
voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
) )
messages = [ messages = [
{ {
"role": "system", "role": "system",
"content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.", "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
}, },
] ]
tma_in = LLMUserResponseAggregator(messages) tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages) tma_out = LLMAssistantResponseAggregator(messages)
pipeline = Pipeline([ pipeline = Pipeline(
transport.input(), # Websocket input from client [
stt, # Speech-To-Text transport.input(), # Websocket input from client
tma_in, # User responses stt, # Speech-To-Text
llm, # LLM tma_in, # User responses
tts, # Text-To-Speech llm, # LLM
tts, # Text-To-Speech
transport.output(), # Websocket output to client transport.output(), # Websocket output to client
tma_out # LLM responses tma_out, # LLM responses
]) ]
)
task = PipelineTask(pipeline) task = PipelineTask(pipeline)
@transport.event_handler("on_client_connected") @transport.event_handler("on_client_connected")
async def on_client_connected(transport, client): async def on_client_connected(transport, client):
# Kick off the conversation. # Kick off the conversation.
messages.append( messages.append({"role": "system", "content": "Please introduce yourself to the user."})
{"role": "system", "content": "Please introduce yourself to the user."}) await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([LLMMessagesFrame(messages)])
runner = PipelineRunner() runner = PipelineRunner()
await runner.run(task)
await runner.run(task)
if __name__ == "__main__": if __name__ == "__main__":
asyncio.run(main()) asyncio.run(main())

1
examples/websocket-server/frames.proto
View File

@@ -24,6 +24,7 @@ message AudioRawFrame {
bytes audio = 3; bytes audio = 3;
uint32 sample_rate = 4; uint32 sample_rate = 4;
uint32 num_channels = 5; uint32 num_channels = 5;
optional uint64 pts = 6;
} }
message TranscriptionFrame { message TranscriptionFrame {

2
examples/websocket-server/requirements.txt
View File

@@ -1,2 +1,2 @@
python-dotenv python-dotenv
pipecat-ai[openai,silero,websocket,whisper] pipecat-ai[cartesia,openai,silero,websocket,whisper]

15
pyproject.toml
View File

@@ -35,29 +35,30 @@ Website = "https://pipecat.ai"
[project.optional-dependencies] [project.optional-dependencies]
anthropic = [ "anthropic~=0.34.0" ] anthropic = [ "anthropic~=0.34.0" ]
aws = [ "boto3~=1.35.27" ]
azure = [ "azure-cognitiveservices-speech~=1.40.0" ] azure = [ "azure-cognitiveservices-speech~=1.40.0" ]
canonical = [ "aiofiles~=24.1.0" ] canonical = [ "aiofiles~=24.1.0" ]
cartesia = [ "websockets~=12.0" ] cartesia = [ "cartesia~=1.0.13", "websockets~=12.0" ]
daily = [ "daily-python~=0.10.1" ] daily = [ "daily-python~=0.11.0" ]
deepgram = [ "deepgram-sdk~=3.5.0" ] deepgram = [ "deepgram-sdk~=3.5.0" ]
elevenlabs = [ "elevenlabs~=1.7.0" ] elevenlabs = [ "websockets~=12.0" ]
examples = [ "python-dotenv~=1.0.1", "flask~=3.0.3", "flask_cors~=4.0.1" ] examples = [ "python-dotenv~=1.0.1", "flask~=3.0.3", "flask_cors~=4.0.1" ]
fal = [ "fal-client~=0.4.1" ] fal = [ "fal-client~=0.4.1" ]
gladia = [ "websockets~=12.0" ] gladia = [ "websockets~=12.0" ]
google = [ "google-generativeai~=0.7.2" ] google = [ "google-generativeai~=0.7.2", "google-cloud-texttospeech~=2.17.2" ]
gstreamer = [ "pygobject~=3.48.2" ] gstreamer = [ "pygobject~=3.48.2" ]
fireworks = [ "openai~=1.37.2" ] fireworks = [ "openai~=1.37.2" ]
langchain = [ "langchain~=0.2.14", "langchain-community~=0.2.12", "langchain-openai~=0.1.20" ] langchain = [ "langchain~=0.2.14", "langchain-community~=0.2.12", "langchain-openai~=0.1.20" ]
livekit = [ "livekit~=0.13.1" ] livekit = [ "livekit~=0.13.1", "tenacity~=9.0.0" ]
lmnt = [ "lmnt~=1.1.4" ] lmnt = [ "lmnt~=1.1.4" ]
local = [ "pyaudio~=0.2.14" ] local = [ "pyaudio~=0.2.14" ]
moondream = [ "einops~=0.8.0", "timm~=1.0.8", "transformers~=4.44.0" ] moondream = [ "einops~=0.8.0", "timm~=1.0.8", "transformers~=4.44.0" ]
openai = [ "openai~=1.37.2" ] openai = [ "openai~=1.37.2" ]
openpipe = [ "openpipe~=4.24.0" ] openpipe = [ "openpipe~=4.24.0" ]
playht = [ "pyht~=0.0.28" ] playht = [ "pyht~=0.0.28" ]
silero = [ "silero-vad~=5.1" ] silero = [ "onnxruntime>=1.16.1" ]
together = [ "together~=1.2.7" ] together = [ "together~=1.2.7" ]
websocket = [ "websockets~=12.0", "fastapi~=0.112.1" ] websocket = [ "websockets~=12.0", "fastapi~=0.115.0" ]
whisper = [ "faster-whisper~=1.0.3" ] whisper = [ "faster-whisper~=1.0.3" ]
xtts = [ "resampy~=0.4.3" ] xtts = [ "resampy~=0.4.3" ]

0
src/pipecat/clocks/__init__.py Normal file
View File

17
src/pipecat/clocks/base_clock.py Normal file
View File

@@ -0,0 +1,17 @@
#
# Copyright (c) 2024, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
from abc import ABC, abstractmethod
class BaseClock(ABC):
@abstractmethod
def get_time(self) -> int:
pass
@abstractmethod
def start(self):
pass

Some files were not shown because too many files have changed in this diff Show More