[ENG-7781] Enable docker build without git

Install Silero VAD via pip

.github/workflows/publish_test.yaml

@@ -1,10 +1,6 @@
 name: publish-test
 
-on:
-  workflow_dispatch:
-  push:
-    branches:
-      - main
+on: workflow_dispatch
 
 jobs:
   build:
@@ -14,7 +10,6 @@ jobs:
       - name: Checkout repo
         uses: actions/checkout@v4
         with:
-          ref: ${{ github.event.inputs.gitref }}
           fetch-tags: true
           fetch-depth: 100
       - name: Set up Python

CHANGELOG.md

@@ -5,6 +5,109 @@ All notable changes to **pipecat** will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Added
+
+- `ElevenLabsTTSService` can now specify ElevenLabs input parameters such as
+  `output_format`.
+
+- `TwilioFrameSerializer` can now specify Twilio's and Pipecat's desired sample
+  rates to use.
+
+- Added new `on_participant_updated` event to `DailyTransport`.
+
+- Added `DailyRESTHelper.delete_room_by_name()`.
+
+- Added LLM and TTS usage metrics. Those will be enabled by when
+  `enable_usage_metrics` is True.
+
+- `AudioRawFrame`s are not pushed downstream from the base output
+  transport. This allows capturing the exact words the bot says by adding an STT
+  service at the end of the pipeline.
+
+- Added new `GStreamerPipelineSource`. This processor can generate image or
+  audio frames from a GStreamer pipeline (e.g. reading an MP4 file, and RTP
+  stream or anything supported by GStreamer).
+
+- Added `TransportParams.audio_out_is_live`. This flag is False by default and
+  it is useful to indicate we should not synchronize audio with sporadic images.
+
+- Added new `BotStartedSpeakingFrame` and `BotStoppedSpeakingFrame` control
+  frames. These frames are pushed upstream and they should wrap
+  `BotSpeakingFrame`.
+
+- Transports now allow you to register event handlers without decorators.
+
+### Changed
+
+- `ElevenLabsTTSService` now uses `eleven_turbo_v2_5` model by default.
+
+- `BotSpeakingFrame` is now a control frame.
+
+- `StartFrame` is now a control frame similar to `EndFrame`.
+
+- `DeepgramTTSService` now is more customizable. You can adjust the encoding and
+  sample rate.
+
+### Fixed
+
+- Fixed and issue with `DailyRESTHelper.create_room()` expirations which would
+  cause this function to stop working after the initial expiration elapsed.
+
+- Improved `EndFrame` and `CancelFrame` handling. `EndFrame` should end things
+  gracefully while a `CancelFrame` should cancel all running tasks as soon as
+  possible.
+
+- Fixed an issue in `AIService` that would cause a yielded `None` value to be
+  processed.
+
+- RTVI's `bot-ready` message is now sent when the RTVI pipeline is ready and
+  a first participant joins.
+
+- Fixed a `BaseInputTransport` issue that was causing incoming system frames to
+  be queued instead of being pushed immediately.
+
+- Fixed a `BaseInputTransport` issue that was causing start/stop interruptions
+  incoming frames to not cancel tasks and be processed properly.
+
+### Other
+
+- Added examples `foundational/18-gstreamer-filesrc.py` and
+  `foundational/18a-gstreamer-videotestsrc.py` that show how to use
+  `GStreamerPipelineSource`
+
+- Remove `requests` library usage.
+
+- Cleanup examples and use `DailyRESTHelper`.
+
+## [0.0.39] - 2024-07-23
+
+### Fixed
+
+- Fixed a regression introduced in 0.0.38 that would cause Daily transcription
+  to stop the Pipeline.
+
+## [0.0.38] - 2024-07-23
+
+### Added
+
+- Added `force_reload`, `skip_validation` and `trust_repo` to `SileroVAD` and
+  `SileroVADAnalyzer`. This allows caching and various GitHub repo validations.
+
+- Added `send_initial_empty_metrics` flag to `PipelineParams` to request for
+  initial empty metrics (zero values). True by default.
+
+### Fixed
+
+- Fixed initial metrics format. It was using the wrong keys name/time instead of
+  processor/value.
+
+- STT services should be using ISO 8601 time format for transcription frames.
+
+- Fixed an issue that would cause Daily transport to show a stop transcription
+  error when actually none occurred.
+
 ## [0.0.37] - 2024-07-22
 
 ### Added

dev-requirements.txt

@@ -4,5 +4,5 @@ grpcio-tools~=1.62.2
 pip-tools~=7.4.1
 pyright~=1.1.367
 pytest~=8.2.0
-setuptools~=69.5.1
+setuptools~=71.1.0
 setuptools_scm~=8.1.0

examples/deployment/flyio-example/Dockerfile

@@ -9,8 +9,5 @@ COPY *.py .
 COPY ./requirements.txt requirements.txt
 RUN pip3 install --no-cache-dir --upgrade -r requirements.txt
 
-# Install models
-RUN python3 install_deps.py
-
 # Start the FastAPI server
 CMD python3 bot_runner.py --port ${FAST_API_PORT}

examples/deployment/flyio-example/README.md

@@ -2,8 +2,6 @@
 
 This project modifies the `bot_runner.py` server to launch a new machine for each user session. This is a recommended approach for production vs. running shell processess as your deployment will quickly run out of system resources under load.
 
-To speed up machine boot times, we also download and cache Silero VAD as part of the Dockerfile (`install_deps.py`). If you are using other custom models, you can add them here too.
-
 For this example, we are using Daily as a WebRTC transport and provisioning a new room and token for each session. You can use another transport, such as WebSockets, by modifying the `bot.py` and `bot_runner.py` files accordingly.
 
 ## Setting up your fly.io deployment
@@ -14,7 +12,7 @@ You can copy the `example-fly.toml` as a reference. Be sure to change the app na
 
 ### Create your .env file
 
-Copy the base `env.example` to `.env` and enter the necessary API keys. 
+Copy the base `env.example` to `.env` and enter the necessary API keys.
 
 `FLY_APP_NAME` should match that in the `fly.toml` file.
 
@@ -32,7 +30,6 @@ Note: you can do this manually via the fly.io dashboard under the "secrets" sub-
 
 `fly deploy`
 
-
 ## Connecting to your bot
 
 Send a post request to your running fly.io instance:
@@ -40,4 +37,3 @@ Send a post request to your running fly.io instance:
 `curl --location --request POST 'https://YOUR_FLY_APP_NAME/start_bot'`
 
 This request will wait until the machine enters into a `starting` state, before returning the a room URL and token to join.
-

examples/deployment/flyio-example/bot_runner.py

@@ -1,14 +1,23 @@
-import os
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import aiohttp
 import argparse
 import subprocess
-import requests
+import os
 
-from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper, DailyRoomObject, DailyRoomProperties, DailyRoomParams
+from contextlib import asynccontextmanager
 
 from fastapi import FastAPI, Request, HTTPException
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.responses import JSONResponse
 
+from pipecat.transports.services.helpers.daily_rest import (
+    DailyRESTHelper, DailyRoomObject, DailyRoomProperties, DailyRoomParams)
+
 from dotenv import load_dotenv
 load_dotenv(override=True)
 
@@ -32,14 +41,23 @@ FLY_HEADERS = {
     'Content-Type': 'application/json'
 }
 
-daily_rest_helper = DailyRESTHelper(
-    os.getenv("DAILY_API_KEY", ""),
-    os.getenv("DAILY_API_URL", 'https://api.daily.co/v1'))
+daily_helpers = {}
 
 
 # ----------------- API ----------------- #
 
-app = FastAPI()
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    aiohttp_session = aiohttp.ClientSession()
+    daily_helpers["rest"] = DailyRESTHelper(
+        daily_api_key=os.getenv("DAILY_API_KEY", ""),
+        daily_api_url=os.getenv("DAILY_API_URL", 'https://api.daily.co/v1'),
+        aiohttp_session=aiohttp_session
+    )
+    yield
+    await aiohttp_session.close()
+
+app = FastAPI(lifespan=lifespan)
 
 app.add_middleware(
     CORSMiddleware,
@@ -52,53 +70,52 @@ app.add_middleware(
 # ----------------- Main ----------------- #
 
 
-def spawn_fly_machine(room_url: str, token: str):
-    # Use the same image as the bot runner
-    res = requests.get(f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines", headers=FLY_HEADERS)
-    if res.status_code != 200:
-        raise Exception(f"Unable to get machine info from Fly: {res.text}")
-    image = res.json()[0]['config']['image']
+async def spawn_fly_machine(room_url: str, token: str):
+    async with aiohttp.ClientSession() as session:
+        # Use the same image as the bot runner
+        async with session.get(f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines", headers=FLY_HEADERS) as r:
+            if r.status != 200:
+                text = await r.text()
+                raise Exception(f"Unable to get machine info from Fly: {text}")
 
-    # Machine configuration
-    cmd = f"python3 bot.py -u {room_url} -t {token}"
-    cmd = cmd.split()
-    worker_props = {
-        "config": {
-            "image": image,
-            "auto_destroy": True,
-            "init": {
-                "cmd": cmd
+            data = await r.json()
+            image = data[0]['config']['image']
+
+        # Machine configuration
+        cmd = f"python3 bot.py -u {room_url} -t {token}"
+        cmd = cmd.split()
+        worker_props = {
+            "config": {
+                "image": image,
+                "auto_destroy": True,
+                "init": {
+                    "cmd": cmd
+                },
+                "restart": {
+                    "policy": "no"
+                },
+                "guest": {
+                    "cpu_kind": "shared",
+                    "cpus": 1,
+                    "memory_mb": 1024
+                }
             },
-            "restart": {
-                "policy": "no"
-            },
-            "guest": {
-                "cpu_kind": "shared",
-                "cpus": 1,
-                "memory_mb": 1024
-            }
-        },
+        }
 
-    }
+        # Spawn a new machine instance
+        async with session.post(f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines", headers=FLY_HEADERS, json=worker_props) as r:
+            if r.status != 200:
+                text = await r.text()
+                raise Exception(f"Problem starting a bot worker: {text}")
 
-    # Spawn a new machine instance
-    res = requests.post(
-        f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines",
-        headers=FLY_HEADERS,
-        json=worker_props)
+            data = await r.json()
+            # Wait for the machine to enter the started state
+            vm_id = data['id']
 
-    if res.status_code != 200:
-        raise Exception(f"Problem starting a bot worker: {res.text}")
-
-    # Wait for the machine to enter the started state
-    vm_id = res.json()['id']
-
-    res = requests.get(
-        f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines/{vm_id}/wait?state=started",
-        headers=FLY_HEADERS)
-
-    if res.status_code != 200:
-        raise Exception(f"Bot was unable to enter started state: {res.text}")
+        async with session.get(f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines/{vm_id}/wait?state=started", headers=FLY_HEADERS) as r:
+            if r.status != 200:
+                text = await r.text()
+                raise Exception(f"Bot was unable to enter started state: {text}")
 
     print(f"Machine joined room: {room_url}")
 
@@ -121,7 +138,7 @@ async def start_bot(request: Request) -> JSONResponse:
             properties=DailyRoomProperties()
         )
         try:
-            room: DailyRoomObject = daily_rest_helper.create_room(params=params)
+            room: DailyRoomObject = await daily_helpers["rest"].create_room(params=params)
         except Exception as e:
             raise HTTPException(
                 status_code=500,
@@ -129,13 +146,13 @@ async def start_bot(request: Request) -> JSONResponse:
     else:
         # Check passed room URL exists, we should assume that it already has a sip set up
         try:
-            room: DailyRoomObject = daily_rest_helper.get_room_from_url(room_url)
+            room: DailyRoomObject = await daily_helpers["rest"].get_room_from_url(room_url)
         except Exception:
             raise HTTPException(
                 status_code=500, detail=f"Room not found: {room_url}")
 
     # Give the agent a token to join the session
-    token = daily_rest_helper.get_token(room.url, MAX_SESSION_TIME)
+    token = await daily_helpers["rest"].get_token(room.url, MAX_SESSION_TIME)
 
     if not room or not token:
         raise HTTPException(
@@ -156,13 +173,13 @@ async def start_bot(request: Request) -> JSONResponse:
                 status_code=500, detail=f"Failed to start subprocess: {e}")
     else:
         try:
-            spawn_fly_machine(room.url, token)
+            await spawn_fly_machine(room.url, token)
         except Exception as e:
             raise HTTPException(
                 status_code=500, detail=f"Failed to spawn VM: {e}")
 
     # Grab a token for the user to join with
-    user_token = daily_rest_helper.get_token(room.url, MAX_SESSION_TIME)
+    user_token = await daily_helpers["rest"].get_token(room.url, MAX_SESSION_TIME)
 
     return JSONResponse({
         "room_url": room.url,
@@ -194,6 +211,5 @@ if __name__ == "__main__":
             port=config.port,
             reload=config.reload
         )
-
     except KeyboardInterrupt:
         print("Pipecat runner shutting down...")

examples/deployment/flyio-example/install_deps.py

@@ -1,4 +0,0 @@
-import torch
-
-# Download (cache) the Silero VAD model
-torch.hub.load(repo_or_dir='snakers4/silero-vad', model='silero_vad', force_reload=True)

examples/deployment/flyio-example/requirements.txt

@@ -1,6 +1,5 @@
 pipecat-ai[daily,openai,silero]
 fastapi
 uvicorn
-requests
 python-dotenv
-loguru
+loguru

examples/dialin-chatbot/bot_runner.py

@@ -6,15 +6,27 @@ provisioning a room and starting a Pipecat bot in response.
 
 Refer to README for more information.
 """
+
+
+import aiohttp
 import os
 import argparse
 import subprocess
-from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper, DailyRoomObject, DailyRoomProperties, DailyRoomSipParams, DailyRoomParams
+
+from contextlib import asynccontextmanager
+
 from fastapi import FastAPI, Request, HTTPException
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.responses import JSONResponse, PlainTextResponse
 from twilio.twiml.voice_response import VoiceResponse
 
+from pipecat.transports.services.helpers.daily_rest import (
+    DailyRESTHelper,
+    DailyRoomObject,
+    DailyRoomProperties,
+    DailyRoomSipParams,
+    DailyRoomParams)
+
 from dotenv import load_dotenv
 load_dotenv(override=True)
 
@@ -25,14 +37,23 @@ MAX_SESSION_TIME = 5 * 60  # 5 minutes
 REQUIRED_ENV_VARS = ['OPENAI_API_KEY', 'DAILY_API_KEY',
                      'ELEVENLABS_API_KEY', 'ELEVENLABS_VOICE_ID']
 
-daily_rest_helper = DailyRESTHelper(
-    os.getenv("DAILY_API_KEY", ""),
-    os.getenv("DAILY_API_URL", 'https://api.daily.co/v1'))
-
+daily_helpers = {}
 
 # ----------------- API ----------------- #
 
-app = FastAPI()
+
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    aiohttp_session = aiohttp.ClientSession()
+    daily_helpers["rest"] = DailyRESTHelper(
+        daily_api_key=os.getenv("DAILY_API_KEY", ""),
+        daily_api_url=os.getenv("DAILY_API_URL", 'https://api.daily.co/v1'),
+        aiohttp_session=aiohttp_session
+    )
+    yield
+    await aiohttp_session.close()
+
+app = FastAPI(lifespan=lifespan)
 
 app.add_middleware(
     CORSMiddleware,
@@ -53,7 +74,7 @@ action using the Twilio Client library.
 """
 
 
-def _create_daily_room(room_url, callId, callDomain=None, vendor="daily"):
+async def _create_daily_room(room_url, callId, callDomain=None, vendor="daily"):
     if not room_url:
         params = DailyRoomParams(
             properties=DailyRoomProperties(
@@ -68,14 +89,13 @@ def _create_daily_room(room_url, callId, callDomain=None, vendor="daily"):
         )
 
         print(f"Creating new room...")
-        room: DailyRoomObject = daily_rest_helper.create_room(params=params)
+        room: DailyRoomObject = await daily_helpers["rest"].create_room(params=params)
 
     else:
         # Check passed room URL exist (we assume that it already has a sip set up!)
         try:
             print(f"Joining existing room: {room_url}")
-            room: DailyRoomObject = daily_rest_helper.get_room_from_url(
-                room_url)
+            room: DailyRoomObject = await daily_helpers["rest"].get_room_from_url(room_url)
         except Exception:
             raise HTTPException(
                 status_code=500, detail=f"Room not found: {room_url}")
@@ -83,7 +103,7 @@ def _create_daily_room(room_url, callId, callDomain=None, vendor="daily"):
     print(f"Daily room: {room.url} {room.config.sip_endpoint}")
 
     # Give the agent a token to join the session
-    token = daily_rest_helper.get_token(room.url, MAX_SESSION_TIME)
+    token = await daily_helpers["rest"].get_token(room.url, MAX_SESSION_TIME)
 
     if not room or not token:
         raise HTTPException(
@@ -92,11 +112,11 @@ def _create_daily_room(room_url, callId, callDomain=None, vendor="daily"):
     # Spawn a new agent, and join the user session
     # Note: this is mostly for demonstration purposes (refer to 'deployment' in docs)
     if vendor == "daily":
-        bot_proc = f"python3 -m bot_daily -u {room.url} -t {token} -i {
-            callId} -d {callDomain}"
+        bot_proc = f"python3 - m bot_daily - u {room.url} - t {token} - i {
+            callId} - d {callDomain}"
     else:
-        bot_proc = f"python3 -m bot_twilio -u {room.url} -t {
-            token} -i {callId} -s {room.config.sip_endpoint}"
+        bot_proc = f"python3 - m bot_twilio - u {room.url} - t {
+            token} - i {callId} - s {room.config.sip_endpoint}"
 
     try:
         subprocess.Popen(
@@ -140,8 +160,7 @@ async def twilio_start_bot(request: Request):
 
     # create room and tell the bot to join the created room
     # note: Twilio does not require a callDomain
-    room: DailyRoomObject = _create_daily_room(
-        room_url, callId, None, "twilio")
+    room: DailyRoomObject = await _create_daily_room(room_url, callId, None, "twilio")
 
     print(f"Put Twilio on hold...")
     # We have the room and the SIP URI,
@@ -178,8 +197,7 @@ async def daily_start_bot(request: Request) -> JSONResponse:
             detail="Missing properties 'callId' or 'callDomain'")
 
     print(f"CallId: {callId}, CallDomain: {callDomain}")
-    room: DailyRoomObject = _create_daily_room(
-        room_url, callId, callDomain, "daily")
+    room: DailyRoomObject = await _create_daily_room(room_url, callId, callDomain, "daily")
 
     # Grab a token for the user to join with
     return JSONResponse({

examples/dialin-chatbot/requirements.txt

@@ -1,7 +1,5 @@
 pipecat-ai[daily,openai,silero]
 fastapi
 uvicorn
-requests
 python-dotenv
-loguru
-twilio
+twilio

examples/foundational/01-say-one-thing.py

@@ -27,8 +27,10 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
-async def main(room_url):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, _) = await configure(session)
+
         transport = DailyTransport(
             room_url, None, "Say One Thing", DailyParams(audio_out_enabled=True))
 
@@ -52,5 +54,4 @@ async def main(room_url):
         await runner.run(task)
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url))
+    asyncio.run(main())

examples/foundational/02-llm-say-one-thing.py

@@ -28,8 +28,10 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
-async def main(room_url):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, _) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             None,
@@ -64,5 +66,4 @@ async def main(room_url):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url))
+    asyncio.run(main())

examples/foundational/03-still-frame.py

@@ -27,8 +27,10 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
-async def main(room_url):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, _) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             None,
@@ -64,5 +66,4 @@ async def main(room_url):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url))
+    asyncio.run(main())

examples/foundational/04-utterance-and-speech.py

@@ -30,8 +30,10 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
-async def main(room_url: str):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, _) = await configure(session)
+
         transport = DailyTransport(room_url, None, "Static And Dynamic Speech")
 
         meeting = TransportServiceOutput(transport, mic_enabled=True)
@@ -82,5 +84,4 @@ async def main(room_url: str):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url))
+    asyncio.run(main())

examples/foundational/05-sync-speech-and-image.py

@@ -73,8 +73,10 @@ class MonthPrepender(FrameProcessor):
             await self.push_frame(frame, direction)
 
 
-async def main(room_url):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, _) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             None,
@@ -162,5 +164,4 @@ async def main(room_url):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url))
+    asyncio.run(main())

examples/foundational/06-listen-and-respond.py

@@ -9,14 +9,15 @@ import aiohttp
 import os
 import sys
 
-from pipecat.frames.frames import LLMMessagesFrame
+from pipecat.frames.frames import Frame, LLMMessagesFrame, MetricsFrame
 from pipecat.pipeline.pipeline import Pipeline
 from pipecat.pipeline.runner import PipelineRunner
-from pipecat.pipeline.task import PipelineTask
+from pipecat.pipeline.task import PipelineParams, PipelineTask
 from pipecat.processors.aggregators.llm_response import (
     LLMAssistantResponseAggregator,
     LLMUserResponseAggregator,
 )
+from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
 from pipecat.processors.logger import FrameLogger
 from pipecat.services.elevenlabs import ElevenLabsTTSService
 from pipecat.services.openai import OpenAILLMService
@@ -34,8 +35,18 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
-async def main(room_url: str, token):
+class MetricsLogger(FrameProcessor):
+    async def process_frame(self, frame: Frame, direction: FrameDirection):
+        if isinstance(frame, MetricsFrame):
+            print(
+                f"!!! MetricsFrame: {frame}, ttfb: {frame.ttfb}, processing: {frame.processing}, tokens: {frame.tokens}, characters: {frame.characters}")
+        await self.push_frame(frame, direction)
+
+
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -56,11 +67,10 @@ async def main(room_url: str, token):
 
         llm = OpenAILLMService(
             api_key=os.getenv("OPENAI_API_KEY"),
-            model="gpt-4o")
+            model="gpt-4o"
+        )
 
-        fl = FrameLogger("!!! after LLM", "red")
-        fltts = FrameLogger("@@@ out of tts", "green")
-        flend = FrameLogger("### out of the end", "magenta")
+        ml = MetricsLogger()
 
         messages = [
             {
@@ -75,15 +85,18 @@ async def main(room_url: str, token):
             transport.input(),
             tma_in,
             llm,
-            fl,
             tts,
-            fltts,
+            ml,
             transport.output(),
             tma_out,
-            flend
         ])
 
         task = PipelineTask(pipeline)
+        task = PipelineTask(pipeline, PipelineParams(
+            allow_interruptions=True,
+            enable_metrics=True,
+            report_only_initial_ttfb=False,
+        ))
 
         @transport.event_handler("on_first_participant_joined")
         async def on_first_participant_joined(transport, participant):
@@ -99,5 +112,4 @@ async def main(room_url: str, token):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/06a-image-sync.py

@@ -51,7 +51,7 @@ class ImageSyncAggregator(FrameProcessor):
     async def process_frame(self, frame: Frame, direction: FrameDirection):
         await super().process_frame(frame, direction)
 
-        if not isinstance(frame, SystemFrame):
+        if not isinstance(frame, SystemFrame) and direction == FrameDirection.DOWNSTREAM:
             await self.push_frame(ImageRawFrame(image=self._speaking_image_bytes, size=(1024, 1024), format=self._speaking_image_format))
             await self.push_frame(frame)
             await self.push_frame(ImageRawFrame(image=self._waiting_image_bytes, size=(1024, 1024), format=self._waiting_image_format))
@@ -59,8 +59,10 @@ class ImageSyncAggregator(FrameProcessor):
             await self.push_frame(frame)
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -125,5 +127,4 @@ async def main(room_url: str, token):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/07-interruptible.py

@@ -31,8 +31,10 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -77,6 +79,7 @@ async def main(room_url: str, token):
         task = PipelineTask(pipeline, PipelineParams(
             allow_interruptions=True,
             enable_metrics=True,
+            enable_usage_metrics=True,
             report_only_initial_ttfb=True,
         ))
 
@@ -94,5 +97,4 @@ async def main(room_url: str, token):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/07a-interruptible-anthropic.py

@@ -31,8 +31,10 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -91,5 +93,4 @@ async def main(room_url: str, token):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/07b-interruptible-langchain.py

@@ -47,8 +47,10 @@ def get_session_history(session_id: str) -> BaseChatMessageHistory:
     return message_store[session_id]
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -121,5 +123,4 @@ async def main(room_url: str, token):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/07c-interruptible-deepgram.py

@@ -31,8 +31,10 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -93,5 +95,4 @@ async def main(room_url: str, token):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/07d-interruptible-cartesia.py

@@ -4,6 +4,7 @@
 # SPDX-License-Identifier: BSD 2-Clause License
 #
 
+import aiohttp
 import asyncio
 import os
 import sys
@@ -31,64 +32,66 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
-async def main(room_url: str, token):
-    transport = DailyTransport(
-        room_url,
-        token,
-        "Respond bot",
-        DailyParams(
-            audio_out_sample_rate=44100,
-            audio_out_enabled=True,
-            transcription_enabled=True,
-            vad_enabled=True,
-            vad_analyzer=SileroVADAnalyzer()
+async def main():
+    async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
+        transport = DailyTransport(
+            room_url,
+            token,
+            "Respond bot",
+            DailyParams(
+                audio_out_sample_rate=44100,
+                audio_out_enabled=True,
+                transcription_enabled=True,
+                vad_enabled=True,
+                vad_analyzer=SileroVADAnalyzer()
+            )
         )
-    )
 
-    tts = CartesiaTTSService(
-        api_key=os.getenv("CARTESIA_API_KEY"),
-        voice_id="a0e99841-438c-4a64-b679-ae501e7d6091",  # Barbershop Man
-        sample_rate=44100,
-    )
+        tts = CartesiaTTSService(
+            api_key=os.getenv("CARTESIA_API_KEY"),
+            voice_id="a0e99841-438c-4a64-b679-ae501e7d6091",  # Barbershop Man
+            sample_rate=44100,
+        )
 
-    llm = OpenAILLMService(
-        api_key=os.getenv("OPENAI_API_KEY"),
-        model="gpt-4o")
+        llm = OpenAILLMService(
+            api_key=os.getenv("OPENAI_API_KEY"),
+            model="gpt-4o")
 
-    messages = [
-        {
-            "role": "system",
-            "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
-        },
-    ]
+        messages = [
+            {
+                "role": "system",
+                "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
+            },
+        ]
 
-    tma_in = LLMUserResponseAggregator(messages)
-    tma_out = LLMAssistantResponseAggregator(messages)
+        tma_in = LLMUserResponseAggregator(messages)
+        tma_out = LLMAssistantResponseAggregator(messages)
 
-    pipeline = Pipeline([
-        transport.input(),   # Transport user input
-        tma_in,              # User responses
-        llm,                 # LLM
-        tts,                 # TTS
-        tma_out,             # Goes before the transport because cartesia has word-level timestamps!
-        transport.output(),  # Transport bot output
-    ])
+        pipeline = Pipeline([
+            transport.input(),   # Transport user input
+            tma_in,              # User responses
+            llm,                 # LLM
+            tts,                 # TTS
+            tma_out,             # Goes before the transport because cartesia has word-level timestamps!
+            transport.output(),  # Transport bot output
+        ])
 
-    task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True, enable_metrics=True))
+        task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True, enable_metrics=True))
 
-    @transport.event_handler("on_first_participant_joined")
-    async def on_first_participant_joined(transport, participant):
-        transport.capture_participant_transcription(participant["id"])
-        # Kick off the conversation.
-        messages.append(
-            {"role": "system", "content": "Please introduce yourself to the user."})
-        await task.queue_frames([LLMMessagesFrame(messages)])
+        @transport.event_handler("on_first_participant_joined")
+        async def on_first_participant_joined(transport, participant):
+            transport.capture_participant_transcription(participant["id"])
+            # Kick off the conversation.
+            messages.append(
+                {"role": "system", "content": "Please introduce yourself to the user."})
+            await task.queue_frames([LLMMessagesFrame(messages)])
 
-    runner = PipelineRunner()
+        runner = PipelineRunner()
 
-    await runner.run(task)
+        await runner.run(task)
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/07e-interruptible-playht.py

@@ -4,6 +4,7 @@
 # SPDX-License-Identifier: BSD 2-Clause License
 #
 
+import aiohttp
 import asyncio
 import os
 import sys
@@ -30,64 +31,66 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
-async def main(room_url: str, token):
-    transport = DailyTransport(
-        room_url,
-        token,
-        "Respond bot",
-        DailyParams(
-            audio_out_enabled=True,
-            audio_out_sample_rate=16000,
-            transcription_enabled=True,
-            vad_enabled=True,
-            vad_analyzer=SileroVADAnalyzer()
+async def main():
+    async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
+        transport = DailyTransport(
+            room_url,
+            token,
+            "Respond bot",
+            DailyParams(
+                audio_out_enabled=True,
+                audio_out_sample_rate=16000,
+                transcription_enabled=True,
+                vad_enabled=True,
+                vad_analyzer=SileroVADAnalyzer()
+            )
         )
-    )
 
-    tts = PlayHTTTSService(
-        user_id=os.getenv("PLAYHT_USER_ID"),
-        api_key=os.getenv("PLAYHT_API_KEY"),
-        voice_url="s3://voice-cloning-zero-shot/801a663f-efd0-4254-98d0-5c175514c3e8/jennifer/manifest.json",
-    )
+        tts = PlayHTTTSService(
+            user_id=os.getenv("PLAYHT_USER_ID"),
+            api_key=os.getenv("PLAYHT_API_KEY"),
+            voice_url="s3://voice-cloning-zero-shot/801a663f-efd0-4254-98d0-5c175514c3e8/jennifer/manifest.json",
+        )
 
-    llm = OpenAILLMService(
-        api_key=os.getenv("OPENAI_API_KEY"),
-        model="gpt-4o")
+        llm = OpenAILLMService(
+            api_key=os.getenv("OPENAI_API_KEY"),
+            model="gpt-4o")
 
-    messages = [
-        {
-            "role": "system",
-            "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
-        },
-    ]
+        messages = [
+            {
+                "role": "system",
+                "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
+            },
+        ]
 
-    tma_in = LLMUserResponseAggregator(messages)
-    tma_out = LLMAssistantResponseAggregator(messages)
+        tma_in = LLMUserResponseAggregator(messages)
+        tma_out = LLMAssistantResponseAggregator(messages)
 
-    pipeline = Pipeline([
-        transport.input(),   # Transport user input
-        tma_in,              # User responses
-        llm,                 # LLM
-        tts,                 # TTS
-        transport.output(),  # Transport bot output
-        tma_out              # Assistant spoken responses
-    ])
+        pipeline = Pipeline([
+            transport.input(),   # Transport user input
+            tma_in,              # User responses
+            llm,                 # LLM
+            tts,                 # TTS
+            transport.output(),  # Transport bot output
+            tma_out              # Assistant spoken responses
+        ])
 
-    task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
+        task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
 
-    @transport.event_handler("on_first_participant_joined")
-    async def on_first_participant_joined(transport, participant):
-        transport.capture_participant_transcription(participant["id"])
-        # Kick off the conversation.
-        messages.append(
-            {"role": "system", "content": "Please introduce yourself to the user."})
-        await task.queue_frames([LLMMessagesFrame(messages)])
+        @transport.event_handler("on_first_participant_joined")
+        async def on_first_participant_joined(transport, participant):
+            transport.capture_participant_transcription(participant["id"])
+            # Kick off the conversation.
+            messages.append(
+                {"role": "system", "content": "Please introduce yourself to the user."})
+            await task.queue_frames([LLMMessagesFrame(messages)])
 
-    runner = PipelineRunner()
+        runner = PipelineRunner()
 
-    await runner.run(task)
+        await runner.run(task)
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/07f-interruptible-azure.py

@@ -4,6 +4,7 @@
 # SPDX-License-Identifier: BSD 2-Clause License
 #
 
+import aiohttp
 import asyncio
 import os
 import sys
@@ -30,71 +31,73 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
-async def main(room_url: str, token):
-    transport = DailyTransport(
-        room_url,
-        token,
-        "Respond bot",
-        DailyParams(
-            audio_out_enabled=True,
-            audio_out_sample_rate=16000,
-            vad_enabled=True,
-            vad_analyzer=SileroVADAnalyzer(),
-            vad_audio_passthrough=True,
+async def main():
+    async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
+        transport = DailyTransport(
+            room_url,
+            token,
+            "Respond bot",
+            DailyParams(
+                audio_out_enabled=True,
+                audio_out_sample_rate=16000,
+                vad_enabled=True,
+                vad_analyzer=SileroVADAnalyzer(),
+                vad_audio_passthrough=True,
+            )
         )
-    )
 
-    stt = AzureSTTService(
-        api_key=os.getenv("AZURE_SPEECH_API_KEY"),
-        region=os.getenv("AZURE_SPEECH_REGION"),
-    )
+        stt = AzureSTTService(
+            api_key=os.getenv("AZURE_SPEECH_API_KEY"),
+            region=os.getenv("AZURE_SPEECH_REGION"),
+        )
 
-    tts = AzureTTSService(
-        api_key=os.getenv("AZURE_SPEECH_API_KEY"),
-        region=os.getenv("AZURE_SPEECH_REGION"),
-    )
+        tts = AzureTTSService(
+            api_key=os.getenv("AZURE_SPEECH_API_KEY"),
+            region=os.getenv("AZURE_SPEECH_REGION"),
+        )
 
-    llm = AzureLLMService(
-        api_key=os.getenv("AZURE_CHATGPT_API_KEY"),
-        endpoint=os.getenv("AZURE_CHATGPT_ENDPOINT"),
-        model=os.getenv("AZURE_CHATGPT_MODEL"),
-    )
+        llm = AzureLLMService(
+            api_key=os.getenv("AZURE_CHATGPT_API_KEY"),
+            endpoint=os.getenv("AZURE_CHATGPT_ENDPOINT"),
+            model=os.getenv("AZURE_CHATGPT_MODEL"),
+        )
 
-    messages = [
-        {
-            "role": "system",
-            "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
-        },
-    ]
+        messages = [
+            {
+                "role": "system",
+                "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
+            },
+        ]
 
-    tma_in = LLMUserResponseAggregator(messages)
-    tma_out = LLMAssistantResponseAggregator(messages)
+        tma_in = LLMUserResponseAggregator(messages)
+        tma_out = LLMAssistantResponseAggregator(messages)
 
-    pipeline = Pipeline([
-        transport.input(),   # Transport user input
-        stt,                 # STT
-        tma_in,              # User responses
-        llm,                 # LLM
-        tts,                 # TTS
-        transport.output(),  # Transport bot output
-        tma_out              # Assistant spoken responses
-    ])
+        pipeline = Pipeline([
+            transport.input(),   # Transport user input
+            stt,                 # STT
+            tma_in,              # User responses
+            llm,                 # LLM
+            tts,                 # TTS
+            transport.output(),  # Transport bot output
+            tma_out              # Assistant spoken responses
+        ])
 
-    task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
+        task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
 
-    @transport.event_handler("on_first_participant_joined")
-    async def on_first_participant_joined(transport, participant):
-        transport.capture_participant_transcription(participant["id"])
-        # Kick off the conversation.
-        messages.append(
-            {"role": "system", "content": "Please introduce yourself to the user."})
-        await task.queue_frames([LLMMessagesFrame(messages)])
+        @transport.event_handler("on_first_participant_joined")
+        async def on_first_participant_joined(transport, participant):
+            transport.capture_participant_transcription(participant["id"])
+            # Kick off the conversation.
+            messages.append(
+                {"role": "system", "content": "Please introduce yourself to the user."})
+            await task.queue_frames([LLMMessagesFrame(messages)])
 
-    runner = PipelineRunner()
+        runner = PipelineRunner()
 
-    await runner.run(task)
+        await runner.run(task)
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/07g-interruptible-openai-tts.py

@@ -4,6 +4,7 @@
 # SPDX-License-Identifier: BSD 2-Clause License
 #
 
+import aiohttp
 import asyncio
 import os
 import sys
@@ -30,63 +31,65 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
-async def main(room_url: str, token):
-    transport = DailyTransport(
-        room_url,
-        token,
-        "Respond bot",
-        DailyParams(
-            audio_out_enabled=True,
-            audio_out_sample_rate=24000,
-            transcription_enabled=True,
-            vad_enabled=True,
-            vad_analyzer=SileroVADAnalyzer()
+async def main():
+    async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
+        transport = DailyTransport(
+            room_url,
+            token,
+            "Respond bot",
+            DailyParams(
+                audio_out_enabled=True,
+                audio_out_sample_rate=24000,
+                transcription_enabled=True,
+                vad_enabled=True,
+                vad_analyzer=SileroVADAnalyzer()
+            )
         )
-    )
 
-    tts = OpenAITTSService(
-        api_key=os.getenv("OPENAI_API_KEY"),
-        voice="alloy"
-    )
+        tts = OpenAITTSService(
+            api_key=os.getenv("OPENAI_API_KEY"),
+            voice="alloy"
+        )
 
-    llm = OpenAILLMService(
-        api_key=os.getenv("OPENAI_API_KEY"),
-        model="gpt-4o")
+        llm = OpenAILLMService(
+            api_key=os.getenv("OPENAI_API_KEY"),
+            model="gpt-4o")
 
-    messages = [
-        {
-            "role": "system",
-            "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
-        },
-    ]
+        messages = [
+            {
+                "role": "system",
+                "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
+            },
+        ]
 
-    tma_in = LLMUserResponseAggregator(messages)
-    tma_out = LLMAssistantResponseAggregator(messages)
+        tma_in = LLMUserResponseAggregator(messages)
+        tma_out = LLMAssistantResponseAggregator(messages)
 
-    pipeline = Pipeline([
-        transport.input(),   # Transport user input
-        tma_in,              # User responses
-        llm,                 # LLM
-        tts,                 # TTS
-        transport.output(),  # Transport bot output
-        tma_out              # Assistant spoken responses
-    ])
+        pipeline = Pipeline([
+            transport.input(),   # Transport user input
+            tma_in,              # User responses
+            llm,                 # LLM
+            tts,                 # TTS
+            transport.output(),  # Transport bot output
+            tma_out              # Assistant spoken responses
+        ])
 
-    task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
+        task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
 
-    @transport.event_handler("on_first_participant_joined")
-    async def on_first_participant_joined(transport, participant):
-        transport.capture_participant_transcription(participant["id"])
-        # Kick off the conversation.
-        messages.append(
-            {"role": "system", "content": "Please introduce yourself to the user."})
-        await task.queue_frames([LLMMessagesFrame(messages)])
+        @transport.event_handler("on_first_participant_joined")
+        async def on_first_participant_joined(transport, participant):
+            transport.capture_participant_transcription(participant["id"])
+            # Kick off the conversation.
+            messages.append(
+                {"role": "system", "content": "Please introduce yourself to the user."})
+            await task.queue_frames([LLMMessagesFrame(messages)])
 
-    runner = PipelineRunner()
+        runner = PipelineRunner()
 
-    await runner.run(task)
+        await runner.run(task)
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/07h-interruptible-openpipe.py

@@ -34,8 +34,10 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -98,5 +100,4 @@ async def main(room_url: str, token):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/07i-interruptible-xtts.py

@@ -32,8 +32,10 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -92,5 +94,4 @@ async def main(room_url: str, token):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/07j-interruptible-gladia.py

@@ -33,8 +33,10 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -97,5 +99,4 @@ async def main(room_url: str, token):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/08-bots-arguing.py

@@ -22,8 +22,10 @@ logger = logging.getLogger("pipecat")
 logger.setLevel(logging.DEBUG)
 
 
-async def main(room_url: str):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, _) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             None,
@@ -144,5 +146,4 @@ async def main(room_url: str):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url))
+    asyncio.run(main())

examples/foundational/09-mirror.py

@@ -4,6 +4,7 @@
 # SPDX-License-Identifier: BSD 2-Clause License
 #
 
+import aiohttp
 import asyncio
 import sys
 
@@ -23,32 +24,34 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
-async def main(room_url, token):
-    transport = DailyTransport(
-        room_url, token, "Test",
-        DailyParams(
-            audio_in_enabled=True,
-            audio_out_enabled=True,
-            camera_out_enabled=True,
-            camera_out_is_live=True,
-            camera_out_width=1280,
-            camera_out_height=720
+async def main():
+    async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
+        transport = DailyTransport(
+            room_url, token, "Test",
+            DailyParams(
+                audio_in_enabled=True,
+                audio_out_enabled=True,
+                camera_out_enabled=True,
+                camera_out_is_live=True,
+                camera_out_width=1280,
+                camera_out_height=720
+            )
         )
-    )
 
-    @transport.event_handler("on_first_participant_joined")
-    async def on_first_participant_joined(transport, participant):
-        transport.capture_participant_video(participant["id"])
+        @transport.event_handler("on_first_participant_joined")
+        async def on_first_participant_joined(transport, participant):
+            transport.capture_participant_video(participant["id"])
 
-    pipeline = Pipeline([transport.input(), transport.output()])
+        pipeline = Pipeline([transport.input(), transport.output()])
 
-    runner = PipelineRunner()
+        runner = PipelineRunner()
 
-    task = PipelineTask(pipeline)
+        task = PipelineTask(pipeline)
 
-    await runner.run(task)
+        await runner.run(task)
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/09a-local-mirror.py

@@ -4,6 +4,7 @@
 # SPDX-License-Identifier: BSD 2-Clause License
 #
 
+import aiohttp
 import asyncio
 import sys
 
@@ -27,40 +28,44 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
-async def main(room_url, token):
-    tk_root = tk.Tk()
-    tk_root.title("Local Mirror")
+async def main():
+    async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
 
-    daily_transport = DailyTransport(room_url, token, "Test", DailyParams(audio_in_enabled=True))
+        tk_root = tk.Tk()
+        tk_root.title("Local Mirror")
 
-    tk_transport = TkLocalTransport(
-        tk_root,
-        TransportParams(
-            audio_out_enabled=True,
-            camera_out_enabled=True,
-            camera_out_is_live=True,
-            camera_out_width=1280,
-            camera_out_height=720))
+        daily_transport = DailyTransport(
+            room_url, token, "Test", DailyParams(
+                audio_in_enabled=True))
 
-    @daily_transport.event_handler("on_first_participant_joined")
-    async def on_first_participant_joined(transport, participant):
-        transport.capture_participant_video(participant["id"])
+        tk_transport = TkLocalTransport(
+            tk_root,
+            TransportParams(
+                audio_out_enabled=True,
+                camera_out_enabled=True,
+                camera_out_is_live=True,
+                camera_out_width=1280,
+                camera_out_height=720))
 
-    pipeline = Pipeline([daily_transport.input(), tk_transport.output()])
+        @daily_transport.event_handler("on_first_participant_joined")
+        async def on_first_participant_joined(transport, participant):
+            transport.capture_participant_video(participant["id"])
 
-    task = PipelineTask(pipeline)
+        pipeline = Pipeline([daily_transport.input(), tk_transport.output()])
 
-    async def run_tk():
-        while not task.has_finished():
-            tk_root.update()
-            tk_root.update_idletasks()
-            await asyncio.sleep(0.1)
+        task = PipelineTask(pipeline)
 
-    runner = PipelineRunner()
+        async def run_tk():
+            while not task.has_finished():
+                tk_root.update()
+                tk_root.update_idletasks()
+                await asyncio.sleep(0.1)
 
-    await asyncio.gather(runner.run(task), run_tk())
+        runner = PipelineRunner()
+
+        await asyncio.gather(runner.run(task), run_tk())
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/10-wake-phrase.py

@@ -31,9 +31,10 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
-async def main(room_url: str, token):
-
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -90,5 +91,4 @@ async def main(room_url: str, token):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/11-sound-effects.py

@@ -83,8 +83,10 @@ class InboundSoundEffectWrapper(FrameProcessor):
             await self.push_frame(frame, direction)
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -148,5 +150,4 @@ async def main(room_url: str, token):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/12-describe-video.py

@@ -49,8 +49,10 @@ class UserImageRequester(FrameProcessor):
         await self.push_frame(frame, direction)
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -108,5 +110,4 @@ async def main(room_url: str, token):
         await runner.run(task)
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/12a-describe-video-gemini-flash.py

@@ -49,8 +49,10 @@ class UserImageRequester(FrameProcessor):
         await self.push_frame(frame, direction)
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -104,5 +106,4 @@ async def main(room_url: str, token):
         await runner.run(task)
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/12b-describe-video-gpt-4o.py

@@ -49,8 +49,10 @@ class UserImageRequester(FrameProcessor):
         await self.push_frame(frame, direction)
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -104,5 +106,4 @@ async def main(room_url: str, token):
         await runner.run(task)
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/12c-describe-video-anthropic.py

@@ -49,8 +49,10 @@ class UserImageRequester(FrameProcessor):
         await self.push_frame(frame, direction)
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -104,5 +106,4 @@ async def main(room_url: str, token):
         await runner.run(task)
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/13-whisper-transcription.py

@@ -4,6 +4,7 @@
 # SPDX-License-Identifier: BSD 2-Clause License
 #
 
+import aiohttp
 import asyncio
 import sys
 
@@ -35,23 +36,25 @@ class TranscriptionLogger(FrameProcessor):
             print(f"Transcription: {frame.text}")
 
 
-async def main(room_url: str):
-    transport = DailyTransport(room_url, None, "Transcription bot",
-                               DailyParams(audio_in_enabled=True))
+async def main():
+    async with aiohttp.ClientSession() as session:
+        (room_url, _) = await configure(session)
 
-    stt = WhisperSTTService()
+        transport = DailyTransport(room_url, None, "Transcription bot",
+                                   DailyParams(audio_in_enabled=True))
 
-    tl = TranscriptionLogger()
+        stt = WhisperSTTService()
 
-    pipeline = Pipeline([transport.input(), stt, tl])
+        tl = TranscriptionLogger()
 
-    task = PipelineTask(pipeline)
+        pipeline = Pipeline([transport.input(), stt, tl])
 
-    runner = PipelineRunner()
+        task = PipelineTask(pipeline)
 
-    await runner.run(task)
+        runner = PipelineRunner()
+
+        await runner.run(task)
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url))
+    asyncio.run(main())

examples/foundational/13b-deepgram-transcription.py

@@ -4,6 +4,7 @@
 # SPDX-License-Identifier: BSD 2-Clause License
 #
 
+import aiohttp
 import asyncio
 import os
 import sys
@@ -36,23 +37,25 @@ class TranscriptionLogger(FrameProcessor):
             print(f"Transcription: {frame.text}")
 
 
-async def main(room_url: str):
-    transport = DailyTransport(room_url, None, "Transcription bot",
-                               DailyParams(audio_in_enabled=True))
+async def main():
+    async with aiohttp.ClientSession() as session:
+        (room_url, _) = await configure(session)
 
-    stt = DeepgramSTTService(os.getenv("DEEPGRAM_API_KEY"))
+        transport = DailyTransport(room_url, None, "Transcription bot",
+                                   DailyParams(audio_in_enabled=True))
 
-    tl = TranscriptionLogger()
+        stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
 
-    pipeline = Pipeline([transport.input(), stt, tl])
+        tl = TranscriptionLogger()
 
-    task = PipelineTask(pipeline)
+        pipeline = Pipeline([transport.input(), stt, tl])
 
-    runner = PipelineRunner()
+        task = PipelineTask(pipeline)
 
-    await runner.run(task)
+        runner = PipelineRunner()
+
+        await runner.run(task)
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url))
+    asyncio.run(main())

examples/foundational/14-function-calling.py

@@ -44,8 +44,10 @@ async def fetch_weather_from_api(llm, args):
     return {"conditions": "nice", "temperature": "75"}
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -124,7 +126,7 @@ async def main(room_url: str, token):
 
         task = PipelineTask(pipeline)
 
-        @ transport.event_handler("on_first_participant_joined")
+        @transport.event_handler("on_first_participant_joined")
         async def on_first_participant_joined(transport, participant):
             transport.capture_participant_transcription(participant["id"])
             # Kick off the conversation.
@@ -134,7 +136,5 @@ async def main(room_url: str, token):
 
         await runner.run(task)
 
-
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/15-switch-voices.py

@@ -4,8 +4,8 @@
 # SPDX-License-Identifier: BSD 2-Clause License
 #
 
-import asyncio
 import aiohttp
+import asyncio
 import os
 import sys
 
@@ -58,8 +58,10 @@ async def barbershop_man_filter(frame) -> bool:
     return current_voice == "Barbershop Man"
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -151,5 +153,4 @@ async def main(room_url: str, token):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/15a-switch-languages.py

@@ -55,8 +55,10 @@ async def spanish_filter(frame) -> bool:
     return current_language == "Spanish"
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -149,5 +151,4 @@ async def main(room_url: str, token):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/16-gpu-container-local-bot.py

@@ -8,7 +8,6 @@ import asyncio
 import aiohttp
 import os
 import sys
-import json
 
 from pipecat.frames.frames import LLMMessagesFrame
 from pipecat.pipeline.pipeline import Pipeline
@@ -32,8 +31,10 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -126,5 +127,4 @@ async def main(room_url: str, token):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/17-detect-user-idle.py

@@ -33,8 +33,10 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -104,5 +106,4 @@ async def main(room_url: str, token):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/foundational/18-gstreamer-filesrc.py

@@ -0,0 +1,78 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import asyncio
+import aiohttp
+import argparse
+import sys
+
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.runner import PipelineRunner
+from pipecat.pipeline.task import PipelineTask
+from pipecat.processors.gstreamer.pipeline_source import GStreamerPipelineSource
+from pipecat.transports.services.daily import DailyParams, DailyTransport
+
+from runner import configure_with_args
+
+from loguru import logger
+
+from dotenv import load_dotenv
+load_dotenv(override=True)
+
+logger.remove(0)
+logger.add(sys.stderr, level="DEBUG")
+
+
+async def main():
+    async with aiohttp.ClientSession() as session:
+        parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample")
+        parser.add_argument(
+            "-i",
+            "--input",
+            type=str,
+            required=True,
+            help="Input video file")
+
+        (room_url, _, args) = await configure_with_args(session, parser)
+
+        transport = DailyTransport(
+            room_url,
+            None,
+            "GStreamer",
+            DailyParams(
+                audio_out_enabled=True,
+                audio_out_is_live=True,
+                camera_out_enabled=True,
+                camera_out_width=1280,
+                camera_out_height=720,
+                camera_out_is_live=True,
+            )
+        )
+
+        gst = GStreamerPipelineSource(
+            pipeline=f"filesrc location={args.input}",
+            out_params=GStreamerPipelineSource.OutputParams(
+                video_width=1280,
+                video_height=720,
+                audio_sample_rate=16000,
+                audio_channels=1,
+            )
+        )
+
+        pipeline = Pipeline([
+            gst,                 # GStreamer file source
+            transport.output(),  # Transport bot output
+        ])
+
+        task = PipelineTask(pipeline)
+
+        runner = PipelineRunner()
+
+        await runner.run(task)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

examples/foundational/18a-gstreamer-videotestsrc.py

@@ -0,0 +1,64 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import asyncio
+import aiohttp
+import sys
+
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.runner import PipelineRunner
+from pipecat.pipeline.task import PipelineTask
+from pipecat.processors.gstreamer.pipeline_source import GStreamerPipelineSource
+from pipecat.transports.services.daily import DailyParams, DailyTransport
+
+from runner import configure
+
+from loguru import logger
+
+from dotenv import load_dotenv
+load_dotenv(override=True)
+
+logger.remove(0)
+logger.add(sys.stderr, level="DEBUG")
+
+
+async def main():
+    async with aiohttp.ClientSession() as session:
+        (room_url, _) = await configure(session)
+
+        transport = DailyTransport(
+            room_url,
+            None,
+            "GStreamer",
+            DailyParams(
+                camera_out_enabled=True,
+                camera_out_width=1280,
+                camera_out_height=720,
+                camera_out_is_live=True,
+            )
+        )
+
+        gst = GStreamerPipelineSource(
+            pipeline="videotestsrc ! capsfilter caps=\"video/x-raw,width=1280,height=720,framerate=30/1\"",
+            out_params=GStreamerPipelineSource.OutputParams(
+                video_width=1280,
+                video_height=720,
+                clock_sync=False))
+
+        pipeline = Pipeline([
+            gst,                 # GStreamer file source
+            transport.output(),  # Transport bot output
+        ])
+
+        task = PipelineTask(pipeline)
+
+        runner = PipelineRunner()
+
+        await runner.run(task)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

examples/foundational/runner.py

@@ -1,12 +1,26 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import aiohttp
 import argparse
 import os
-import time
-import urllib
-import requests
+
+from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper
 
 
-def configure():
-    parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample")
+async def configure(aiohttp_session: aiohttp.ClientSession):
+    (url, token, _) = await configure_with_args(aiohttp_session)
+    return (url, token)
+
+
+async def configure_with_args(
+        aiohttp_session: aiohttp.ClientSession,
+        parser: argparse.ArgumentParser | None = None):
+    if not parser:
+        parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample")
     parser.add_argument(
         "-u",
         "--url",
@@ -33,26 +47,15 @@ def configure():
     if not key:
         raise Exception("No Daily API key specified. use the -k/--apikey option from the command line, or set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.")
 
+    daily_rest_helper = DailyRESTHelper(
+        daily_api_key=key,
+        daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
+        aiohttp_session=aiohttp_session)
+
     # Create a meeting token for the given room with an expiration 1 hour in
     # the future.
-    room_name: str = urllib.parse.urlparse(url).path[1:]
-    expiration: float = time.time() + 60 * 60
+    expiry_time: float = 60 * 60
 
-    res: requests.Response = requests.post(
-        f"https://api.daily.co/v1/meeting-tokens",
-        headers={
-            "Authorization": f"Bearer {key}"},
-        json={
-            "properties": {
-                "room_name": room_name,
-                "is_owner": True,
-                "exp": expiration}},
-    )
+    token = await daily_rest_helper.get_token(url, expiry_time)
 
-    if res.status_code != 200:
-        raise Exception(
-            f"Failed to create meeting token: {res.status_code} {res.text}")
-
-    token: str = res.json()["token"]
-
-    return (url, token)
+    return (url, token, args)

examples/moondream-chatbot/bot.py

@@ -1,3 +1,9 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
 import asyncio
 import aiohttp
 import os
@@ -128,8 +134,10 @@ class ImageFilterProcessor(FrameProcessor):
             await self.push_frame(frame)
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -204,5 +212,4 @@ async def main(room_url: str, token):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/moondream-chatbot/requirements.txt

@@ -1,5 +1,4 @@
 python-dotenv
-requests
 fastapi[all]
 uvicorn
 pipecat-ai[daily,moondream,openai,silero]

examples/moondream-chatbot/runner.py

@@ -1,11 +1,17 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import aiohttp
 import argparse
 import os
-import time
-import urllib
-import requests
+
+from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper
 
 
-def configure():
+async def configure(aiohttp_session: aiohttp.ClientSession):
     parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample")
     parser.add_argument(
         "-u",
@@ -33,26 +39,16 @@ def configure():
     if not key:
         raise Exception("No Daily API key specified. use the -k/--apikey option from the command line, or set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.")
 
-    # Create a meeting token for the given room with an expiration 1 hour in
-    # the future.
-    room_name: str = urllib.parse.urlparse(url).path[1:]
-    expiration: float = time.time() + 60 * 60
-
-    res: requests.Response = requests.post(
-        f"https://api.daily.co/v1/meeting-tokens",
-        headers={
-            "Authorization": f"Bearer {key}"},
-        json={
-            "properties": {
-                "room_name": room_name,
-                "is_owner": True,
-                "exp": expiration}},
+    daily_rest_helper = DailyRESTHelper(
+        daily_api_key=key,
+        daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
+        aiohttp_session=aiohttp_session
     )
 
-    if res.status_code != 200:
-        raise Exception(
-            f"Failed to create meeting token: {res.status_code} {res.text}")
+    # Create a meeting token for the given room with an expiration 1 hour in
+    # the future.
+    expiry_time: float = 60 * 60
 
-    token: str = res.json()["token"]
+    token = await daily_rest_helper.get_token(url, expiry_time)
 
     return (url, token)

examples/moondream-chatbot/server.py

@@ -1,31 +1,51 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import aiohttp
 import os
 import argparse
 import subprocess
-import atexit
+
+from contextlib import asynccontextmanager
 
 from fastapi import FastAPI, Request, HTTPException
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.responses import JSONResponse, RedirectResponse
 
-from utils.daily_helpers import create_room as _create_room, get_token
+from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper, DailyRoomParams
 
 MAX_BOTS_PER_ROOM = 1
 
 # Bot sub-process dict for status reporting and concurrency control
 bot_procs = {}
 
+daily_helpers = {}
+
 
 def cleanup():
     # Clean up function, just to be extra safe
-    for proc in bot_procs.values():
+    for entry in bot_procs.values():
+        proc = entry[0]
         proc.terminate()
         proc.wait()
 
 
-atexit.register(cleanup)
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    aiohttp_session = aiohttp.ClientSession()
+    daily_helpers["rest"] = DailyRESTHelper(
+        daily_api_key=os.getenv("DAILY_API_KEY", ""),
+        daily_api_url=os.getenv("DAILY_API_URL", 'https://api.daily.co/v1'),
+        aiohttp_session=aiohttp_session
+    )
+    yield
+    await aiohttp_session.close()
+    cleanup()
 
-
-app = FastAPI()
+app = FastAPI(lifespan=lifespan)
 
 app.add_middleware(
     CORSMiddleware,
@@ -39,45 +59,45 @@ app.add_middleware(
 @app.get("/start")
 async def start_agent(request: Request):
     print(f"!!! Creating room")
-    room_url, room_name = _create_room()
-    print(f"!!! Room URL: {room_url}")
+    room = await daily_helpers["rest"].create_room(DailyRoomParams())
+    print(f"!!! Room URL: {room.url}")
     # Ensure the room property is present
-    if not room_url:
+    if not room.url:
         raise HTTPException(
             status_code=500,
             detail="Missing 'room' property in request data. Cannot start agent without a target room!")
 
     # Check if there is already an existing process running in this room
     num_bots_in_room = sum(
-        1 for proc in bot_procs.values() if proc[1] == room_url and proc[0].poll() is None)
+        1 for proc in bot_procs.values() if proc[1] == room.url and proc[0].poll() is None)
     if num_bots_in_room >= MAX_BOTS_PER_ROOM:
         raise HTTPException(
-            status_code=500, detail=f"Max bot limited reach for room: {room_url}")
+            status_code=500, detail=f"Max bot limited reach for room: {room.url}")
 
     # Get the token for the room
-    token = get_token(room_url)
+    token = await daily_helpers["rest"].get_token(room.url)
 
     if not token:
         raise HTTPException(
-            status_code=500, detail=f"Failed to get token for room: {room_url}")
+            status_code=500, detail=f"Failed to get token for room: {room.url}")
 
     # Spawn a new agent, and join the user session
     # Note: this is mostly for demonstration purposes (refer to 'deployment' in README)
     try:
         proc = subprocess.Popen(
             [
-                f"python3 -m bot -u {room_url} -t {token}"
+                f"python3 -m bot -u {room.url} -t {token}"
             ],
             shell=True,
             bufsize=1,
             cwd=os.path.dirname(os.path.abspath(__file__))
         )
-        bot_procs[proc.pid] = (proc, room_url)
+        bot_procs[proc.pid] = (proc, room.url)
     except Exception as e:
         raise HTTPException(
             status_code=500, detail=f"Failed to start subprocess: {e}")
 
-    return RedirectResponse(room_url)
+    return RedirectResponse(room.url)
 
 
 @app.get("/status/{pid}")

examples/moondream-chatbot/utils/daily_helpers.py

@@ -1,109 +0,0 @@
-
-import urllib.parse
-import os
-import time
-import urllib
-import requests
-
-from dotenv import load_dotenv
-load_dotenv()
-
-
-daily_api_path = os.getenv("DAILY_API_URL") or "api.daily.co/v1"
-daily_api_key = os.getenv("DAILY_API_KEY")
-
-
-def create_room() -> tuple[str, str]:
-    """
-    Helper function to create a Daily room.
-    # See: https://docs.daily.co/reference/rest-api/rooms
-
-    Returns:
-        tuple: A tuple containing the room URL and room name.
-
-    Raises:
-        Exception: If the request to create the room fails or if the response does not contain the room URL or room name.
-    """
-    room_props = {
-        "exp": time.time() + 60 * 60,  # 1 hour
-        "enable_chat": True,
-        "enable_emoji_reactions": True,
-        "eject_at_room_exp": True,
-        "enable_prejoin_ui": False,  # Important for the bot to be able to join headlessly
-    }
-    res = requests.post(
-        f"https://{daily_api_path}/rooms",
-        headers={"Authorization": f"Bearer {daily_api_key}"},
-        json={
-            "properties": room_props
-        },
-    )
-    if res.status_code != 200:
-        raise Exception(f"Unable to create room: {res.text}")
-
-    data = res.json()
-    room_url: str = data.get("url")
-    room_name: str = data.get("name")
-    if room_url is None or room_name is None:
-        raise Exception("Missing room URL or room name in response")
-
-    return room_url, room_name
-
-
-def get_name_from_url(room_url: str) -> str:
-    """
-    Extracts the name from a given room URL.
-
-    Args:
-        room_url (str): The URL of the room.
-
-    Returns:
-        str: The extracted name from the room URL.
-    """
-    return urllib.parse.urlparse(room_url).path[1:]
-
-
-def get_token(room_url: str) -> str:
-    """
-    Retrieves a meeting token for the specified Daily room URL.
-    # See: https://docs.daily.co/reference/rest-api/meeting-tokens
-
-    Args:
-        room_url (str): The URL of the Daily room.
-
-    Returns:
-        str: The meeting token.
-
-    Raises:
-        Exception: If no room URL is specified or if no Daily API key is specified.
-        Exception: If there is an error creating the meeting token.
-    """
-    if not room_url:
-        raise Exception(
-            "No Daily room specified. You must specify a Daily room in order a token to be generated.")
-
-    if not daily_api_key:
-        raise Exception(
-            "No Daily API key specified. set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.")
-
-    expiration: float = time.time() + 60 * 60
-    room_name = get_name_from_url(room_url)
-
-    res: requests.Response = requests.post(
-        f"https://{daily_api_path}/meeting-tokens",
-        headers={
-            "Authorization": f"Bearer {daily_api_key}"},
-        json={
-            "properties": {
-                "room_name": room_name,
-                "is_owner": True,  # Owner tokens required for transcription
-                "exp": expiration}},
-    )
-
-    if res.status_code != 200:
-        raise Exception(
-            f"Failed to create meeting token: {res.status_code} {res.text}")
-
-    token: str = res.json()["token"]
-
-    return token

examples/patient-intake/bot.py

@@ -257,17 +257,16 @@ class IntakeProcessor:
         return None
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
             "Chatbot",
             DailyParams(
                 audio_out_enabled=True,
-                camera_out_enabled=True,
-                camera_out_width=1024,
-                camera_out_height=576,
                 vad_enabled=True,
                 vad_analyzer=SileroVADAnalyzer(),
                 transcription_enabled=True,
@@ -351,5 +350,4 @@ async def main(room_url: str, token):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/patient-intake/requirements.txt

@@ -1,5 +1,4 @@
 python-dotenv
-requests
 fastapi[all]
 uvicorn
 pipecat-ai[daily,openai,silero]

examples/patient-intake/runner.py

@@ -1,11 +1,17 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import aiohttp
 import argparse
 import os
-import time
-import urllib
-import requests
+
+from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper
 
 
-def configure():
+async def configure(aiohttp_session: aiohttp.ClientSession):
     parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample")
     parser.add_argument(
         "-u",
@@ -33,26 +39,15 @@ def configure():
     if not key:
         raise Exception("No Daily API key specified. use the -k/--apikey option from the command line, or set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.")
 
+    daily_rest_helper = DailyRESTHelper(
+        daily_api_key=key,
+        daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
+        aiohttp_session=aiohttp_session)
+
     # Create a meeting token for the given room with an expiration 1 hour in
     # the future.
-    room_name: str = urllib.parse.urlparse(url).path[1:]
-    expiration: float = time.time() + 60 * 60
+    expiry_time: float = 60 * 60
 
-    res: requests.Response = requests.post(
-        f"https://api.daily.co/v1/meeting-tokens",
-        headers={
-            "Authorization": f"Bearer {key}"},
-        json={
-            "properties": {
-                "room_name": room_name,
-                "is_owner": True,
-                "exp": expiration}},
-    )
-
-    if res.status_code != 200:
-        raise Exception(
-            f"Failed to create meeting token: {res.status_code} {res.text}")
-
-    token: str = res.json()["token"]
+    token = await daily_rest_helper.get_token(url, expiry_time)
 
     return (url, token)

examples/patient-intake/server.py

@@ -1,31 +1,51 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import aiohttp
 import os
 import argparse
 import subprocess
-import atexit
+
+from contextlib import asynccontextmanager
 
 from fastapi import FastAPI, Request, HTTPException
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.responses import JSONResponse, RedirectResponse
 
-from utils.daily_helpers import create_room as _create_room, get_token
+from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper, DailyRoomParams
 
 MAX_BOTS_PER_ROOM = 1
 
 # Bot sub-process dict for status reporting and concurrency control
 bot_procs = {}
 
+daily_helpers = {}
+
 
 def cleanup():
     # Clean up function, just to be extra safe
-    for proc in bot_procs.values():
+    for entry in bot_procs.values():
+        proc = entry[0]
         proc.terminate()
         proc.wait()
 
 
-atexit.register(cleanup)
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    aiohttp_session = aiohttp.ClientSession()
+    daily_helpers["rest"] = DailyRESTHelper(
+        daily_api_key=os.getenv("DAILY_API_KEY", ""),
+        daily_api_url=os.getenv("DAILY_API_URL", 'https://api.daily.co/v1'),
+        aiohttp_session=aiohttp_session
+    )
+    yield
+    await aiohttp_session.close()
+    cleanup()
 
-
-app = FastAPI()
+app = FastAPI(lifespan=lifespan)
 
 app.add_middleware(
     CORSMiddleware,
@@ -39,45 +59,45 @@ app.add_middleware(
 @app.get("/start")
 async def start_agent(request: Request):
     print(f"!!! Creating room")
-    room_url, room_name = _create_room()
-    print(f"!!! Room URL: {room_url}")
+    room = await daily_helpers["rest"].create_room(DailyRoomParams())
+    print(f"!!! Room URL: {room.url}")
     # Ensure the room property is present
-    if not room_url:
+    if not room.url:
         raise HTTPException(
             status_code=500,
             detail="Missing 'room' property in request data. Cannot start agent without a target room!")
 
     # Check if there is already an existing process running in this room
     num_bots_in_room = sum(
-        1 for proc in bot_procs.values() if proc[1] == room_url and proc[0].poll() is None)
+        1 for proc in bot_procs.values() if proc[1] == room.url and proc[0].poll() is None)
     if num_bots_in_room >= MAX_BOTS_PER_ROOM:
         raise HTTPException(
-            status_code=500, detail=f"Max bot limited reach for room: {room_url}")
+            status_code=500, detail=f"Max bot limited reach for room: {room.url}")
 
     # Get the token for the room
-    token = get_token(room_url)
+    token = await daily_helpers["rest"].get_token(room.url)
 
     if not token:
         raise HTTPException(
-            status_code=500, detail=f"Failed to get token for room: {room_url}")
+            status_code=500, detail=f"Failed to get token for room: {room.url}")
 
     # Spawn a new agent, and join the user session
     # Note: this is mostly for demonstration purposes (refer to 'deployment' in README)
     try:
         proc = subprocess.Popen(
             [
-                f"python3 -m bot -u {room_url} -t {token}"
+                f"python3 -m bot -u {room.url} -t {token}"
             ],
             shell=True,
             bufsize=1,
             cwd=os.path.dirname(os.path.abspath(__file__))
         )
-        bot_procs[proc.pid] = (proc, room_url)
+        bot_procs[proc.pid] = (proc, room.url)
     except Exception as e:
         raise HTTPException(
             status_code=500, detail=f"Failed to start subprocess: {e}")
 
-    return RedirectResponse(room_url)
+    return RedirectResponse(room.url)
 
 
 @app.get("/status/{pid}")

examples/patient-intake/utils/daily_helpers.py

@@ -1,109 +0,0 @@
-
-import urllib.parse
-import os
-import time
-import urllib
-import requests
-
-from dotenv import load_dotenv
-load_dotenv()
-
-
-daily_api_path = os.getenv("DAILY_API_URL") or "api.daily.co/v1"
-daily_api_key = os.getenv("DAILY_API_KEY")
-
-
-def create_room() -> tuple[str, str]:
-    """
-    Helper function to create a Daily room.
-    # See: https://docs.daily.co/reference/rest-api/rooms
-
-    Returns:
-        tuple: A tuple containing the room URL and room name.
-
-    Raises:
-        Exception: If the request to create the room fails or if the response does not contain the room URL or room name.
-    """
-    room_props = {
-        "exp": time.time() + 60 * 60,  # 1 hour
-        "enable_chat": True,
-        "enable_emoji_reactions": True,
-        "eject_at_room_exp": True,
-        "enable_prejoin_ui": False,  # Important for the bot to be able to join headlessly
-    }
-    res = requests.post(
-        f"https://{daily_api_path}/rooms",
-        headers={"Authorization": f"Bearer {daily_api_key}"},
-        json={
-            "properties": room_props
-        },
-    )
-    if res.status_code != 200:
-        raise Exception(f"Unable to create room: {res.text}")
-
-    data = res.json()
-    room_url: str = data.get("url")
-    room_name: str = data.get("name")
-    if room_url is None or room_name is None:
-        raise Exception("Missing room URL or room name in response")
-
-    return room_url, room_name
-
-
-def get_name_from_url(room_url: str) -> str:
-    """
-    Extracts the name from a given room URL.
-
-    Args:
-        room_url (str): The URL of the room.
-
-    Returns:
-        str: The extracted name from the room URL.
-    """
-    return urllib.parse.urlparse(room_url).path[1:]
-
-
-def get_token(room_url: str) -> str:
-    """
-    Retrieves a meeting token for the specified Daily room URL.
-    # See: https://docs.daily.co/reference/rest-api/meeting-tokens
-
-    Args:
-        room_url (str): The URL of the Daily room.
-
-    Returns:
-        str: The meeting token.
-
-    Raises:
-        Exception: If no room URL is specified or if no Daily API key is specified.
-        Exception: If there is an error creating the meeting token.
-    """
-    if not room_url:
-        raise Exception(
-            "No Daily room specified. You must specify a Daily room in order a token to be generated.")
-
-    if not daily_api_key:
-        raise Exception(
-            "No Daily API key specified. set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.")
-
-    expiration: float = time.time() + 60 * 60
-    room_name = get_name_from_url(room_url)
-
-    res: requests.Response = requests.post(
-        f"https://{daily_api_path}/meeting-tokens",
-        headers={
-            "Authorization": f"Bearer {daily_api_key}"},
-        json={
-            "properties": {
-                "room_name": room_name,
-                "is_owner": True,  # Owner tokens required for transcription
-                "exp": expiration}},
-    )
-
-    if res.status_code != 200:
-        raise Exception(
-            f"Failed to create meeting token: {res.status_code} {res.text}")
-
-    token: str = res.json()["token"]
-
-    return token

examples/simple-chatbot/bot.py

@@ -1,3 +1,9 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
 import asyncio
 import aiohttp
 import os
@@ -77,8 +83,10 @@ class TalkingAnimation(FrameProcessor):
         await self.push_frame(frame)
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -165,5 +173,4 @@ async def main(room_url: str, token):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/simple-chatbot/requirements.txt

@@ -1,5 +1,4 @@
 python-dotenv
-requests
 fastapi[all]
 uvicorn
 pipecat-ai[daily,openai,silero]

examples/simple-chatbot/runner.py

@@ -1,11 +1,17 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import aiohttp
 import argparse
 import os
-import time
-import urllib
-import requests
+
+from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper
 
 
-def configure():
+async def configure(aiohttp_session: aiohttp.ClientSession):
     parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample")
     parser.add_argument(
         "-u",
@@ -33,26 +39,16 @@ def configure():
     if not key:
         raise Exception("No Daily API key specified. use the -k/--apikey option from the command line, or set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.")
 
-    # Create a meeting token for the given room with an expiration 1 hour in
-    # the future.
-    room_name: str = urllib.parse.urlparse(url).path[1:]
-    expiration: float = time.time() + 60 * 60
-
-    res: requests.Response = requests.post(
-        f"https://api.daily.co/v1/meeting-tokens",
-        headers={
-            "Authorization": f"Bearer {key}"},
-        json={
-            "properties": {
-                "room_name": room_name,
-                "is_owner": True,
-                "exp": expiration}},
+    daily_rest_helper = DailyRESTHelper(
+        daily_api_key=key,
+        daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
+        aiohttp_session=aiohttp_session
     )
 
-    if res.status_code != 200:
-        raise Exception(
-            f"Failed to create meeting token: {res.status_code} {res.text}")
+    # Create a meeting token for the given room with an expiration 1 hour in
+    # the future.
+    expiry_time: float = 60 * 60
 
-    token: str = res.json()["token"]
+    token = await daily_rest_helper.get_token(url, expiry_time)
 
     return (url, token)

examples/simple-chatbot/server.py

@@ -1,31 +1,51 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import aiohttp
 import os
 import argparse
 import subprocess
-import atexit
+
+from contextlib import asynccontextmanager
 
 from fastapi import FastAPI, Request, HTTPException
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.responses import JSONResponse, RedirectResponse
 
-from utils.daily_helpers import create_room as _create_room, get_token
+from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper, DailyRoomParams
 
 MAX_BOTS_PER_ROOM = 1
 
 # Bot sub-process dict for status reporting and concurrency control
 bot_procs = {}
 
+daily_helpers = {}
+
 
 def cleanup():
     # Clean up function, just to be extra safe
-    for proc in bot_procs.values():
+    for entry in bot_procs.values():
+        proc = entry[0]
         proc.terminate()
         proc.wait()
 
 
-atexit.register(cleanup)
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    aiohttp_session = aiohttp.ClientSession()
+    daily_helpers["rest"] = DailyRESTHelper(
+        daily_api_key=os.getenv("DAILY_API_KEY", ""),
+        daily_api_url=os.getenv("DAILY_API_URL", 'https://api.daily.co/v1'),
+        aiohttp_session=aiohttp_session
+    )
+    yield
+    await aiohttp_session.close()
+    cleanup()
 
-
-app = FastAPI()
+app = FastAPI(lifespan=lifespan)
 
 app.add_middleware(
     CORSMiddleware,
@@ -39,45 +59,45 @@ app.add_middleware(
 @app.get("/start")
 async def start_agent(request: Request):
     print(f"!!! Creating room")
-    room_url, room_name = _create_room()
-    print(f"!!! Room URL: {room_url}")
+    room = await daily_helpers["rest"].create_room(DailyRoomParams())
+    print(f"!!! Room URL: {room.url}")
     # Ensure the room property is present
-    if not room_url:
+    if not room.url:
         raise HTTPException(
             status_code=500,
             detail="Missing 'room' property in request data. Cannot start agent without a target room!")
 
     # Check if there is already an existing process running in this room
     num_bots_in_room = sum(
-        1 for proc in bot_procs.values() if proc[1] == room_url and proc[0].poll() is None)
+        1 for proc in bot_procs.values() if proc[1] == room.url and proc[0].poll() is None)
     if num_bots_in_room >= MAX_BOTS_PER_ROOM:
         raise HTTPException(
-            status_code=500, detail=f"Max bot limited reach for room: {room_url}")
+            status_code=500, detail=f"Max bot limited reach for room: {room.url}")
 
     # Get the token for the room
-    token = get_token(room_url)
+    token = await daily_helpers["rest"].get_token(room.url)
 
     if not token:
         raise HTTPException(
-            status_code=500, detail=f"Failed to get token for room: {room_url}")
+            status_code=500, detail=f"Failed to get token for room: {room.url}")
 
     # Spawn a new agent, and join the user session
     # Note: this is mostly for demonstration purposes (refer to 'deployment' in README)
     try:
         proc = subprocess.Popen(
             [
-                f"python3 -m bot -u {room_url} -t {token}"
+                f"python3 -m bot -u {room.url} -t {token}"
             ],
             shell=True,
             bufsize=1,
             cwd=os.path.dirname(os.path.abspath(__file__))
         )
-        bot_procs[proc.pid] = (proc, room_url)
+        bot_procs[proc.pid] = (proc, room.url)
     except Exception as e:
         raise HTTPException(
             status_code=500, detail=f"Failed to start subprocess: {e}")
 
-    return RedirectResponse(room_url)
+    return RedirectResponse(room.url)
 
 
 @app.get("/status/{pid}")

examples/simple-chatbot/utils/daily_helpers.py

@@ -1,109 +0,0 @@
-
-import urllib.parse
-import os
-import time
-import urllib
-import requests
-
-from dotenv import load_dotenv
-load_dotenv()
-
-
-daily_api_path = os.getenv("DAILY_API_URL") or "api.daily.co/v1"
-daily_api_key = os.getenv("DAILY_API_KEY")
-
-
-def create_room() -> tuple[str, str]:
-    """
-    Helper function to create a Daily room.
-    # See: https://docs.daily.co/reference/rest-api/rooms
-
-    Returns:
-        tuple: A tuple containing the room URL and room name.
-
-    Raises:
-        Exception: If the request to create the room fails or if the response does not contain the room URL or room name.
-    """
-    room_props = {
-        "exp": time.time() + 60 * 60,  # 1 hour
-        "enable_chat": True,
-        "enable_emoji_reactions": True,
-        "eject_at_room_exp": True,
-        "enable_prejoin_ui": False,  # Important for the bot to be able to join headlessly
-    }
-    res = requests.post(
-        f"https://{daily_api_path}/rooms",
-        headers={"Authorization": f"Bearer {daily_api_key}"},
-        json={
-            "properties": room_props
-        },
-    )
-    if res.status_code != 200:
-        raise Exception(f"Unable to create room: {res.text}")
-
-    data = res.json()
-    room_url: str = data.get("url")
-    room_name: str = data.get("name")
-    if room_url is None or room_name is None:
-        raise Exception("Missing room URL or room name in response")
-
-    return room_url, room_name
-
-
-def get_name_from_url(room_url: str) -> str:
-    """
-    Extracts the name from a given room URL.
-
-    Args:
-        room_url (str): The URL of the room.
-
-    Returns:
-        str: The extracted name from the room URL.
-    """
-    return urllib.parse.urlparse(room_url).path[1:]
-
-
-def get_token(room_url: str) -> str:
-    """
-    Retrieves a meeting token for the specified Daily room URL.
-    # See: https://docs.daily.co/reference/rest-api/meeting-tokens
-
-    Args:
-        room_url (str): The URL of the Daily room.
-
-    Returns:
-        str: The meeting token.
-
-    Raises:
-        Exception: If no room URL is specified or if no Daily API key is specified.
-        Exception: If there is an error creating the meeting token.
-    """
-    if not room_url:
-        raise Exception(
-            "No Daily room specified. You must specify a Daily room in order a token to be generated.")
-
-    if not daily_api_key:
-        raise Exception(
-            "No Daily API key specified. set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.")
-
-    expiration: float = time.time() + 60 * 60
-    room_name = get_name_from_url(room_url)
-
-    res: requests.Response = requests.post(
-        f"https://{daily_api_path}/meeting-tokens",
-        headers={
-            "Authorization": f"Bearer {daily_api_key}"},
-        json={
-            "properties": {
-                "room_name": room_name,
-                "is_owner": True,  # Owner tokens required for transcription
-                "exp": expiration}},
-    )
-
-    if res.status_code != 200:
-        raise Exception(
-            f"Failed to create meeting token: {res.status_code} {res.text}")
-
-    token: str = res.json()["token"]
-
-    return token

examples/storytelling-chatbot/requirements.txt

@@ -1,6 +1,5 @@
 async_timeout
 fastapi
 uvicorn
-requests
 python-dotenv
 pipecat-ai[daily,openai,fal]

examples/storytelling-chatbot/src/bot_runner.py

@@ -1,16 +1,26 @@
-import os
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import aiohttp
 import argparse
 import subprocess
-import requests
+import os
+
 from pathlib import Path
 from typing import Optional
 
+from contextlib import asynccontextmanager
+
 from fastapi import FastAPI, Request, HTTPException
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.staticfiles import StaticFiles
 from fastapi.responses import FileResponse, JSONResponse
 
-from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper, DailyRoomObject, DailyRoomProperties, DailyRoomParams
+from pipecat.transports.services.helpers.daily_rest import (
+    DailyRESTHelper, DailyRoomObject, DailyRoomProperties, DailyRoomParams)
 
 
 from dotenv import load_dotenv
@@ -20,12 +30,21 @@ load_dotenv(override=True)
 
 MAX_SESSION_TIME = 5 * 60  # 5 minutes
 
-daily_rest_helper = DailyRESTHelper(
-    os.getenv("DAILY_API_KEY", ""),
-    os.getenv("DAILY_API_URL", 'https://api.daily.co/v1'))
+daily_helpers = {}
 
 
-app = FastAPI()
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    aiohttp_session = aiohttp.ClientSession()
+    daily_helpers["rest"] = DailyRESTHelper(
+        daily_api_key=os.getenv("DAILY_API_KEY", ""),
+        daily_api_url=os.getenv("DAILY_API_URL", 'https://api.daily.co/v1'),
+        aiohttp_session=aiohttp_session
+    )
+    yield
+    await aiohttp_session.close()
+
+app = FastAPI(lifespan=lifespan)
 
 app.add_middleware(
     CORSMiddleware,
@@ -70,7 +89,7 @@ async def start_bot(request: Request) -> JSONResponse:
             properties=DailyRoomProperties()
         )
         try:
-            room: DailyRoomObject = daily_rest_helper.create_room(params=params)
+            room: DailyRoomObject = await daily_helpers["rest"].create_room(params=params)
         except Exception as e:
             raise HTTPException(
                 status_code=500,
@@ -78,13 +97,13 @@ async def start_bot(request: Request) -> JSONResponse:
     else:
         # Check passed room URL exists, we should assume that it already has a sip set up
         try:
-            room: DailyRoomObject = daily_rest_helper.get_room_from_url(room_url)
+            room: DailyRoomObject = await daily_helpers["rest"].get_room_from_url(room_url)
         except Exception:
             raise HTTPException(
                 status_code=500, detail=f"Room not found: {room_url}")
 
     # Give the agent a token to join the session
-    token = daily_rest_helper.get_token(room.url, MAX_SESSION_TIME)
+    token = await daily_helpers["rest"].get_token(room.url, MAX_SESSION_TIME)
 
     if not room or not token:
         raise HTTPException(
@@ -93,7 +112,7 @@ async def start_bot(request: Request) -> JSONResponse:
     # Launch a new VM, or run as a shell process (not recommended)
     if os.getenv("RUN_AS_VM", False):
         try:
-            virtualize_bot(room.url, token)
+            await virtualize_bot(room.url, token)
         except Exception as e:
             raise HTTPException(
                 status_code=500, detail=f"Failed to spawn VM: {e}")
@@ -109,7 +128,7 @@ async def start_bot(request: Request) -> JSONResponse:
                 status_code=500, detail=f"Failed to start subprocess: {e}")
 
     # Grab a token for the user to join with
-    user_token = daily_rest_helper.get_token(room.url, MAX_SESSION_TIME)
+    user_token = await daily_helpers["rest"].get_token(room.url, MAX_SESSION_TIME)
 
     return JSONResponse({
         "room_url": room.url,
@@ -136,7 +155,7 @@ async def catch_all(path_name: Optional[str] = ""):
 
 # ------------ Virtualization ------------ #
 
-def virtualize_bot(room_url: str, token: str):
+async def virtualize_bot(room_url: str, token: str):
     """
     This is an example of how to virtualize the bot using Fly.io
     You can adapt this method to use whichever cloud provider you prefer.
@@ -149,54 +168,53 @@ def virtualize_bot(room_url: str, token: str):
         'Content-Type': 'application/json'
     }
 
-    # Use the same image as the bot runner
-    res = requests.get(f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines", headers=FLY_HEADERS)
-    if res.status_code != 200:
-        raise Exception(f"Unable to get machine info from Fly: {res.text}")
-    image = res.json()[0]['config']['image']
+    async with aiohttp.ClientSession() as session:
+        # Use the same image as the bot runner
+        async with session.get(f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines", headers=FLY_HEADERS) as r:
+            if r.status != 200:
+                text = await r.text()
+                raise Exception(f"Unable to get machine info from Fly: {text}")
 
-    # Machine configuration
-    cmd = f"python3 src/bot.py -u {room_url} -t {token}"
-    cmd = cmd.split()
-    worker_props = {
-        "config": {
-            "image": image,
-            "auto_destroy": True,
-            "init": {
-                "cmd": cmd
+            data = await r.json()
+            image = data[0]['config']['image']
+
+        # Machine configuration
+        cmd = f"python3 src/bot.py -u {room_url} -t {token}"
+        cmd = cmd.split()
+        worker_props = {
+            "config": {
+                "image": image,
+                "auto_destroy": True,
+                "init": {
+                    "cmd": cmd
+                },
+                "restart": {
+                    "policy": "no"
+                },
+                "guest": {
+                    "cpu_kind": "shared",
+                    "cpus": 1,
+                    "memory_mb": 512
+                }
             },
-            "restart": {
-                "policy": "no"
-            },
-            "guest": {
-                "cpu_kind": "shared",
-                "cpus": 1,
-                "memory_mb": 512
-            }
-        },
+        }
 
-    }
+        # Spawn a new machine instance
+        async with session.post(f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines", headers=FLY_HEADERS, json=worker_props) as r:
+            if r.status != 200:
+                text = await r.text()
+                raise Exception(f"Problem starting a bot worker: {text}")
 
-    # Spawn a new machine instance
-    res = requests.post(
-        f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines",
-        headers=FLY_HEADERS,
-        json=worker_props)
+            data = await r.json()
+            # Wait for the machine to enter the started state
+            vm_id = data['id']
 
-    if res.status_code != 200:
-        raise Exception(f"Problem starting a bot worker: {res.text}")
+        async with session.get(f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines/{vm_id}/wait?state=started", headers=FLY_HEADERS) as r:
+            if r.status != 200:
+                text = await r.text()
+                raise Exception(f"Bot was unable to enter started state: {text}")
 
-    # Wait for the machine to enter the started state
-    vm_id = res.json()['id']
-
-    res = requests.get(
-        f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines/{vm_id}/wait?state=started",
-        headers=FLY_HEADERS)
-
-    if res.status_code != 200:
-        raise Exception(f"Bot was unable to enter started state: {res.text}")
-
-    print(f"Machine joined room: {room_url}")
+        print(f"Machine joined room: {room_url}")
 
 
 # ------------ Main ------------ #

examples/storytelling-chatbot/src/utils/daily_helpers.py

@@ -1,109 +0,0 @@
-
-import urllib.parse
-import os
-import time
-import urllib
-import requests
-
-from dotenv import load_dotenv
-load_dotenv()
-
-
-daily_api_path = os.getenv("DAILY_API_URL") or "api.daily.co/v1"
-daily_api_key = os.getenv("DAILY_API_KEY")
-
-
-def create_room() -> tuple[str, str]:
-    """
-    Helper function to create a Daily room.
-    # See: https://docs.daily.co/reference/rest-api/rooms
-
-    Returns:
-        tuple: A tuple containing the room URL and room name.
-
-    Raises:
-        Exception: If the request to create the room fails or if the response does not contain the room URL or room name.
-    """
-    room_props = {
-        "exp": time.time() + 60 * 60,  # 1 hour
-        "enable_chat": True,
-        "enable_emoji_reactions": True,
-        "eject_at_room_exp": True,
-        "enable_prejoin_ui": False,  # Important for the bot to be able to join headlessly
-    }
-    res = requests.post(
-        f"https://{daily_api_path}/rooms",
-        headers={"Authorization": f"Bearer {daily_api_key}"},
-        json={
-            "properties": room_props
-        },
-    )
-    if res.status_code != 200:
-        raise Exception(f"Unable to create room: {res.text}")
-
-    data = res.json()
-    room_url: str = data.get("url")
-    room_name: str = data.get("name")
-    if room_url is None or room_name is None:
-        raise Exception("Missing room URL or room name in response")
-
-    return room_url, room_name
-
-
-def get_name_from_url(room_url: str) -> str:
-    """
-    Extracts the name from a given room URL.
-
-    Args:
-        room_url (str): The URL of the room.
-
-    Returns:
-        str: The extracted name from the room URL.
-    """
-    return urllib.parse.urlparse(room_url).path[1:]
-
-
-def get_token(room_url: str) -> str:
-    """
-    Retrieves a meeting token for the specified Daily room URL.
-    # See: https://docs.daily.co/reference/rest-api/meeting-tokens
-
-    Args:
-        room_url (str): The URL of the Daily room.
-
-    Returns:
-        str: The meeting token.
-
-    Raises:
-        Exception: If no room URL is specified or if no Daily API key is specified.
-        Exception: If there is an error creating the meeting token.
-    """
-    if not room_url:
-        raise Exception(
-            "No Daily room specified. You must specify a Daily room in order a token to be generated.")
-
-    if not daily_api_key:
-        raise Exception(
-            "No Daily API key specified. set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.")
-
-    expiration: float = time.time() + 60 * 60
-    room_name = get_name_from_url(room_url)
-
-    res: requests.Response = requests.post(
-        f"https://{daily_api_path}/meeting-tokens",
-        headers={
-            "Authorization": f"Bearer {daily_api_key}"},
-        json={
-            "properties": {
-                "room_name": room_name,
-                "is_owner": True,  # Owner tokens required for transcription
-                "exp": expiration}},
-    )
-
-    if res.status_code != 200:
-        raise Exception(
-            f"Failed to create meeting token: {res.status_code} {res.text}")
-
-    token: str = res.json()["token"]
-
-    return token

examples/translation-chatbot/bot.py

@@ -1,5 +1,11 @@
-import asyncio
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
 import aiohttp
+import asyncio
 import os
 import sys
 
@@ -12,7 +18,11 @@ from pipecat.processors.aggregators.sentence import SentenceAggregator
 from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
 from pipecat.services.azure import AzureTTSService
 from pipecat.services.openai import OpenAILLMService
-from pipecat.transports.services.daily import DailyParams, DailyTranscriptionSettings, DailyTransport, DailyTransportMessageFrame
+from pipecat.transports.services.daily import (
+    DailyParams,
+    DailyTranscriptionSettings,
+    DailyTransport,
+    DailyTransportMessageFrame)
 
 from runner import configure
 
@@ -79,8 +89,10 @@ class TranslationSubtitles(FrameProcessor):
         await self.push_frame(frame)
 
 
-async def main(room_url: str, token):
+async def main():
     async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
         transport = DailyTransport(
             room_url,
             token,
@@ -133,5 +145,4 @@ async def main(room_url: str, token):
 
 
 if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
+    asyncio.run(main())

examples/translation-chatbot/requirements.txt

@@ -1,4 +1,3 @@
 python-dotenv
-requests
 fastapi[all]
 pipecat-ai[daily,openai,azure]

examples/translation-chatbot/runner.py

@@ -1,11 +1,18 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
 import argparse
 import os
-import time
-import urllib
-import requests
+
+import aiohttp
+
+from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper
 
 
-def configure():
+async def configure(aiohttp_session: aiohttp.ClientSession):
     parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample")
     parser.add_argument(
         "-u",
@@ -33,26 +40,16 @@ def configure():
     if not key:
         raise Exception("No Daily API key specified. use the -k/--apikey option from the command line, or set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.")
 
-    # Create a meeting token for the given room with an expiration 1 hour in
-    # the future.
-    room_name: str = urllib.parse.urlparse(url).path[1:]
-    expiration: float = time.time() + 60 * 60
-
-    res: requests.Response = requests.post(
-        f"https://api.daily.co/v1/meeting-tokens",
-        headers={
-            "Authorization": f"Bearer {key}"},
-        json={
-            "properties": {
-                "room_name": room_name,
-                "is_owner": True,
-                "exp": expiration}},
+    daily_rest_helper = DailyRESTHelper(
+        daily_api_key=key,
+        daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
+        aiohttp_session=aiohttp_session
     )
 
-    if res.status_code != 200:
-        raise Exception(
-            f"Failed to create meeting token: {res.status_code} {res.text}")
+    # Create a meeting token for the given room with an expiration 1 hour in
+    # the future.
+    expiry_time: float = 60 * 60
 
-    token: str = res.json()["token"]
+    token = await daily_rest_helper.get_token(url, expiry_time)
 
     return (url, token)

examples/translation-chatbot/server.py

@@ -1,31 +1,51 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import aiohttp
 import os
 import argparse
 import subprocess
-import atexit
+
+from contextlib import asynccontextmanager
 
 from fastapi import FastAPI, Request, HTTPException
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.responses import JSONResponse, RedirectResponse
 
-from utils.daily_helpers import create_room as _create_room, get_token
+from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper, DailyRoomParams
 
 MAX_BOTS_PER_ROOM = 1
 
 # Bot sub-process dict for status reporting and concurrency control
 bot_procs = {}
 
+daily_helpers = {}
+
 
 def cleanup():
     # Clean up function, just to be extra safe
-    for proc in bot_procs.values():
+    for entry in bot_procs.values():
+        proc = entry[0]
         proc.terminate()
         proc.wait()
 
 
-atexit.register(cleanup)
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    aiohttp_session = aiohttp.ClientSession()
+    daily_helpers["rest"] = DailyRESTHelper(
+        daily_api_key=os.getenv("DAILY_API_KEY", ""),
+        daily_api_url=os.getenv("DAILY_API_URL", 'https://api.daily.co/v1'),
+        aiohttp_session=aiohttp_session
+    )
+    yield
+    await aiohttp_session.close()
+    cleanup()
 
-
-app = FastAPI()
+app = FastAPI(lifespan=lifespan)
 
 app.add_middleware(
     CORSMiddleware,
@@ -39,45 +59,45 @@ app.add_middleware(
 @app.get("/start")
 async def start_agent(request: Request):
     print(f"!!! Creating room")
-    room_url, room_name = _create_room()
-    print(f"!!! Room URL: {room_url}")
+    room = await daily_helpers["rest"].create_room(DailyRoomParams())
+    print(f"!!! Room URL: {room.url}")
     # Ensure the room property is present
-    if not room_url:
+    if not room.url:
         raise HTTPException(
             status_code=500,
             detail="Missing 'room' property in request data. Cannot start agent without a target room!")
 
     # Check if there is already an existing process running in this room
     num_bots_in_room = sum(
-        1 for proc in bot_procs.values() if proc[1] == room_url and proc[0].poll() is None)
+        1 for proc in bot_procs.values() if proc[1] == room.url and proc[0].poll() is None)
     if num_bots_in_room >= MAX_BOTS_PER_ROOM:
         raise HTTPException(
-            status_code=500, detail=f"Max bot limited reach for room: {room_url}")
+            status_code=500, detail=f"Max bot limited reach for room: {room.url}")
 
     # Get the token for the room
-    token = get_token(room_url)
+    token = await daily_helpers["rest"].get_token(room.url)
 
     if not token:
         raise HTTPException(
-            status_code=500, detail=f"Failed to get token for room: {room_url}")
+            status_code=500, detail=f"Failed to get token for room: {room.url}")
 
     # Spawn a new agent, and join the user session
     # Note: this is mostly for demonstration purposes (refer to 'deployment' in README)
     try:
         proc = subprocess.Popen(
             [
-                f"python3 -m bot -u {room_url} -t {token}"
+                f"python3 -m bot -u {room.url} -t {token}"
             ],
             shell=True,
             bufsize=1,
             cwd=os.path.dirname(os.path.abspath(__file__))
         )
-        bot_procs[proc.pid] = (proc, room_url)
+        bot_procs[proc.pid] = (proc, room.url)
     except Exception as e:
         raise HTTPException(
             status_code=500, detail=f"Failed to start subprocess: {e}")
 
-    return RedirectResponse(room_url)
+    return RedirectResponse(room.url)
 
 
 @app.get("/status/{pid}")

examples/translation-chatbot/utils/daily_helpers.py

@@ -1,109 +0,0 @@
-
-import urllib.parse
-import os
-import time
-import urllib
-import requests
-
-from dotenv import load_dotenv
-load_dotenv()
-
-
-daily_api_path = os.getenv("DAILY_API_URL") or "api.daily.co/v1"
-daily_api_key = os.getenv("DAILY_API_KEY")
-
-
-def create_room() -> tuple[str, str]:
-    """
-    Helper function to create a Daily room.
-    # See: https://docs.daily.co/reference/rest-api/rooms
-
-    Returns:
-        tuple: A tuple containing the room URL and room name.
-
-    Raises:
-        Exception: If the request to create the room fails or if the response does not contain the room URL or room name.
-    """
-    room_props = {
-        "exp": time.time() + 60 * 60,  # 1 hour
-        "enable_chat": True,
-        "enable_emoji_reactions": True,
-        "eject_at_room_exp": True,
-        "enable_prejoin_ui": False,  # Important for the bot to be able to join headlessly
-    }
-    res = requests.post(
-        f"https://{daily_api_path}/rooms",
-        headers={"Authorization": f"Bearer {daily_api_key}"},
-        json={
-            "properties": room_props
-        },
-    )
-    if res.status_code != 200:
-        raise Exception(f"Unable to create room: {res.text}")
-
-    data = res.json()
-    room_url: str = data.get("url")
-    room_name: str = data.get("name")
-    if room_url is None or room_name is None:
-        raise Exception("Missing room URL or room name in response")
-
-    return room_url, room_name
-
-
-def get_name_from_url(room_url: str) -> str:
-    """
-    Extracts the name from a given room URL.
-
-    Args:
-        room_url (str): The URL of the room.
-
-    Returns:
-        str: The extracted name from the room URL.
-    """
-    return urllib.parse.urlparse(room_url).path[1:]
-
-
-def get_token(room_url: str) -> str:
-    """
-    Retrieves a meeting token for the specified Daily room URL.
-    # See: https://docs.daily.co/reference/rest-api/meeting-tokens
-
-    Args:
-        room_url (str): The URL of the Daily room.
-
-    Returns:
-        str: The meeting token.
-
-    Raises:
-        Exception: If no room URL is specified or if no Daily API key is specified.
-        Exception: If there is an error creating the meeting token.
-    """
-    if not room_url:
-        raise Exception(
-            "No Daily room specified. You must specify a Daily room in order a token to be generated.")
-
-    if not daily_api_key:
-        raise Exception(
-            "No Daily API key specified. set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.")
-
-    expiration: float = time.time() + 60 * 60
-    room_name = get_name_from_url(room_url)
-
-    res: requests.Response = requests.post(
-        f"https://{daily_api_path}/meeting-tokens",
-        headers={
-            "Authorization": f"Bearer {daily_api_key}"},
-        json={
-            "properties": {
-                "room_name": room_name,
-                "is_owner": True,  # Owner tokens required for transcription
-                "exp": expiration}},
-    )
-
-    if res.status_code != 200:
-        raise Exception(
-            f"Failed to create meeting token: {res.status_code} {res.text}")
-
-    token: str = res.json()["token"]
-
-    return token

examples/twilio-chatbot/README.md

@@ -46,10 +46,16 @@ This project is a FastAPI-based chatbot that integrates with Twilio to handle We
 
 ## Configure Twilio URLs
 
-1. **Update the Twilio Webhook**:
+1. **Start ngrok**:
+    In a new terminal, start ngrok to tunnel the local server:
+    ```sh
+    ngrok http 8765
+    ```
+
+2. **Update the Twilio Webhook**:
     Copy the ngrok URL and update your Twilio phone number webhook URL to `http://<ngrok_url>/start_call`.
 
-2. **Update the streams.xml**:
+3. **Update the streams.xml**:
     Copy the ngrok URL and update templates/streams.xml with `wss://<ngrok_url>/ws`.
 
 ## Running the Application
@@ -61,11 +67,6 @@ This project is a FastAPI-based chatbot that integrates with Twilio to handle We
     python server.py
     ```
 
-2. **Start ngrok**:
-    In a new terminal, start ngrok to tunnel the local server:
-    ```sh
-    ngrok http 8765
-    ```
 ### Using Docker
 
 1. **Build the Docker image**:

linux-py3.10-requirements.txt

@@ -8,7 +8,6 @@ aiofiles==24.1.0
     # via deepgram-sdk
 aiohttp==3.9.5
     # via
-    #   cartesia
     #   deepgram-sdk
     #   langchain
     #   langchain-community
@@ -36,17 +35,15 @@ attrs==23.2.0
     # via
     #   aiohttp
     #   openpipe
-av==12.2.0
+av==12.3.0
     # via faster-whisper
 azure-cognitiveservices-speech==1.38.0
     # via pipecat-ai (pyproject.toml)
 blinker==1.8.2
     # via flask
-cachetools==5.3.3
+cachetools==5.4.0
     # via google-auth
-cartesia==1.0.3
-    # via pipecat-ai (pyproject.toml)
-certifi==2024.6.2
+certifi==2024.7.4
     # via
     #   httpcore
     #   httpx
@@ -80,15 +77,13 @@ einops==0.8.0
     # via pipecat-ai (pyproject.toml)
 email-validator==2.2.0
     # via fastapi
-exceptiongroup==1.2.1
-    # via
-    #   anyio
-    #   pytest
+exceptiongroup==1.2.2
+    # via anyio
 fal-client==0.4.1
     # via pipecat-ai (pyproject.toml)
-fastapi==0.111.0
+fastapi==0.111.1
     # via pipecat-ai (pyproject.toml)
-fastapi-cli==0.0.4
+fastapi-cli==0.0.5
     # via fastapi
 faster-whisper==1.0.3
     # via pipecat-ai (pyproject.toml)
@@ -124,9 +119,9 @@ google-api-core[grpc]==2.19.1
     #   google-ai-generativelanguage
     #   google-api-python-client
     #   google-generativeai
-google-api-python-client==2.135.0
+google-api-python-client==2.140.0
     # via google-generativeai
-google-auth==2.31.0
+google-auth==2.33.0
     # via
     #   google-ai-generativelanguage
     #   google-api-core
@@ -135,7 +130,7 @@ google-auth==2.31.0
     #   google-generativeai
 google-auth-httplib2==0.2.0
     # via google-api-python-client
-google-generativeai==0.7.1
+google-generativeai==0.7.2
     # via pipecat-ai (pyproject.toml)
 googleapis-common-protos==1.63.2
     # via
@@ -143,12 +138,12 @@ googleapis-common-protos==1.63.2
     #   grpcio-status
 greenlet==3.0.3
     # via sqlalchemy
-grpcio==1.64.1
+grpcio==1.65.4
     # via
     #   google-api-core
     #   grpcio-status
     #   pyht
-grpcio-status==1.62.2
+grpcio-status==1.62.3
     # via google-api-core
 h11==0.14.0
     # via
@@ -165,7 +160,6 @@ httptools==0.6.1
 httpx==0.27.0
     # via
     #   anthropic
-    #   cartesia
     #   deepgram-sdk
     #   fal-client
     #   fastapi
@@ -173,7 +167,7 @@ httpx==0.27.0
     #   openpipe
 httpx-sse==0.4.0
     # via fal-client
-huggingface-hub==0.23.4
+huggingface-hub==0.24.5
     # via
     #   faster-whisper
     #   timm
@@ -188,8 +182,6 @@ idna==3.7
     #   httpx
     #   requests
     #   yarl
-iniconfig==2.0.0
-    # via pytest
 itsdangerous==2.2.0
     # via flask
 jinja2==3.1.4
@@ -203,23 +195,23 @@ jsonpatch==1.33
     # via langchain-core
 jsonpointer==3.0.0
     # via jsonpatch
-langchain==0.2.6
+langchain==0.2.12
     # via
     #   langchain-community
     #   pipecat-ai (pyproject.toml)
-langchain-community==0.2.6
+langchain-community==0.2.11
     # via pipecat-ai (pyproject.toml)
-langchain-core==0.2.10
+langchain-core==0.2.29
     # via
     #   langchain
     #   langchain-community
     #   langchain-openai
     #   langchain-text-splitters
-langchain-openai==0.1.10
+langchain-openai==0.1.20
     # via pipecat-ai (pyproject.toml)
 langchain-text-splitters==0.2.2
     # via langchain
-langsmith==0.1.83
+langsmith==0.1.98
     # via
     #   langchain
     #   langchain-community
@@ -274,7 +266,7 @@ nvidia-cuda-nvrtc-cu12==12.1.105
     # via torch
 nvidia-cuda-runtime-cu12==12.1.105
     # via torch
-nvidia-cudnn-cu12==8.9.2.26
+nvidia-cudnn-cu12==9.1.0.70
     # via torch
 nvidia-cufft-cu12==11.0.2.54
     # via torch
@@ -288,44 +280,41 @@ nvidia-cusparse-cu12==12.1.0.106
     #   torch
 nvidia-nccl-cu12==2.20.5
     # via torch
-nvidia-nvjitlink-cu12==12.5.82
+nvidia-nvjitlink-cu12==12.6.20
     # via
     #   nvidia-cusolver-cu12
     #   nvidia-cusparse-cu12
 nvidia-nvtx-cu12==12.1.105
     # via torch
 onnxruntime==1.18.1
-    # via faster-whisper
-openai==1.27.0
+    # via
+    #   faster-whisper
+    #   silero-vad
+openai==1.35.15
     # via
     #   langchain-openai
     #   openpipe
     #   pipecat-ai (pyproject.toml)
-openpipe==4.16.0
+openpipe==4.18.0
     # via pipecat-ai (pyproject.toml)
-orjson==3.10.5
-    # via
-    #   fastapi
-    #   langsmith
+orjson==3.10.7
+    # via langsmith
 packaging==24.1
     # via
     #   huggingface-hub
     #   langchain-core
     #   marshmallow
     #   onnxruntime
-    #   pytest
     #   transformers
 pillow==10.3.0
     # via
     #   pipecat-ai (pyproject.toml)
     #   torchvision
-pluggy==1.5.0
-    # via pytest
 proto-plus==1.24.0
     # via
     #   google-ai-generativelanguage
     #   google-api-core
-protobuf==4.25.3
+protobuf==4.25.4
     # via
     #   google-ai-generativelanguage
     #   google-api-core
@@ -344,7 +333,7 @@ pyasn1-modules==0.4.0
     # via google-auth
 pyaudio==0.2.14
     # via pipecat-ai (pyproject.toml)
-pydantic==2.8.0
+pydantic==2.8.2
     # via
     #   anthropic
     #   fastapi
@@ -353,7 +342,7 @@ pydantic==2.8.0
     #   langchain-core
     #   langsmith
     #   openai
-pydantic-core==2.20.0
+pydantic-core==2.20.1
     # via pydantic
 pygments==2.18.0
     # via rich
@@ -363,10 +352,6 @@ pyloudnorm==0.1.1
     # via pipecat-ai (pyproject.toml)
 pyparsing==3.1.2
     # via httplib2
-pytest==8.2.2
-    # via pytest-asyncio
-pytest-asyncio==0.23.7
-    # via cartesia
 python-dateutil==2.9.0.post0
     # via openpipe
 python-dotenv==1.0.1
@@ -375,7 +360,7 @@ python-dotenv==1.0.1
     #   uvicorn
 python-multipart==0.0.9
     # via fastapi
-pyyaml==6.0.1
+pyyaml==6.0.2
     # via
     #   ctranslate2
     #   huggingface-hub
@@ -385,13 +370,12 @@ pyyaml==6.0.1
     #   timm
     #   transformers
     #   uvicorn
-regex==2024.5.15
+regex==2024.7.24
     # via
     #   tiktoken
     #   transformers
 requests==2.32.3
     # via
-    #   cartesia
     #   google-api-core
     #   huggingface-hub
     #   langchain
@@ -406,7 +390,7 @@ rich==13.7.1
     # via typer
 rsa==4.9
     # via google-auth
-safetensors==0.4.3
+safetensors==0.4.4
     # via
     #   timm
     #   transformers
@@ -414,6 +398,8 @@ scipy==1.14.0
     # via pyloudnorm
 shellingham==1.5.4
     # via typer
+silero-vad==5.1
+    # via pipecat-ai (pyproject.toml)
 six==1.16.0
     # via python-dateutil
 sniffio==1.3.1
@@ -422,17 +408,17 @@ sniffio==1.3.1
     #   anyio
     #   httpx
     #   openai
-sqlalchemy==2.0.31
+sqlalchemy==2.0.32
     # via
     #   langchain
     #   langchain-community
 starlette==0.37.2
     # via fastapi
-sympy==1.12.1
+sympy==1.13.1
     # via
     #   onnxruntime
     #   torch
-tenacity==8.4.2
+tenacity==8.5.0
     # via
     #   langchain
     #   langchain-community
@@ -446,19 +432,17 @@ tokenizers==0.19.1
     #   anthropic
     #   faster-whisper
     #   transformers
-tomli==2.0.1
-    # via pytest
-torch==2.3.1
+torch==2.4.0
     # via
-    #   pipecat-ai (pyproject.toml)
+    #   silero-vad
     #   timm
     #   torchaudio
     #   torchvision
-torchaudio==2.3.1
-    # via pipecat-ai (pyproject.toml)
-torchvision==0.18.1
+torchaudio==2.4.0
+    # via silero-vad
+torchvision==0.19.0
     # via timm
-tqdm==4.66.4
+tqdm==4.66.5
     # via
     #   google-generativeai
     #   huggingface-hub
@@ -466,7 +450,7 @@ tqdm==4.66.4
     #   transformers
 transformers==4.40.2
     # via pipecat-ai (pyproject.toml)
-triton==2.3.1
+triton==3.0.0
     # via torch
 typer==0.12.3
     # via fastapi-cli
@@ -478,6 +462,7 @@ typing-extensions==4.12.2
     #   fastapi
     #   google-generativeai
     #   huggingface-hub
+    #   langchain-core
     #   openai
     #   pipecat-ai (pyproject.toml)
     #   pydantic
@@ -489,23 +474,22 @@ typing-extensions==4.12.2
     #   uvicorn
 typing-inspect==0.9.0
     # via dataclasses-json
-ujson==5.10.0
-    # via fastapi
 uritemplate==4.1.1
     # via google-api-python-client
 urllib3==2.2.2
     # via requests
-uvicorn[standard]==0.30.1
-    # via fastapi
+uvicorn[standard]==0.30.5
+    # via
+    #   fastapi
+    #   fastapi-cli
 uvloop==0.19.0
     # via uvicorn
 verboselogs==1.7
     # via deepgram-sdk
-watchfiles==0.22.0
+watchfiles==0.23.0
     # via uvicorn
 websockets==12.0
     # via
-    #   cartesia
     #   deepgram-sdk
     #   pipecat-ai (pyproject.toml)
     #   uvicorn

macos-py3.10-requirements.txt

@@ -8,7 +8,6 @@ aiofiles==24.1.0
     # via deepgram-sdk
 aiohttp==3.9.5
     # via
-    #   cartesia
     #   deepgram-sdk
     #   langchain
     #   langchain-community
@@ -36,17 +35,15 @@ attrs==23.2.0
     # via
     #   aiohttp
     #   openpipe
-av==12.2.0
+av==12.3.0
     # via faster-whisper
 azure-cognitiveservices-speech==1.38.0
     # via pipecat-ai (pyproject.toml)
 blinker==1.8.2
     # via flask
-cachetools==5.3.3
+cachetools==5.4.0
     # via google-auth
-cartesia==1.0.3
-    # via pipecat-ai (pyproject.toml)
-certifi==2024.6.2
+certifi==2024.7.4
     # via
     #   httpcore
     #   httpx
@@ -80,15 +77,13 @@ einops==0.8.0
     # via pipecat-ai (pyproject.toml)
 email-validator==2.2.0
     # via fastapi
-exceptiongroup==1.2.1
-    # via
-    #   anyio
-    #   pytest
+exceptiongroup==1.2.2
+    # via anyio
 fal-client==0.4.1
     # via pipecat-ai (pyproject.toml)
-fastapi==0.111.0
+fastapi==0.111.1
     # via pipecat-ai (pyproject.toml)
-fastapi-cli==0.0.4
+fastapi-cli==0.0.5
     # via fastapi
 faster-whisper==1.0.3
     # via pipecat-ai (pyproject.toml)
@@ -123,9 +118,9 @@ google-api-core[grpc]==2.19.1
     #   google-ai-generativelanguage
     #   google-api-python-client
     #   google-generativeai
-google-api-python-client==2.135.0
+google-api-python-client==2.140.0
     # via google-generativeai
-google-auth==2.31.0
+google-auth==2.33.0
     # via
     #   google-ai-generativelanguage
     #   google-api-core
@@ -134,18 +129,18 @@ google-auth==2.31.0
     #   google-generativeai
 google-auth-httplib2==0.2.0
     # via google-api-python-client
-google-generativeai==0.7.1
+google-generativeai==0.7.2
     # via pipecat-ai (pyproject.toml)
 googleapis-common-protos==1.63.2
     # via
     #   google-api-core
     #   grpcio-status
-grpcio==1.64.1
+grpcio==1.65.4
     # via
     #   google-api-core
     #   grpcio-status
     #   pyht
-grpcio-status==1.62.2
+grpcio-status==1.62.3
     # via google-api-core
 h11==0.14.0
     # via
@@ -162,7 +157,6 @@ httptools==0.6.1
 httpx==0.27.0
     # via
     #   anthropic
-    #   cartesia
     #   deepgram-sdk
     #   fal-client
     #   fastapi
@@ -170,7 +164,7 @@ httpx==0.27.0
     #   openpipe
 httpx-sse==0.4.0
     # via fal-client
-huggingface-hub==0.23.4
+huggingface-hub==0.24.5
     # via
     #   faster-whisper
     #   timm
@@ -185,8 +179,6 @@ idna==3.7
     #   httpx
     #   requests
     #   yarl
-iniconfig==2.0.0
-    # via pytest
 itsdangerous==2.2.0
     # via flask
 jinja2==3.1.4
@@ -200,23 +192,23 @@ jsonpatch==1.33
     # via langchain-core
 jsonpointer==3.0.0
     # via jsonpatch
-langchain==0.2.6
+langchain==0.2.12
     # via
     #   langchain-community
     #   pipecat-ai (pyproject.toml)
-langchain-community==0.2.6
+langchain-community==0.2.11
     # via pipecat-ai (pyproject.toml)
-langchain-core==0.2.10
+langchain-core==0.2.29
     # via
     #   langchain
     #   langchain-community
     #   langchain-openai
     #   langchain-text-splitters
-langchain-openai==0.1.10
+langchain-openai==0.1.20
     # via pipecat-ai (pyproject.toml)
 langchain-text-splitters==0.2.2
     # via langchain
-langsmith==0.1.83
+langsmith==0.1.98
     # via
     #   langchain
     #   langchain-community
@@ -261,37 +253,34 @@ numpy==1.26.4
     #   torchvision
     #   transformers
 onnxruntime==1.18.1
-    # via faster-whisper
-openai==1.27.0
+    # via
+    #   faster-whisper
+    #   silero-vad
+openai==1.35.15
     # via
     #   langchain-openai
     #   openpipe
     #   pipecat-ai (pyproject.toml)
-openpipe==4.16.0
+openpipe==4.18.0
     # via pipecat-ai (pyproject.toml)
-orjson==3.10.5
-    # via
-    #   fastapi
-    #   langsmith
+orjson==3.10.7
+    # via langsmith
 packaging==24.1
     # via
     #   huggingface-hub
     #   langchain-core
     #   marshmallow
     #   onnxruntime
-    #   pytest
     #   transformers
 pillow==10.3.0
     # via
     #   pipecat-ai (pyproject.toml)
     #   torchvision
-pluggy==1.5.0
-    # via pytest
 proto-plus==1.24.0
     # via
     #   google-ai-generativelanguage
     #   google-api-core
-protobuf==4.25.3
+protobuf==4.25.4
     # via
     #   google-ai-generativelanguage
     #   google-api-core
@@ -310,7 +299,7 @@ pyasn1-modules==0.4.0
     # via google-auth
 pyaudio==0.2.14
     # via pipecat-ai (pyproject.toml)
-pydantic==2.8.0
+pydantic==2.8.2
     # via
     #   anthropic
     #   fastapi
@@ -319,7 +308,7 @@ pydantic==2.8.0
     #   langchain-core
     #   langsmith
     #   openai
-pydantic-core==2.20.0
+pydantic-core==2.20.1
     # via pydantic
 pygments==2.18.0
     # via rich
@@ -329,10 +318,6 @@ pyloudnorm==0.1.1
     # via pipecat-ai (pyproject.toml)
 pyparsing==3.1.2
     # via httplib2
-pytest==8.2.2
-    # via pytest-asyncio
-pytest-asyncio==0.23.7
-    # via cartesia
 python-dateutil==2.9.0.post0
     # via openpipe
 python-dotenv==1.0.1
@@ -341,7 +326,7 @@ python-dotenv==1.0.1
     #   uvicorn
 python-multipart==0.0.9
     # via fastapi
-pyyaml==6.0.1
+pyyaml==6.0.2
     # via
     #   ctranslate2
     #   huggingface-hub
@@ -351,13 +336,12 @@ pyyaml==6.0.1
     #   timm
     #   transformers
     #   uvicorn
-regex==2024.5.15
+regex==2024.7.24
     # via
     #   tiktoken
     #   transformers
 requests==2.32.3
     # via
-    #   cartesia
     #   google-api-core
     #   huggingface-hub
     #   langchain
@@ -372,7 +356,7 @@ rich==13.7.1
     # via typer
 rsa==4.9
     # via google-auth
-safetensors==0.4.3
+safetensors==0.4.4
     # via
     #   timm
     #   transformers
@@ -380,6 +364,8 @@ scipy==1.14.0
     # via pyloudnorm
 shellingham==1.5.4
     # via typer
+silero-vad==5.1
+    # via pipecat-ai (pyproject.toml)
 six==1.16.0
     # via python-dateutil
 sniffio==1.3.1
@@ -388,17 +374,17 @@ sniffio==1.3.1
     #   anyio
     #   httpx
     #   openai
-sqlalchemy==2.0.31
+sqlalchemy==2.0.32
     # via
     #   langchain
     #   langchain-community
 starlette==0.37.2
     # via fastapi
-sympy==1.12.1
+sympy==1.13.1
     # via
     #   onnxruntime
     #   torch
-tenacity==8.4.2
+tenacity==8.5.0
     # via
     #   langchain
     #   langchain-community
@@ -412,19 +398,17 @@ tokenizers==0.19.1
     #   anthropic
     #   faster-whisper
     #   transformers
-tomli==2.0.1
-    # via pytest
-torch==2.3.1
+torch==2.4.0
     # via
-    #   pipecat-ai (pyproject.toml)
+    #   silero-vad
     #   timm
     #   torchaudio
     #   torchvision
-torchaudio==2.3.1
-    # via pipecat-ai (pyproject.toml)
-torchvision==0.18.1
+torchaudio==2.4.0
+    # via silero-vad
+torchvision==0.19.0
     # via timm
-tqdm==4.66.4
+tqdm==4.66.5
     # via
     #   google-generativeai
     #   huggingface-hub
@@ -442,6 +426,7 @@ typing-extensions==4.12.2
     #   fastapi
     #   google-generativeai
     #   huggingface-hub
+    #   langchain-core
     #   openai
     #   pipecat-ai (pyproject.toml)
     #   pydantic
@@ -453,23 +438,22 @@ typing-extensions==4.12.2
     #   uvicorn
 typing-inspect==0.9.0
     # via dataclasses-json
-ujson==5.10.0
-    # via fastapi
 uritemplate==4.1.1
     # via google-api-python-client
 urllib3==2.2.2
     # via requests
-uvicorn[standard]==0.30.1
-    # via fastapi
+uvicorn[standard]==0.30.5
+    # via
+    #   fastapi
+    #   fastapi-cli
 uvloop==0.19.0
     # via uvicorn
 verboselogs==1.7
     # via deepgram-sdk
-watchfiles==0.22.0
+watchfiles==0.23.0
     # via uvicorn
 websockets==12.0
     # via
-    #   cartesia
     #   deepgram-sdk
     #   pipecat-ai (pyproject.toml)
     #   uvicorn

pyproject.toml

@@ -43,14 +43,14 @@ examples = [ "python-dotenv~=1.0.0", "flask~=3.0.3", "flask_cors~=4.0.1" ]
 fal = [ "fal-client~=0.4.1" ]
 gladia = [ "websockets~=12.0" ]
 google = [ "google-generativeai~=0.7.1" ]
-fireworks = [ "openai~=1.27.0" ]
-langchain = [ "langchain~=0.2.6", "langchain-community~=0.2.6", "langchain-openai~=0.1.10" ]
+fireworks = [ "openai~=1.35.0" ]
+langchain = [ "langchain~=0.2.10", "langchain-community~=0.2.9", "langchain-openai~=0.1.17" ]
 local = [ "pyaudio~=0.2.0" ]
 moondream = [ "einops~=0.8.0", "timm~=0.9.16", "transformers~=4.40.2" ]
-openai = [ "openai~=1.27.0" ]
-openpipe = [ "openpipe~=4.16.0" ]
+openai = [ "openai~=1.35.0" ]
+openpipe = [ "openpipe~=4.18.0" ]
 playht = [ "pyht~=0.0.28" ]
-silero = [ "torch~=2.3.1", "torchaudio~=2.3.1" ]
+silero = [ "silero-vad~=5.1" ]
 websocket = [ "websockets~=12.0", "fastapi~=0.111.0" ]
 whisper = [ "faster-whisper~=1.0.3" ]
 xtts = [ "resampy~=0.4.3" ]
@@ -64,3 +64,4 @@ pythonpath = ["src"]
 
 [tool.setuptools_scm]
 local_scheme = "no-local-version"
+fallback_version = "0.0.0-dev"

src/pipecat/frames/frames.py

@@ -101,7 +101,7 @@ class UserImageRawFrame(ImageRawFrame):
 class SpriteFrame(Frame):
     """An animated sprite. Will be shown by the transport if the transport's
     camera is enabled. Will play at the framerate specified in the transport's
-    `fps` constructor parameter.
+    `camera_out_framerate` constructor parameter.
 
     """
     images: List[ImageRawFrame]
@@ -212,14 +212,6 @@ class SystemFrame(Frame):
     pass
 
 
-@dataclass
-class StartFrame(SystemFrame):
-    """This is the first frame that should be pushed down a pipeline."""
-    allow_interruptions: bool = False
-    enable_metrics: bool = False
-    report_only_initial_ttfb: bool = False
-
-
 @dataclass
 class CancelFrame(SystemFrame):
     """Indicates that a pipeline needs to stop right away."""
@@ -278,23 +270,14 @@ class BotInterruptionFrame(SystemFrame):
     pass
 
 
-@dataclass
-class BotSpeakingFrame(SystemFrame):
-    """Emitted by transport outputs while the bot is still speaking. This can be
-    used, for example, to detect when a user is idle. That is, while the bot is
-    speaking we don't want to trigger any user idle timeout since the user might
-    be listening.
-
-    """
-    pass
-
-
 @dataclass
 class MetricsFrame(SystemFrame):
     """Emitted by processor that can compute metrics like latencies.
     """
     ttfb: List[Mapping[str, Any]] | None = None
     processing: List[Mapping[str, Any]] | None = None
+    tokens: List[Mapping[str, Any]] | None = None
+    characters: List[Mapping[str, Any]] | None = None
 
 #
 # Control frames
@@ -306,6 +289,15 @@ class ControlFrame(Frame):
     pass
 
 
+@dataclass
+class StartFrame(ControlFrame):
+    """This is the first frame that should be pushed down a pipeline."""
+    allow_interruptions: bool = False
+    enable_metrics: bool = False
+    enable_usage_metrics: bool = False
+    report_only_initial_ttfb: bool = False
+
+
 @dataclass
 class EndFrame(ControlFrame):
     """Indicates that a pipeline has ended and frame processors and pipelines
@@ -348,6 +340,33 @@ class UserStoppedSpeakingFrame(ControlFrame):
     pass
 
 
+@dataclass
+class BotStartedSpeakingFrame(ControlFrame):
+    """Emitted upstream by transport outputs to indicate the bot started speaking.
+
+    """
+    pass
+
+
+@dataclass
+class BotStoppedSpeakingFrame(ControlFrame):
+    """Emitted upstream by transport outputs to indicate the bot stopped speaking.
+
+    """
+    pass
+
+
+@dataclass
+class BotSpeakingFrame(ControlFrame):
+    """Emitted upstream by transport outputs while the bot is still
+    speaking. This can be used, for example, to detect when a user is idle. That
+    is, while the bot is speaking we don't want to trigger any user idle timeout
+    since the user might be listening.
+
+    """
+    pass
+
+
 @dataclass
 class TTSStartedFrame(ControlFrame):
     """Used to indicate the beginning of a TTS response. Following

src/pipecat/pipeline/pipeline.py

@@ -64,7 +64,7 @@ class Pipeline(BasePipeline):
         services = []
         for p in self._processors:
             if isinstance(p, BasePipeline):
-                services += p.processors_with_metrics()
+                services.extend(p.processors_with_metrics())
             elif p.can_generate_metrics():
                 services.append(p)
         return services

src/pipecat/pipeline/task.py

@@ -21,6 +21,8 @@ from loguru import logger
 class PipelineParams(BaseModel):
     allow_interruptions: bool = False
     enable_metrics: bool = False
+    enable_usage_metrics: bool = False
+    send_initial_empty_metrics: bool = True
     report_only_initial_ttfb: bool = False
 
 
@@ -95,18 +97,21 @@ class PipelineTask:
 
     def _initial_metrics_frame(self) -> MetricsFrame:
         processors = self._pipeline.processors_with_metrics()
-        ttfb = [{"name": p.name, "time": 0.0} for p in processors]
-        processing = [{"name": p.name, "time": 0.0} for p in processors]
+        ttfb = [{"processor": p.name, "value": 0.0} for p in processors]
+        processing = [{"processor": p.name, "value": 0.0} for p in processors]
         return MetricsFrame(ttfb=ttfb, processing=processing)
 
     async def _process_down_queue(self):
         start_frame = StartFrame(
             allow_interruptions=self._params.allow_interruptions,
             enable_metrics=self._params.enable_metrics,
+            enable_usage_metrics=self._params.enable_metrics,
             report_only_initial_ttfb=self._params.report_only_initial_ttfb
         )
         await self._source.process_frame(start_frame, FrameDirection.DOWNSTREAM)
-        await self._source.process_frame(self._initial_metrics_frame(), FrameDirection.DOWNSTREAM)
+
+        if self._params.send_initial_empty_metrics:
+            await self._source.process_frame(self._initial_metrics_frame(), FrameDirection.DOWNSTREAM)
 
         running = True
         should_cleanup = True

src/pipecat/processors/frame_processor.py

@@ -61,6 +61,19 @@ class FrameProcessorMetrics:
         self._start_processing_time = 0
         return MetricsFrame(processing=[processing])
 
+    async def start_llm_usage_metrics(self, tokens: dict):
+        logger.debug(
+            f"{self._name} prompt tokens: {tokens['prompt_tokens']}, completion tokens: {tokens['completion_tokens']}")
+        return MetricsFrame(tokens=[tokens])
+
+    async def start_tts_usage_metrics(self, text: str):
+        characters = {
+            "processor": self._name,
+            "value": len(text),
+        }
+        logger.debug(f"{self._name} usage characters: {characters['value']}")
+        return MetricsFrame(characters=[characters])
+
 
 class FrameProcessor:
 
@@ -80,6 +93,7 @@ class FrameProcessor:
         # Properties
         self._allow_interruptions = False
         self._enable_metrics = False
+        self._enable_usage_metrics = False
         self._report_only_initial_ttfb = False
 
         # Metrics
@@ -93,6 +107,10 @@ class FrameProcessor:
     def metrics_enabled(self):
         return self._enable_metrics
 
+    @property
+    def usage_metrics_enabled(self):
+        return self._enable_usage_metrics
+
     @property
     def report_only_initial_ttfb(self):
         return self._report_only_initial_ttfb
@@ -120,6 +138,18 @@ class FrameProcessor:
             if frame:
                 await self.push_frame(frame)
 
+    async def start_llm_usage_metrics(self, tokens: dict):
+        if self.can_generate_metrics() and self.usage_metrics_enabled:
+            frame = await self._metrics.start_llm_usage_metrics(tokens)
+            if frame:
+                await self.push_frame(frame)
+
+    async def start_tts_usage_metrics(self, text: str):
+        if self.can_generate_metrics() and self.usage_metrics_enabled:
+            frame = await self._metrics.start_tts_usage_metrics(text)
+            if frame:
+                await self.push_frame(frame)
+
     async def stop_all_metrics(self):
         await self.stop_ttfb_metrics()
         await self.stop_processing_metrics()
@@ -145,6 +175,7 @@ class FrameProcessor:
         if isinstance(frame, StartFrame):
             self._allow_interruptions = frame.allow_interruptions
             self._enable_metrics = frame.enable_metrics
+            self._enable_usage_metrics = frame.enable_usage_metrics
             self._report_only_initial_ttfb = frame.report_only_initial_ttfb
         elif isinstance(frame, StartInterruptionFrame):
             await self.stop_all_metrics()

src/pipecat/processors/frameworks/rtvi.py

@@ -7,11 +7,13 @@
 import asyncio
 import dataclasses
 
-from typing import List, Literal, Optional, Type
-from pydantic import BaseModel, ValidationError
+from typing import Any, Awaitable, Callable, Dict, List, Literal, Optional, Type
+from pydantic import PrivateAttr, BaseModel, ValidationError
 
 from pipecat.frames.frames import (
     BotInterruptionFrame,
+    CancelFrame,
+    EndFrame,
     Frame,
     InterimTranscriptionFrame,
     LLMFullResponseEndFrame,
@@ -19,6 +21,7 @@ from pipecat.frames.frames import (
     LLMMessagesAppendFrame,
     LLMMessagesUpdateFrame,
     LLMModelUpdateFrame,
+    MetricsFrame,
     StartFrame,
     SystemFrame,
     TTSSpeakFrame,
@@ -32,62 +35,76 @@ from pipecat.pipeline.pipeline import Pipeline
 from pipecat.processors.aggregators.llm_response import (
     LLMAssistantResponseAggregator, LLMUserResponseAggregator)
 from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
-from pipecat.services.ai_services import AIService
 from pipecat.services.cartesia import CartesiaTTSService
 from pipecat.services.openai import OpenAILLMService, OpenAILLMContext
 from pipecat.transports.base_transport import BaseTransport
 
-DEFAULT_MESSAGES = [
-    {
-        "role": "system",
-        "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
-    }
-]
-
-DEFAULT_MODEL = "llama3-70b-8192"
-
-DEFAULT_VOICE = "79a125e8-cd45-4c13-8a67-188112f4dd22"
+from loguru import logger
 
 
-class RTVILLMConfig(BaseModel):
-    model: Optional[str] = None
-    messages: Optional[List[dict]] = None
+class RTVIServiceOption(BaseModel):
+    name: str
+    handler: Optional[Callable[['RTVIProcessor',
+                                'RTVIServiceOptionConfig'],
+                               Awaitable[None]]] = None
 
 
-class RTVITTSConfig(BaseModel):
-    voice: Optional[str] = None
+class RTVIService(BaseModel):
+    name: str
+    cls: Type[FrameProcessor]
+    options: List[RTVIServiceOption]
+    _options_dict: Dict[str, RTVIServiceOption] = PrivateAttr(default={})
+
+    def model_post_init(self, __context: Any) -> None:
+        self._options_dict = {}
+        for option in self.options:
+            self._options_dict[option.name] = option
+        return super().model_post_init(__context)
+
+#
+# Client -> Pipecat messages.
+#
+
+
+class RTVIServiceOptionConfig(BaseModel):
+    name: str
+    value: Any
+
+
+class RTVIServiceConfig(BaseModel):
+    service: str
+    options: List[RTVIServiceOptionConfig]
 
 
 class RTVIConfig(BaseModel):
-    llm: Optional[RTVILLMConfig] = None
-    tts: Optional[RTVITTSConfig] = None
+    config: List[RTVIServiceConfig]
+    _config_dict: Dict[str, RTVIServiceConfig] = PrivateAttr(default={})
+
+    def model_post_init(self, __context: Any) -> None:
+        self._config_dict = {}
+        for c in self.config:
+            self._config_dict[c.service] = c
+        return super().model_post_init(__context)
 
 
-class RTVISetup(BaseModel):
-    config: Optional[RTVIConfig] = None
-
-
-class RTVILLMMessageData(BaseModel):
+class RTVILLMContextData(BaseModel):
     messages: List[dict]
 
 
-class RTVITTSMessageData(BaseModel):
+class RTVITTSSpeakData(BaseModel):
     text: str
     interrupt: Optional[bool] = False
 
 
-class RTVIMessageData(BaseModel):
-    setup: Optional[RTVISetup] = None
-    config: Optional[RTVIConfig] = None
-    llm: Optional[RTVILLMMessageData] = None
-    tts: Optional[RTVITTSMessageData] = None
-
-
 class RTVIMessage(BaseModel):
-    label: Literal["rtvi"] = "rtvi"
+    label: Literal["rtvi-ai"] = "rtvi-ai"
     type: str
     id: str
-    data: Optional[RTVIMessageData] = None
+    data: Optional[Dict[str, Any]] = None
+
+#
+# Pipecat -> Client responses and messages.
+#
 
 
 class RTVIResponseData(BaseModel):
@@ -96,7 +113,7 @@ class RTVIResponseData(BaseModel):
 
 
 class RTVIResponse(BaseModel):
-    label: Literal["rtvi"] = "rtvi"
+    label: Literal["rtvi-ai"] = "rtvi-ai"
     type: Literal["response"] = "response"
     id: str
     data: RTVIResponseData
@@ -107,7 +124,7 @@ class RTVIErrorData(BaseModel):
 
 
 class RTVIError(BaseModel):
-    label: Literal["rtvi"] = "rtvi"
+    label: Literal["rtvi-ai"] = "rtvi-ai"
     type: Literal["error"] = "error"
     data: RTVIErrorData
 
@@ -117,7 +134,7 @@ class RTVILLMContextMessageData(BaseModel):
 
 
 class RTVILLMContextMessage(BaseModel):
-    label: Literal["rtvi"] = "rtvi"
+    label: Literal["rtvi-ai"] = "rtvi-ai"
     type: Literal["llm-context"] = "llm-context"
     data: RTVILLMContextMessageData
 
@@ -127,13 +144,13 @@ class RTVITTSTextMessageData(BaseModel):
 
 
 class RTVITTSTextMessage(BaseModel):
-    label: Literal["rtvi"] = "rtvi"
+    label: Literal["rtvi-ai"] = "rtvi-ai"
     type: Literal["tts-text"] = "tts-text"
     data: RTVITTSTextMessageData
 
 
 class RTVIBotReady(BaseModel):
-    label: Literal["rtvi"] = "rtvi"
+    label: Literal["rtvi-ai"] = "rtvi-ai"
     type: Literal["bot-ready"] = "bot-ready"
 
 
@@ -145,23 +162,23 @@ class RTVITranscriptionMessageData(BaseModel):
 
 
 class RTVITranscriptionMessage(BaseModel):
-    label: Literal["rtvi"] = "rtvi"
+    label: Literal["rtvi-ai"] = "rtvi-ai"
     type: Literal["user-transcription"] = "user-transcription"
     data: RTVITranscriptionMessageData
 
 
 class RTVIUserStartedSpeakingMessage(BaseModel):
-    label: Literal["rtvi"] = "rtvi"
+    label: Literal["rtvi-ai"] = "rtvi-ai"
     type: Literal["user-started-speaking"] = "user-started-speaking"
 
 
 class RTVIUserStoppedSpeakingMessage(BaseModel):
-    label: Literal["rtvi"] = "rtvi"
+    label: Literal["rtvi-ai"] = "rtvi-ai"
     type: Literal["user-stopped-speaking"] = "user-stopped-speaking"
 
 
 class RTVIJSONCompletion(BaseModel):
-    label: Literal["rtvi"] = "rtvi"
+    label: Literal["rtvi-ai"] = "rtvi-ai"
     type: Literal["json-completion"] = "json-completion"
     data: str
 
@@ -264,59 +281,128 @@ class RTVITTSTextProcessor(FrameProcessor):
             await self.push_frame(TransportMessageFrame(message=message.model_dump(exclude_none=True)))
 
 
+async def handle_llm_model_update(rtvi: 'RTVIProcessor', option: RTVIServiceOptionConfig):
+    frame = LLMModelUpdateFrame(option.value)
+    await rtvi.push_frame(frame)
+
+
+async def handle_llm_messages_update(rtvi: 'RTVIProcessor', option: RTVIServiceOptionConfig):
+    frame = LLMMessagesUpdateFrame(option.value)
+    await rtvi.push_frame(frame)
+
+
+async def handle_tts_voice_update(rtvi: 'RTVIProcessor', option: RTVIServiceOptionConfig):
+    frame = TTSVoiceUpdateFrame(option.value)
+    await rtvi.push_frame(frame)
+
+DEFAULT_LLM_SERVICE = RTVIService(
+    name="llm",
+    cls=OpenAILLMService,
+    options=[
+        RTVIServiceOption(name="model", handler=handle_llm_model_update),
+        RTVIServiceOption(name="messages", handler=handle_llm_messages_update)
+    ])
+
+DEFAULT_TTS_SERVICE = RTVIService(
+    name="tts",
+    cls=CartesiaTTSService,
+    options=[
+        RTVIServiceOption(name="voice_id", handler=handle_tts_voice_update),
+    ])
+
+
 class RTVIProcessor(FrameProcessor):
 
-    def __init__(
-            self,
-            *,
-            transport: BaseTransport,
-            setup: RTVISetup | None = None,
-            llm_api_key: str = "",
-            llm_base_url: str = "https://api.groq.com/openai/v1",
-            tts_api_key: str = "",
-            llm_cls: Type[AIService] = OpenAILLMService,
-            tts_cls: Type[AIService] = CartesiaTTSService):
+    def __init__(self, *, transport: BaseTransport):
         super().__init__()
         self._transport = transport
-        self._setup = setup
-        self._llm_api_key = llm_api_key
-        self._llm_base_url = llm_base_url
-        self._tts_api_key = tts_api_key
-        self._llm_cls = llm_cls
-        self._tts_cls = tts_cls
+        self._config: RTVIConfig | None = None
+        self._ctor_args: Dict[str, Any] = {}
+
         self._start_frame: Frame | None = None
-        self._llm: FrameProcessor | None = None
-        self._tts: FrameProcessor | None = None
         self._pipeline: FrameProcessor | None = None
+        self._first_participant_joined: bool = False
+
+        # Register transport event so we can send a `bot-ready` event (and maybe
+        # others) when the participant joins.
+        transport.add_event_handler(
+            "on_first_participant_joined",
+            self._on_first_participant_joined)
+
+        # Register default services.
+        self._registered_services: Dict[str, RTVIService] = {}
+        self.register_service(DEFAULT_LLM_SERVICE)
+        self.register_service(DEFAULT_TTS_SERVICE)
 
         self._frame_handler_task = self.get_event_loop().create_task(self._frame_handler())
         self._frame_queue = asyncio.Queue()
 
+    def register_service(self, service: RTVIService):
+        self._registered_services[service.name] = service
+
+    def setup_on_start(self, config: RTVIConfig | None, ctor_args: Dict[str, Any]):
+        self._config = config
+        self._ctor_args = ctor_args
+
+    async def update_config(self, config: RTVIConfig):
+        if self._pipeline:
+            await self._handle_config_update(config)
+        self._config = config
+
     async def process_frame(self, frame: Frame, direction: FrameDirection):
         await super().process_frame(frame, direction)
 
-        if isinstance(frame, SystemFrame):
+        # Specific system frames
+        if isinstance(frame, CancelFrame):
+            await self._cancel(frame)
             await self.push_frame(frame, direction)
+        # All other system frames
+        elif isinstance(frame, SystemFrame):
+            await self.push_frame(frame, direction)
+        # Control frames
+        elif isinstance(frame, StartFrame):
+            await self._start(frame)
+            await self._internal_push_frame(frame, direction)
+        elif isinstance(frame, EndFrame):
+            # Push EndFrame before stop(), because stop() waits on the task to
+            # finish and the task finishes when EndFrame is processed.
+            await self._internal_push_frame(frame, direction)
+            await self._stop(frame)
+        # Other frames
         else:
-            await self._frame_queue.put((frame, direction))
-
-        if isinstance(frame, StartFrame):
-            self._start_frame = frame
-            try:
-                await self._handle_setup(self._setup)
-            except Exception as e:
-                await self._send_error(f"unable to setup RTVI: {e}")
+            await self._internal_push_frame(frame, direction)
 
     async def cleanup(self):
+        if self._pipeline:
+            await self._pipeline.cleanup()
+
+    async def _start(self, frame: StartFrame):
+        try:
+            await self._handle_pipeline_setup(frame, self._config)
+        except Exception as e:
+            await self._send_error(f"unable to setup RTVI pipeline: {e}")
+
+    async def _stop(self, frame: EndFrame):
+        await self._frame_handler_task
+
+    async def _cancel(self, frame: CancelFrame):
         self._frame_handler_task.cancel()
         await self._frame_handler_task
 
+    async def _internal_push_frame(
+            self,
+            frame: Frame | None,
+            direction: FrameDirection | None = FrameDirection.DOWNSTREAM):
+        await self._frame_queue.put((frame, direction))
+
     async def _frame_handler(self):
-        while True:
+        running = True
+        while running:
             try:
                 (frame, direction) = await self._frame_queue.get()
                 await self._handle_frame(frame, direction)
                 self._frame_queue.task_done()
+                running = not isinstance(frame, EndFrame)
             except asyncio.CancelledError:
                 break
 
@@ -371,106 +457,106 @@ class RTVIProcessor(FrameProcessor):
         try:
             message = RTVIMessage.model_validate(frame.message)
         except ValidationError as e:
-            await self._send_error(f"invalid message: {e}")
+            await self._send_error(f"Invalid incoming message: {e}")
+            logger.warning(f"Invalid incoming  message: {e}")
             return
 
         try:
             success = True
             error = None
             match message.type:
-                case "setup":
-                    setup = None
-                    if message.data:
-                        setup = message.data.setup
-                    await self._handle_setup(message.id, setup)
                 case "config-update":
-                    await self._handle_config_update(message.data.config)
+                    await self._handle_config_update(RTVIConfig.model_validate(message.data))
                 case "llm-get-context":
                     await self._handle_llm_get_context()
                 case "llm-append-context":
-                    await self._handle_llm_append_context(message.data.llm)
+                    await self._handle_llm_append_context(RTVILLMContextData.model_validate(message.data))
                 case "llm-update-context":
-                    await self._handle_llm_update_context(message.data.llm)
+                    await self._handle_llm_update_context(RTVILLMContextData.model_validate(message.data))
                 case "tts-speak":
-                    await self._handle_tts_speak(message.data.tts)
+                    await self._handle_tts_speak(RTVITTSSpeakData.model_validate(message.data))
                 case "tts-interrupt":
                     await self._handle_tts_interrupt()
                 case _:
                     success = False
-                    error = f"unsupported type {message.type}"
+                    error = f"Unsupported type {message.type}"
 
             await self._send_response(message.id, success, error)
         except ValidationError as e:
-            await self._send_response(message.id, False, f"invalid message: {e}")
+            await self._send_response(message.id, False, f"Invalid incoming message: {e}")
+            logger.warning(f"Invalid incoming  message: {e}")
         except Exception as e:
-            await self._send_response(message.id, False, f"{e}")
+            await self._send_response(message.id, False, f"Exception processing message: {e}")
+            logger.warning(f"Exception processing message: {e}")
 
-    async def _handle_setup(self, setup: RTVISetup | None):
-        model = DEFAULT_MODEL
-        if setup and setup.config and setup.config.llm and setup.config.llm.model:
-            model = setup.config.llm.model
+    async def _handle_pipeline_setup(self, start_frame: StartFrame, config: RTVIConfig | None):
+        # TODO(aleix): We shouldn't need to save this in `self._tma_in`.
+        self._tma_in = LLMUserResponseAggregator()
+        tma_out = LLMAssistantResponseAggregator()
 
-        messages = DEFAULT_MESSAGES
-        if setup and setup.config and setup.config.llm and setup.config.llm.messages:
-            messages = setup.config.llm.messages
+        llm_cls = self._registered_services["llm"].cls
+        llm_args = self._ctor_args["llm"]
+        llm = llm_cls(**llm_args)
 
-        voice = DEFAULT_VOICE
-        if setup and setup.config and setup.config.tts and setup.config.tts.voice:
-            voice = setup.config.tts.voice
-
-        self._tma_in = LLMUserResponseAggregator(messages)
-        self._tma_out = LLMAssistantResponseAggregator(messages)
-
-        self._llm = self._llm_cls(
-            name="LLM",
-            base_url=self._llm_base_url,
-            api_key=self._llm_api_key,
-            model=model)
-
-        self._tts = self._tts_cls(name="TTS", api_key=self._tts_api_key, voice_id=voice)
+        tts_cls = self._registered_services["tts"].cls
+        tts_args = self._ctor_args["tts"]
+        tts = tts_cls(**tts_args)
 
         # TODO-CB: Eventually we'll need to switch the context aggregators to use the
         # OpenAI context frames instead of message frames
-        context = OpenAILLMContext(messages=messages)
-        self._fc = FunctionCaller(context)
+        context = OpenAILLMContext()
+        fc = FunctionCaller(context)
 
-        self._tts_text = RTVITTSTextProcessor()
+        tts_text = RTVITTSTextProcessor()
 
         pipeline = Pipeline([
             self._tma_in,
-            self._llm,
-            self._fc,
-            self._tts,
-            self._tts_text,
-            self._tma_out,
+            llm,
+            fc,
+            tts,
+            tts_text,
+            tma_out,
             self._transport.output(),
         ])
-        self._pipeline = pipeline
 
         parent = self.get_parent()
-        if parent and self._start_frame:
+        if parent:
             parent.link(pipeline)
 
             # We need to initialize the new pipeline with the same settings
             # as the initial one.
-            start_frame = dataclasses.replace(self._start_frame)
+            start_frame = dataclasses.replace(start_frame)
             await self.push_frame(start_frame)
 
-        message = RTVIBotReady()
-        frame = TransportMessageFrame(message=message.model_dump(exclude_none=True))
-        await self.push_frame(frame)
+            # Configure the pipeline
+            if config:
+                await self._handle_config_update(config)
 
-    async def _handle_config_update(self, config: RTVIConfig):
-        # Change voice before LLM updates, so we can hear the new vocie.
-        if config.tts and config.tts.voice:
-            frame = TTSVoiceUpdateFrame(config.tts.voice)
-            await self.push_frame(frame)
-        if config.llm and config.llm.model:
-            frame = LLMModelUpdateFrame(config.llm.model)
-            await self.push_frame(frame)
-        if config.llm and config.llm.messages:
-            frame = LLMMessagesUpdateFrame(config.llm.messages)
-            await self.push_frame(frame)
+            # Send new initial metrics with the new processors
+            processors = parent.processors_with_metrics()
+            processors.extend(pipeline.processors_with_metrics())
+            ttfb = [{"processor": p.name, "value": 0.0} for p in processors]
+            processing = [{"processor": p.name, "value": 0.0} for p in processors]
+            tokens = [{"processor": p.name, "value": {"prompt_tokens": 0,
+                                                      "completion_tokens": 0,
+                                                      "total_tokens": 0}} for p in processors]
+            characters = [{"processor": p.name, "value": 0} for p in processors]
+            await self.push_frame(MetricsFrame(ttfb=ttfb, processing=processing, tokens=tokens, characters=characters))
+
+        self._pipeline = pipeline
+
+        await self._maybe_send_bot_ready()
+
+    async def _handle_config_service(self, config: RTVIServiceConfig):
+        service = self._registered_services[config.service]
+        for option in config.options:
+            handler = service._options_dict[option.name].handler
+            if handler:
+                await handler(self, option)
+
+    async def _handle_config_update(self, data: RTVIConfig):
+        for config in data.config:
+            await self._handle_config_service(config)
 
     async def _handle_llm_get_context(self):
         data = RTVILLMContextMessageData(messages=self._tma_in.messages)
@@ -478,17 +564,17 @@ class RTVIProcessor(FrameProcessor):
         frame = TransportMessageFrame(message=message.model_dump(exclude_none=True))
         await self.push_frame(frame)
 
-    async def _handle_llm_append_context(self, data: RTVILLMMessageData):
+    async def _handle_llm_append_context(self, data: RTVILLMContextData):
         if data and data.messages:
             frame = LLMMessagesAppendFrame(data.messages)
             await self.push_frame(frame)
 
-    async def _handle_llm_update_context(self, data: RTVILLMMessageData):
+    async def _handle_llm_update_context(self, data: RTVILLMContextData):
         if data and data.messages:
             frame = LLMMessagesUpdateFrame(data.messages)
             await self.push_frame(frame)
 
-    async def _handle_tts_speak(self, data: RTVITTSMessageData):
+    async def _handle_tts_speak(self, data: RTVITTSSpeakData):
         if data and data.text:
             if data.interrupt:
                 await self._handle_tts_interrupt()
@@ -498,6 +584,16 @@ class RTVIProcessor(FrameProcessor):
     async def _handle_tts_interrupt(self):
         await self.push_frame(BotInterruptionFrame(), FrameDirection.UPSTREAM)
 
+    async def _on_first_participant_joined(self, transport, participant):
+        self._first_participant_joined = True
+        await self._maybe_send_bot_ready()
+
+    async def _maybe_send_bot_ready(self):
+        if self._pipeline and self._first_participant_joined:
+            message = RTVIBotReady()
+            frame = TransportMessageFrame(message=message.model_dump(exclude_none=True))
+            await self.push_frame(frame)
+
     async def _send_error(self, error: str):
         message = RTVIError(data=RTVIErrorData(message=error))
         frame = TransportMessageFrame(message=message.model_dump(exclude_none=True))
@@ -515,7 +611,7 @@ class RTVIProcessor(FrameProcessor):
             self._pipeline = pipeline
 
             parent = self.get_parent()
-            if parent and self._start_frame:
+            if parent:
                 parent.link(pipeline)
 
         message = RTVIResponse(id=id, data=RTVIResponseData(success=success, error=error))

src/pipecat/processors/gstreamer/pipeline_source.py

@@ -0,0 +1,234 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import asyncio
+
+from pydantic import BaseModel
+
+from pipecat.frames.frames import (
+    AudioRawFrame,
+    CancelFrame,
+    EndFrame,
+    Frame,
+    ImageRawFrame,
+    StartFrame,
+    SystemFrame)
+from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
+
+from loguru import logger
+
+try:
+    import gi
+    gi.require_version('Gst', '1.0')
+    gi.require_version('GstApp', '1.0')
+    from gi.repository import Gst, GstApp
+except ModuleNotFoundError as e:
+    logger.error(f"Exception: {e}")
+    logger.error("In order to use GStreamer processors, you need to install GStreamer in your system`.")
+    raise Exception(f"Missing module: {e}")
+
+
+class GStreamerPipelineSource(FrameProcessor):
+    class OutputParams(BaseModel):
+        video_width: int = 1280
+        video_height: int = 720
+        audio_sample_rate: int = 16000
+        audio_channels: int = 1
+        clock_sync: bool = True
+
+    def __init__(self, *, pipeline: str, out_params: OutputParams = OutputParams(), **kwargs):
+        super().__init__(**kwargs)
+
+        self._out_params = out_params
+
+        Gst.init()
+
+        self._player = Gst.Pipeline.new("player")
+
+        source = Gst.parse_bin_from_description(pipeline, True)
+
+        decodebin = Gst.ElementFactory.make("decodebin", None)
+        decodebin.connect("pad-added", self._decodebin_callback)
+
+        self._player.add(source)
+        self._player.add(decodebin)
+        source.link(decodebin)
+
+        bus = self._player.get_bus()
+        bus.add_signal_watch()
+        bus.connect("message", self._on_gstreamer_message)
+
+        # Create push frame task. This is the task that will push frames in
+        # order. We also guarantee that all frames are pushed in the same task.
+        self._create_push_task()
+
+    async def process_frame(self, frame: Frame, direction: FrameDirection):
+        await super().process_frame(frame, direction)
+
+        # Specific system frames
+        if isinstance(frame, CancelFrame):
+            await self._cancel(frame)
+            await self.push_frame(frame, direction)
+        # All other system frames
+        elif isinstance(frame, SystemFrame):
+            await self.push_frame(frame, direction)
+        # Control frames
+        elif isinstance(frame, StartFrame):
+            await self._start(frame)
+            await self._internal_push_frame(frame, direction)
+        elif isinstance(frame, EndFrame):
+            # Push EndFrame before stop(), because stop() waits on the task to
+            # finish and the task finishes when EndFrame is processed.
+            await self._internal_push_frame(frame, direction)
+            await self._stop(frame)
+        # Other frames
+        else:
+            await self._internal_push_frame(frame, direction)
+
+    async def _start(self, frame: StartFrame):
+        self._player.set_state(Gst.State.PLAYING)
+
+    async def _stop(self, frame: EndFrame):
+        self._player.set_state(Gst.State.NULL)
+        # Wait for the push frame task to finish. It will finish when the
+        # EndFrame is actually processed.
+        await self._push_frame_task
+
+    async def _cancel(self, frame: CancelFrame):
+        self._player.set_state(Gst.State.NULL)
+        # Cancel all the tasks and wait for them to finish.
+        self._push_frame_task.cancel()
+        await self._push_frame_task
+
+    #
+    # Push frames task
+    #
+
+    def _create_push_task(self):
+        loop = self.get_event_loop()
+        self._push_queue = asyncio.Queue()
+        self._push_frame_task = loop.create_task(self._push_frame_task_handler())
+
+    async def _internal_push_frame(
+            self,
+            frame: Frame | None,
+            direction: FrameDirection | None = FrameDirection.DOWNSTREAM):
+        await self._push_queue.put((frame, direction))
+
+    async def _push_frame_task_handler(self):
+        running = True
+        while running:
+            try:
+                (frame, direction) = await self._push_queue.get()
+                await self.push_frame(frame, direction)
+                running = not isinstance(frame, EndFrame)
+                self._push_queue.task_done()
+            except asyncio.CancelledError:
+                break
+
+    #
+    # GStreaner
+    #
+
+    def _on_gstreamer_message(self, bus: Gst.Bus, message: Gst.Message):
+        t = message.type
+        if t == Gst.MessageType.ERROR:
+            err, debug = message.parse_error()
+            logger.error(f"{self} error: {err} : {debug}")
+        return True
+
+    def _decodebin_callback(self, decodebin: Gst.Element, pad: Gst.Pad):
+        caps_string = pad.get_current_caps().to_string()
+        if caps_string.startswith("audio"):
+            self._decodebin_audio(pad)
+        elif caps_string.startswith("video"):
+            self._decodebin_video(pad)
+
+    def _decodebin_audio(self, pad: Gst.Pad):
+        queue_audio = Gst.ElementFactory.make("queue", None)
+        audioconvert = Gst.ElementFactory.make("audioconvert", None)
+        audioresample = Gst.ElementFactory.make("audioresample", None)
+        audiocapsfilter = Gst.ElementFactory.make("capsfilter", None)
+        audiocaps = Gst.Caps.from_string(
+            f"audio/x-raw,format=S16LE,rate={self._out_params.audio_sample_rate},channels={self._out_params.audio_channels},layout=interleaved")
+        audiocapsfilter.set_property("caps", audiocaps)
+        appsink_audio = Gst.ElementFactory.make("appsink", None)
+        appsink_audio.set_property("emit-signals", True)
+        appsink_audio.set_property("sync", self._out_params.clock_sync)
+        appsink_audio.connect("new-sample", self._appsink_audio_new_sample)
+
+        self._player.add(queue_audio)
+        self._player.add(audioconvert)
+        self._player.add(audioresample)
+        self._player.add(audiocapsfilter)
+        self._player.add(appsink_audio)
+        queue_audio.sync_state_with_parent()
+        audioconvert.sync_state_with_parent()
+        audioresample.sync_state_with_parent()
+        audiocapsfilter.sync_state_with_parent()
+        appsink_audio.sync_state_with_parent()
+
+        queue_audio.link(audioconvert)
+        audioconvert.link(audioresample)
+        audioresample.link(audiocapsfilter)
+        audiocapsfilter.link(appsink_audio)
+
+        queue_pad = queue_audio.get_static_pad("sink")
+        pad.link(queue_pad)
+
+    def _decodebin_video(self, pad: Gst.Pad):
+        queue_video = Gst.ElementFactory.make("queue", None)
+        videoconvert = Gst.ElementFactory.make("videoconvert", None)
+        videoscale = Gst.ElementFactory.make("videoscale", None)
+        videocapsfilter = Gst.ElementFactory.make("capsfilter", None)
+        videocaps = Gst.Caps.from_string(
+            f"video/x-raw,format=RGB,width={self._out_params.video_width},height={self._out_params.video_height}")
+        videocapsfilter.set_property("caps", videocaps)
+
+        appsink_video = Gst.ElementFactory.make("appsink", None)
+        appsink_video.set_property("emit-signals", True)
+        appsink_video.set_property("sync", self._out_params.clock_sync)
+        appsink_video.connect("new-sample", self._appsink_video_new_sample)
+
+        self._player.add(queue_video)
+        self._player.add(videoconvert)
+        self._player.add(videoscale)
+        self._player.add(videocapsfilter)
+        self._player.add(appsink_video)
+        queue_video.sync_state_with_parent()
+        videoconvert.sync_state_with_parent()
+        videoscale.sync_state_with_parent()
+        videocapsfilter.sync_state_with_parent()
+        appsink_video.sync_state_with_parent()
+
+        queue_video.link(videoconvert)
+        videoconvert.link(videoscale)
+        videoscale.link(videocapsfilter)
+        videocapsfilter.link(appsink_video)
+
+        queue_pad = queue_video.get_static_pad("sink")
+        pad.link(queue_pad)
+
+    def _appsink_audio_new_sample(self, appsink: GstApp.AppSink):
+        buffer = appsink.pull_sample().get_buffer()
+        (_, info) = buffer.map(Gst.MapFlags.READ)
+        frame = AudioRawFrame(audio=info.data,
+                              sample_rate=self._out_params.audio_sample_rate,
+                              num_channels=self._out_params.audio_channels)
+        asyncio.run_coroutine_threadsafe(self._internal_push_frame(frame), self.get_event_loop())
+        buffer.unmap(info)
+        return Gst.FlowReturn.OK
+
+    def _appsink_video_new_sample(self, appsink: GstApp.AppSink):
+        buffer = appsink.pull_sample().get_buffer()
+        (_, info) = buffer.map(Gst.MapFlags.READ)
+        frame = ImageRawFrame(
+            image=info.data,
+            size=(self._out_params.video_width, self._out_params.video_height),
+            format="RGB")
+        asyncio.run_coroutine_threadsafe(self._internal_push_frame(frame), self.get_event_loop())
+        buffer.unmap(info)
+        return Gst.FlowReturn.OK

src/pipecat/processors/user_idle_processor.py

@@ -8,7 +8,12 @@ import asyncio
 
 from typing import Awaitable, Callable
 
-from pipecat.frames.frames import BotSpeakingFrame, Frame, StartInterruptionFrame, StopInterruptionFrame, SystemFrame
+from pipecat.frames.frames import (
+    BotSpeakingFrame,
+    Frame,
+    SystemFrame,
+    UserStartedSpeakingFrame,
+    UserStoppedSpeakingFrame)
 from pipecat.processors.async_frame_processor import AsyncFrameProcessor
 from pipecat.processors.frame_processor import FrameDirection
 
@@ -47,10 +52,10 @@ class UserIdleProcessor(AsyncFrameProcessor):
             await self.queue_frame(frame, direction)
 
         # We shouldn't call the idle callback if the user or the bot are speaking.
-        if isinstance(frame, StartInterruptionFrame):
+        if isinstance(frame, UserStartedSpeakingFrame):
             self._interrupted = True
             self._idle_event.set()
-        elif isinstance(frame, StopInterruptionFrame):
+        elif isinstance(frame, UserStoppedSpeakingFrame):
             self._interrupted = False
             self._idle_event.set()
         elif isinstance(frame, BotSpeakingFrame):

src/pipecat/serializers/twilio.py

@@ -7,18 +7,25 @@
 import base64
 import json
 
+from pydantic import BaseModel
+
 from pipecat.frames.frames import AudioRawFrame, Frame
 from pipecat.serializers.base_serializer import FrameSerializer
-from pipecat.utils.audio import ulaw_8000_to_pcm_16000, pcm_16000_to_ulaw_8000
+from pipecat.utils.audio import ulaw_to_pcm, pcm_to_ulaw
 
 
 class TwilioFrameSerializer(FrameSerializer):
+    class InputParams(BaseModel):
+        twilio_sample_rate: int = 8000
+        sample_rate: int = 16000
+
     SERIALIZABLE_TYPES = {
         AudioRawFrame: "audio",
     }
 
-    def __init__(self, stream_sid: str):
+    def __init__(self, stream_sid: str, params: InputParams = InputParams()):
         self._stream_sid = stream_sid
+        self._params = params
 
     def serialize(self, frame: Frame) -> str | bytes | None:
         if not isinstance(frame, AudioRawFrame):
@@ -26,7 +33,7 @@ class TwilioFrameSerializer(FrameSerializer):
 
         data = frame.audio
 
-        serialized_data = pcm_16000_to_ulaw_8000(data)
+        serialized_data = pcm_to_ulaw(data, frame.sample_rate, self._params.twilio_sample_rate)
         payload = base64.b64encode(serialized_data).decode("utf-8")
         answer = {
             "event": "media",
@@ -47,6 +54,12 @@ class TwilioFrameSerializer(FrameSerializer):
             payload_base64 = message["media"]["payload"]
             payload = base64.b64decode(payload_base64)
 
-            deserialized_data = ulaw_8000_to_pcm_16000(payload)
-            audio_frame = AudioRawFrame(audio=deserialized_data, num_channels=1, sample_rate=16000)
+            deserialized_data = ulaw_to_pcm(
+                payload,
+                self._params.twilio_sample_rate,
+                self._params.sample_rate)
+            audio_frame = AudioRawFrame(
+                audio=deserialized_data,
+                num_channels=1,
+                sample_rate=self._params.sample_rate)
             return audio_frame

src/pipecat/services/ai_services.py

@@ -75,10 +75,11 @@ class AIService(FrameProcessor):
 
     async def process_generator(self, generator: AsyncGenerator[Frame, None]):
         async for f in generator:
-            if isinstance(f, ErrorFrame):
-                await self.push_error(f)
-            else:
-                await self.push_frame(f)
+            if f:
+                if isinstance(f, ErrorFrame):
+                    await self.push_error(f)
+                else:
+                    await self.push_frame(f)
 
 
 class AsyncAIService(AsyncFrameProcessor):
@@ -283,14 +284,17 @@ class STTService(AIService):
             await self.stop_processing_metrics()
             (self._content, self._wave) = self._new_wave()
 
+    async def stop(self, frame: EndFrame):
+        self._wave.close()
+
+    async def cancel(self, frame: CancelFrame):
+        self._wave.close()
+
     async def process_frame(self, frame: Frame, direction: FrameDirection):
         """Processes a frame of audio data, either buffering or transcribing it."""
         await super().process_frame(frame, direction)
 
-        if isinstance(frame, CancelFrame) or isinstance(frame, EndFrame):
-            self._wave.close()
-            await self.push_frame(frame, direction)
-        elif isinstance(frame, AudioRawFrame):
+        if isinstance(frame, AudioRawFrame):
             # In this service we accumulate audio internally and at the end we
             # push a TextFrame. We don't really want to push audio frames down.
             await self._append_audio(frame)

src/pipecat/services/azure.py

@@ -18,6 +18,7 @@ from pipecat.frames.frames import (
     EndFrame,
     ErrorFrame,
     Frame,
+    MetricsFrame,
     StartFrame,
     SystemFrame,
     TranscriptionFrame,
@@ -103,6 +104,7 @@ class AzureTTSService(TTSService):
         result = await asyncio.to_thread(self._speech_synthesizer.speak_ssml, (ssml))
 
         if result.reason == ResultReason.SynthesizingAudioCompleted:
+            await self.start_tts_usage_metrics(text)
             await self.stop_ttfb_metrics()
             # Azure always sends a 44-byte header. Strip it off.
             yield AudioRawFrame(audio=result.audio_data[44:], sample_rate=16000, num_channels=1)
@@ -147,13 +149,16 @@ class AzureSTTService(AsyncAIService):
             await self._push_queue.put((frame, direction))
 
     async def start(self, frame: StartFrame):
+        await super().start(frame)
         self._speech_recognizer.start_continuous_recognition_async()
 
     async def stop(self, frame: EndFrame):
+        await super().stop(frame)
         self._speech_recognizer.stop_continuous_recognition_async()
         self._audio_stream.close()
 
     async def cancel(self, frame: CancelFrame):
+        await super().cancel(frame)
         self._speech_recognizer.stop_continuous_recognition_async()
         self._audio_stream.close()
 
@@ -168,11 +173,11 @@ class AzureImageGenServiceREST(ImageGenService):
     def __init__(
         self,
         *,
-        aiohttp_session: aiohttp.ClientSession,
         image_size: str,
         api_key: str,
         endpoint: str,
         model: str,
+        aiohttp_session: aiohttp.ClientSession,
         api_version="2023-06-01-preview",
     ):
         super().__init__()
@@ -181,8 +186,8 @@ class AzureImageGenServiceREST(ImageGenService):
         self._azure_endpoint = endpoint
         self._api_version = api_version
         self._model = model
-        self._aiohttp_session = aiohttp_session
         self._image_size = image_size
+        self._aiohttp_session = aiohttp_session
 
     async def run_image_gen(self, prompt: str) -> AsyncGenerator[Frame, None]:
         url = f"{self._azure_endpoint}openai/images/generations:submit?api-version={self._api_version}"

src/pipecat/services/cartesia.py

@@ -14,12 +14,14 @@ from typing import AsyncGenerator
 
 from pipecat.processors.frame_processor import FrameDirection
 from pipecat.frames.frames import (
+    CancelFrame,
     Frame,
     AudioRawFrame,
     StartInterruptionFrame,
     StartFrame,
     EndFrame,
     TextFrame,
+    MetricsFrame,
     LLMFullResponseEndFrame
 )
 from pipecat.services.ai_services import TTSService
@@ -98,6 +100,10 @@ class CartesiaTTSService(TTSService):
         await super().stop(frame)
         await self._disconnect()
 
+    async def cancel(self, frame: CancelFrame):
+        await super().cancel(frame)
+        await self._disconnect()
+
     async def _connect(self):
         try:
             self._websocket = await websockets.connect(
@@ -111,6 +117,8 @@ class CartesiaTTSService(TTSService):
 
     async def _disconnect(self):
         try:
+            await self.stop_all_metrics()
+
             if self._context_appending_task:
                 self._context_appending_task.cancel()
                 await self._context_appending_task
@@ -120,13 +128,12 @@ class CartesiaTTSService(TTSService):
                 await self._receive_task
                 self._receive_task = None
             if self._websocket:
-                ws = self._websocket
+                await self._websocket.close()
                 self._websocket = None
-                await ws.close()
+
             self._context_id = None
             self._context_id_start_timestamp = None
             self._timestamped_words_buffer = []
-            await self.stop_all_metrics()
         except Exception as e:
             logger.exception(f"{self} error closing websocket: {e}")
 
@@ -142,13 +149,13 @@ class CartesiaTTSService(TTSService):
         try:
             async for message in self._websocket:
                 msg = json.loads(message)
-                # logger.debug(f"Received message: {msg['type']} {msg['context_id']}")
                 if not msg or msg["context_id"] != self._context_id:
                     continue
                 if msg["type"] == "done":
                     await self.stop_ttfb_metrics()
-                    # unset _context_id but not the _context_id_start_timestamp because we are likely still
-                    # playing out audio and need the timestamp to set send context frames
+                    # Unset _context_id but not the _context_id_start_timestamp
+                    # because we are likely still playing out audio and need the
+                    # timestamp to set send context frames.
                     self._context_id = None
                     self._timestamped_words_buffer.append(("LLMFullResponseEndFrame", 0))
                 elif msg["type"] == "timestamps":
@@ -166,6 +173,8 @@ class CartesiaTTSService(TTSService):
                         num_channels=1
                     )
                     await self.push_frame(frame)
+        except asyncio.CancelledError:
+            pass
         except Exception as e:
             logger.exception(f"{self} exception: {e}")
 
@@ -176,15 +185,17 @@ class CartesiaTTSService(TTSService):
                 if not self._context_id_start_timestamp:
                     continue
                 elapsed_seconds = time.time() - self._context_id_start_timestamp
-                # pop all words from self._timestamped_words_buffer that are older than the
-                # elapsed time and print a message about them to the console
+                # Pop all words from self._timestamped_words_buffer that are
+                # older than the elapsed time and print a message about them to
+                # the console.
                 while self._timestamped_words_buffer and self._timestamped_words_buffer[0][1] <= elapsed_seconds:
                     word, timestamp = self._timestamped_words_buffer.pop(0)
                     if word == "LLMFullResponseEndFrame" and timestamp == 0:
                         await self.push_frame(LLMFullResponseEndFrame())
                         continue
-                    # print(f"Word '{word}' with timestamp {timestamp:.2f}s has been spoken.")
                     await self.push_frame(TextFrame(word))
+        except asyncio.CancelledError:
+            pass
         except Exception as e:
             logger.exception(f"{self} exception: {e}")
 
@@ -212,9 +223,9 @@ class CartesiaTTSService(TTSService):
                 "language": self._language,
                 "add_timestamps": True,
             }
-            # logger.debug(f"SENDING MESSAGE {json.dumps(msg)}")
             try:
                 await self._websocket.send(json.dumps(msg))
+                await self.start_tts_usage_metrics(text)
             except Exception as e:
                 logger.exception(f"{self} error sending message: {e}")
                 await self._disconnect()

src/pipecat/services/deepgram.py

@@ -5,7 +5,6 @@
 #
 
 import aiohttp
-import time
 
 from typing import AsyncGenerator
 
@@ -16,14 +15,17 @@ from pipecat.frames.frames import (
     ErrorFrame,
     Frame,
     InterimTranscriptionFrame,
+    MetricsFrame,
     StartFrame,
     SystemFrame,
     TranscriptionFrame)
 from pipecat.processors.frame_processor import FrameDirection
 from pipecat.services.ai_services import AsyncAIService, TTSService
+from pipecat.utils.time import time_now_iso8601
 
 from loguru import logger
 
+
 # See .env.example for Deepgram configuration needed
 try:
     from deepgram import (
@@ -44,17 +46,21 @@ class DeepgramTTSService(TTSService):
     def __init__(
             self,
             *,
-            aiohttp_session: aiohttp.ClientSession,
             api_key: str,
+            aiohttp_session: aiohttp.ClientSession,
             voice: str = "aura-helios-en",
             base_url: str = "https://api.deepgram.com/v1/speak",
+            sample_rate: int = 16000,
+            encoding: str = "linear16",
             **kwargs):
         super().__init__(**kwargs)
 
         self._voice = voice
         self._api_key = api_key
-        self._aiohttp_session = aiohttp_session
         self._base_url = base_url
+        self._sample_rate = sample_rate
+        self._encoding = encoding
+        self._aiohttp_session = aiohttp_session
 
     def can_generate_metrics(self) -> bool:
         return True
@@ -67,7 +73,7 @@ class DeepgramTTSService(TTSService):
         logger.debug(f"Generating TTS: [{text}]")
 
         base_url = self._base_url
-        request_url = f"{base_url}?model={self._voice}&encoding=linear16&container=none&sample_rate=16000"
+        request_url = f"{base_url}?model={self._voice}&encoding={self._encoding}&container=none&sample_rate={self._sample_rate}"
         headers = {"authorization": f"token {self._api_key}"}
         body = {"text": text}
 
@@ -88,9 +94,11 @@ class DeepgramTTSService(TTSService):
                     yield ErrorFrame(f"Error getting audio (status: {r.status}, error: {response_text})")
                     return
 
+                await self.start_tts_usage_metrics(text)
+
                 async for data in r.content:
                     await self.stop_ttfb_metrics()
-                    frame = AudioRawFrame(audio=data, sample_rate=16000, num_channels=1)
+                    frame = AudioRawFrame(audio=data, sample_rate=self._sample_rate, num_channels=1)
                     yield frame
         except Exception as e:
             logger.exception(f"{self} exception: {e}")
@@ -131,15 +139,18 @@ class DeepgramSTTService(AsyncAIService):
             await self.queue_frame(frame, direction)
 
     async def start(self, frame: StartFrame):
+        await super().start(frame)
         if await self._connection.start(self._live_options):
             logger.debug(f"{self}: Connected to Deepgram")
         else:
             logger.error(f"{self}: Unable to connect to Deepgram")
 
     async def stop(self, frame: EndFrame):
+        await super().stop(frame)
         await self._connection.finish()
 
     async def cancel(self, frame: CancelFrame):
+        await super().cancel(frame)
         await self._connection.finish()
 
     async def _on_message(self, *args, **kwargs):
@@ -148,6 +159,6 @@ class DeepgramSTTService(AsyncAIService):
         transcript = result.channel.alternatives[0].transcript
         if len(transcript) > 0:
             if is_final:
-                await self.queue_frame(TranscriptionFrame(transcript, "", int(time.time_ns() / 1000000)))
+                await self.queue_frame(TranscriptionFrame(transcript, "", time_now_iso8601()))
             else:
-                await self.queue_frame(InterimTranscriptionFrame(transcript, "", int(time.time_ns() / 1000000)))
+                await self.queue_frame(InterimTranscriptionFrame(transcript, "", time_now_iso8601()))

src/pipecat/services/elevenlabs.py

@@ -6,30 +6,35 @@
 
 import aiohttp
 
-from typing import AsyncGenerator
+from typing import AsyncGenerator, Literal
+from pydantic import BaseModel
 
-from pipecat.frames.frames import AudioRawFrame, ErrorFrame, Frame
+from pipecat.frames.frames import AudioRawFrame, ErrorFrame, Frame, MetricsFrame
 from pipecat.services.ai_services import TTSService
 
 from loguru import logger
 
 
 class ElevenLabsTTSService(TTSService):
+    class InputParams(BaseModel):
+        output_format: Literal["pcm_16000", "pcm_22050", "pcm_24000", "pcm_44100"] = "pcm_16000"
 
     def __init__(
             self,
             *,
-            aiohttp_session: aiohttp.ClientSession,
             api_key: str,
             voice_id: str,
-            model: str = "eleven_turbo_v2",
+            aiohttp_session: aiohttp.ClientSession,
+            model: str = "eleven_turbo_v2_5",
+            params: InputParams = InputParams(),
             **kwargs):
         super().__init__(**kwargs)
 
         self._api_key = api_key
         self._voice_id = voice_id
-        self._aiohttp_session = aiohttp_session
         self._model = model
+        self._params = params
+        self._aiohttp_session = aiohttp_session
 
     def can_generate_metrics(self) -> bool:
         return True
@@ -46,8 +51,8 @@ class ElevenLabsTTSService(TTSService):
         payload = {"text": text, "model_id": self._model}
 
         querystring = {
-            "output_format": "pcm_16000",
-            "optimize_streaming_latency": 2}
+            "output_format": self._params.output_format
+        }
 
         headers = {
             "xi-api-key": self._api_key,
@@ -63,6 +68,8 @@ class ElevenLabsTTSService(TTSService):
                 yield ErrorFrame(f"Error getting audio (status: {r.status}, error: {text})")
                 return
 
+            await self.start_tts_usage_metrics(text)
+
             async for chunk in r.content:
                 if len(chunk) > 0:
                     await self.stop_ttfb_metrics()

src/pipecat/services/fal.py

@@ -39,8 +39,8 @@ class FalImageGenService(ImageGenService):
     def __init__(
         self,
         *,
-        aiohttp_session: aiohttp.ClientSession,
         params: InputParams,
+        aiohttp_session: aiohttp.ClientSession,
         model: str = "fal-ai/fast-sdxl",
         key: str | None = None,
     ):

src/pipecat/services/gladia.py

@@ -6,7 +6,6 @@
 
 import base64
 import json
-import time
 
 from typing import Optional
 from pydantic.main import BaseModel
@@ -22,6 +21,7 @@ from pipecat.frames.frames import (
     TranscriptionFrame)
 from pipecat.processors.frame_processor import FrameDirection
 from pipecat.services.ai_services import AsyncAIService
+from pipecat.utils.time import time_now_iso8601
 
 from loguru import logger
 
@@ -68,14 +68,17 @@ class GladiaSTTService(AsyncAIService):
             await self.queue_frame(frame, direction)
 
     async def start(self, frame: StartFrame):
+        await super().start(frame)
         self._websocket = await websockets.connect(self._url)
         self._receive_task = self.get_event_loop().create_task(self._receive_task_handler())
         await self._setup_gladia()
 
     async def stop(self, frame: EndFrame):
+        await super().stop(frame)
         await self._websocket.close()
 
     async def cancel(self, frame: CancelFrame):
+        await super().cancel(frame)
         await self._websocket.close()
 
     async def _setup_gladia(self):
@@ -110,6 +113,6 @@ class GladiaSTTService(AsyncAIService):
                 transcript = utterance["transcription"]
                 if confidence >= self._confidence:
                     if type == "final":
-                        await self.queue_frame(TranscriptionFrame(transcript, "", int(time.time_ns() / 1000000)))
+                        await self.queue_frame(TranscriptionFrame(transcript, "", time_now_iso8601()))
                     else:
-                        await self.queue_frame(InterimTranscriptionFrame(transcript, "", int(time.time_ns() / 1000000)))
+                        await self.queue_frame(InterimTranscriptionFrame(transcript, "", time_now_iso8601()))

src/pipecat/services/openai.py

@@ -23,6 +23,7 @@ from pipecat.frames.frames import (
     LLMFullResponseStartFrame,
     LLMMessagesFrame,
     LLMModelUpdateFrame,
+    MetricsFrame,
     TextFrame,
     URLImageRawFrame,
     VisionImageRawFrame
@@ -95,6 +96,7 @@ class BaseOpenAILLMService(LLMService):
             messages=messages,
             tools=context.tools,
             tool_choice=context.tool_choice,
+            stream_options={"include_usage": True}
         )
         return chunks
 
@@ -132,6 +134,15 @@ class BaseOpenAILLMService(LLMService):
         )
 
         async for chunk in chunk_stream:
+            if chunk.usage:
+                tokens = {
+                    "processor": self.name,
+                    "prompt_tokens": chunk.usage.prompt_tokens,
+                    "completion_tokens": chunk.usage.completion_tokens,
+                    "total_tokens": chunk.usage.total_tokens
+                }
+                await self.start_llm_usage_metrics(tokens)
+
             if len(chunk.choices) == 0:
                 continue
 
@@ -253,9 +264,9 @@ class OpenAIImageGenService(ImageGenService):
     def __init__(
         self,
         *,
-        image_size: Literal["256x256", "512x512", "1024x1024", "1792x1024", "1024x1792"],
-        aiohttp_session: aiohttp.ClientSession,
         api_key: str,
+        aiohttp_session: aiohttp.ClientSession,
+        image_size: Literal["256x256", "512x512", "1024x1024", "1792x1024", "1024x1792"],
         model: str = "dall-e-3",
     ):
         super().__init__()
@@ -323,7 +334,6 @@ class OpenAITTSService(TTSService):
 
     async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]:
         logger.debug(f"Generating TTS: [{text}]")
-
         try:
             await self.start_ttfb_metrics()
 
@@ -339,6 +349,9 @@ class OpenAITTSService(TTSService):
                         f"{self} error getting audio (status: {r.status_code}, error: {error})")
                     yield ErrorFrame(f"Error getting audio (status: {r.status_code}, error: {error})")
                     return
+
+                await self.start_tts_usage_metrics(text)
+
                 async for chunk in r.iter_bytes(8192):
                     if len(chunk) > 0:
                         await self.stop_ttfb_metrics()

src/pipecat/services/playht.py

@@ -9,7 +9,7 @@ import struct
 
 from typing import AsyncGenerator
 
-from pipecat.frames.frames import AudioRawFrame, Frame
+from pipecat.frames.frames import AudioRawFrame, Frame, MetricsFrame
 from pipecat.services.ai_services import TTSService
 
 from loguru import logger
@@ -60,6 +60,8 @@ class PlayHTTTSService(TTSService):
                 voice_engine="PlayHT2.0-turbo",
                 options=self._options)
 
+            await self.start_tts_usage_metrics(text)
+
             async for chunk in playht_gen:
                 # skip the RIFF header.
                 if in_header:

src/pipecat/services/whisper.py

@@ -7,7 +7,6 @@
 """This module implements Whisper transcription with a locally-downloaded model."""
 
 import asyncio
-import time
 
 from enum import Enum
 from typing_extensions import AsyncGenerator
@@ -16,6 +15,7 @@ import numpy as np
 
 from pipecat.frames.frames import ErrorFrame, Frame, TranscriptionFrame
 from pipecat.services.ai_services import STTService
+from pipecat.utils.time import time_now_iso8601
 
 from loguru import logger
 
@@ -91,4 +91,4 @@ class WhisperSTTService(STTService):
         if text:
             await self.stop_ttfb_metrics()
             logger.debug(f"Transcription: [{text}]")
-            yield TranscriptionFrame(text, "", int(time.time_ns() / 1000000))
+            yield TranscriptionFrame(text, "", time_now_iso8601())

src/pipecat/services/xtts.py

@@ -6,15 +6,13 @@
 
 import aiohttp
 
-from typing import AsyncGenerator
+from typing import Any, AsyncGenerator, Dict
 
-from pipecat.frames.frames import AudioRawFrame, ErrorFrame, Frame
+from pipecat.frames.frames import AudioRawFrame, ErrorFrame, Frame, MetricsFrame, StartFrame
 from pipecat.services.ai_services import TTSService
 
 from loguru import logger
 
-import requests
-
 import numpy as np
 
 try:
@@ -38,28 +36,45 @@ class XTTSService(TTSService):
     def __init__(
             self,
             *,
-            aiohttp_session: aiohttp.ClientSession,
             voice_id: str,
             language: str,
             base_url: str,
+            aiohttp_session: aiohttp.ClientSession,
             **kwargs):
         super().__init__(**kwargs)
 
         self._voice_id = voice_id
         self._language = language
         self._base_url = base_url
+        self._studio_speakers: Dict[str, Any] | None = None
         self._aiohttp_session = aiohttp_session
-        self._studio_speakers = requests.get(self._base_url + "/studio_speakers").json()
 
     def can_generate_metrics(self) -> bool:
         return True
 
+    async def start(self, frame: StartFrame):
+        await super().start(frame)
+        async with self._aiohttp_session.get(self._base_url + "/studio_speakers") as r:
+            if r.status != 200:
+                text = await r.text()
+                logger.error(
+                    f"{self} error getting studio speakers (status: {r.status}, error: {text})")
+                await self.push_error(
+                    ErrorFrame(f"Error error getting studio speakers (status: {r.status}, error: {text})"))
+                return
+            self._studio_speakers = await r.json()
+
     async def set_voice(self, voice: str):
         logger.debug(f"Switching TTS voice to: [{voice}]")
         self._voice_id = voice
 
     async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]:
         logger.debug(f"Generating TTS: [{text}]")
+
+        if not self._studio_speakers:
+            logger.error(f"{self} no studio speakers available")
+            return
+
         embeddings = self._studio_speakers[self._voice_id]
 
         url = self._base_url + "/tts_stream"
@@ -82,6 +97,8 @@ class XTTSService(TTSService):
                 yield ErrorFrame(f"Error getting audio (status: {r.status}, error: {text})")
                 return
 
+            await self.start_tts_usage_metrics(text)
+
             buffer = bytearray()
 
             async for chunk in r.content.iter_chunked(1024):

src/pipecat/transports/base_input.py

@@ -18,6 +18,7 @@ from pipecat.frames.frames import (
     Frame,
     StartInterruptionFrame,
     StopInterruptionFrame,
+    SystemFrame,
     UserStartedSpeakingFrame,
     UserStoppedSpeakingFrame)
 from pipecat.transports.base_transport import TransportParams
@@ -45,12 +46,26 @@ class BaseInputTransport(FrameProcessor):
             self._audio_in_queue = asyncio.Queue()
             self._audio_task = self.get_event_loop().create_task(self._audio_task_handler())
 
-    async def stop(self):
-        # Wait for the task to finish.
+    async def stop(self, frame: EndFrame):
+        # Cancel and wait for the audio input task to finish.
         if self._params.audio_in_enabled or self._params.vad_enabled:
             self._audio_task.cancel()
             await self._audio_task
 
+        # Wait for the push frame task to finish. It will finish when the
+        # EndFrame is actually processed.
+        await self._push_frame_task
+
+    async def cancel(self, frame: CancelFrame):
+        # Cancel all the tasks and wait for them to finish.
+
+        if self._params.audio_in_enabled or self._params.vad_enabled:
+            self._audio_task.cancel()
+            await self._audio_task
+
+        self._push_frame_task.cancel()
+        await self._push_frame_task
+
     def vad_analyzer(self) -> VADAnalyzer | None:
         return self._params.vad_analyzer
 
@@ -62,25 +77,32 @@ class BaseInputTransport(FrameProcessor):
     # Frame processor
     #
 
-    async def cleanup(self):
-        self._push_frame_task.cancel()
-        await self._push_frame_task
-
     async def process_frame(self, frame: Frame, direction: FrameDirection):
         await super().process_frame(frame, direction)
 
+        # Specific system frames
         if isinstance(frame, CancelFrame):
-            await self.stop()
-            # We don't queue a CancelFrame since we want to stop ASAP.
+            await self.cancel(frame)
             await self.push_frame(frame, direction)
+        elif isinstance(frame, BotInterruptionFrame):
+            await self._handle_interruptions(frame, False)
+        elif isinstance(frame, StartInterruptionFrame):
+            await self._start_interruption()
+        elif isinstance(frame, StopInterruptionFrame):
+            await self._stop_interruption()
+        # All other system frames
+        elif isinstance(frame, SystemFrame):
+            await self.push_frame(frame, direction)
+        # Control frames
         elif isinstance(frame, StartFrame):
             await self.start(frame)
             await self._internal_push_frame(frame, direction)
         elif isinstance(frame, EndFrame):
+            # Push EndFrame before stop(), because stop() waits on the task to
+            # finish and the task finishes when EndFrame is processed.
             await self._internal_push_frame(frame, direction)
-            await self.stop()
-        elif isinstance(frame, BotInterruptionFrame):
-            await self._handle_interruptions(frame, False)
+            await self.stop(frame)
+        # Other frames
         else:
             await self._internal_push_frame(frame, direction)
 
@@ -100,10 +122,12 @@ class BaseInputTransport(FrameProcessor):
         await self._push_queue.put((frame, direction))
 
     async def _push_frame_task_handler(self):
-        while True:
+        running = True
+        while running:
             try:
                 (frame, direction) = await self._push_queue.get()
                 await self.push_frame(frame, direction)
+                running = not isinstance(frame, EndFrame)
                 self._push_queue.task_done()
             except asyncio.CancelledError:
                 break
@@ -113,6 +137,9 @@ class BaseInputTransport(FrameProcessor):
     #
 
     async def _start_interruption(self):
+        if not self.interruptions_allowed:
+            return
+
         # Cancel the task. This will stop pushing frames downstream.
         self._push_frame_task.cancel()
         await self._push_frame_task
@@ -124,6 +151,9 @@ class BaseInputTransport(FrameProcessor):
         self._create_push_task()
 
     async def _stop_interruption(self):
+        if not self.interruptions_allowed:
+            return
+
         await self.push_frame(StopInterruptionFrame())
 
     async def _handle_interruptions(self, frame: Frame, push_frame: bool):

src/pipecat/transports/base_output.py

@@ -7,6 +7,7 @@
 
 import asyncio
 import itertools
+import time
 
 from PIL import Image
 from typing import List
@@ -15,6 +16,8 @@ from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
 from pipecat.frames.frames import (
     AudioRawFrame,
     BotSpeakingFrame,
+    BotStartedSpeakingFrame,
+    BotStoppedSpeakingFrame,
     CancelFrame,
     MetricsFrame,
     SpriteFrame,
@@ -25,6 +28,8 @@ from pipecat.frames.frames import (
     StartInterruptionFrame,
     StopInterruptionFrame,
     SystemFrame,
+    TTSStartedFrame,
+    TTSStoppedFrame,
     TransportMessageFrame)
 from pipecat.transports.base_transport import TransportParams
 
@@ -47,6 +52,7 @@ class BaseOutputTransport(FrameProcessor):
         audio_bytes_10ms = int(self._params.audio_out_sample_rate / 100) * \
             self._params.audio_out_channels * 2
         self._audio_chunk_size = audio_bytes_10ms * 2
+        self._audio_buffer = bytearray()
 
         self._stopped_event = asyncio.Event()
 
@@ -60,18 +66,43 @@ class BaseOutputTransport(FrameProcessor):
         self._create_push_task()
 
     async def start(self, frame: StartFrame):
-        # Create media threads queues.
+        # Create camera output queue and task if needed.
         if self._params.camera_out_enabled:
             self._camera_out_queue = asyncio.Queue()
             self._camera_out_task = self.get_event_loop().create_task(self._camera_out_task_handler())
+        # Create audio output queue and task if needed.
+        if self._params.audio_out_enabled and self._params.audio_out_is_live:
+            self._audio_out_queue = asyncio.Queue()
+            self._audio_out_task = self.get_event_loop().create_task(self._audio_out_task_handler())
 
-    async def stop(self):
-        # Wait on the threads to finish.
+    async def stop(self, frame: EndFrame):
+        # Cancel and wait for the camera output task to finish.
         if self._params.camera_out_enabled:
             self._camera_out_task.cancel()
             await self._camera_out_task
 
-        self._stopped_event.set()
+        # Cancel and wait for the audio output task to finish.
+        if self._params.audio_out_enabled and self._params.audio_out_is_live:
+            self._audio_out_task.cancel()
+            await self._audio_out_task
+
+        # Wait for the push frame and sink tasks to finish. They will finish when
+        # the EndFrame is actually processed.
+        await self._push_frame_task
+        await self._sink_task
+
+    async def cancel(self, frame: CancelFrame):
+        # Cancel all the tasks and wait for them to finish.
+
+        if self._params.camera_out_enabled:
+            self._camera_out_task.cancel()
+            await self._camera_out_task
+
+        self._push_frame_task.cancel()
+        await self._push_frame_task
+
+        self._sink_task.cancel()
+        await self._sink_task
 
     async def send_message(self, frame: TransportMessageFrame):
         pass
@@ -89,48 +120,40 @@ class BaseOutputTransport(FrameProcessor):
     # Frame processor
     #
 
-    async def cleanup(self):
-        if self._sink_task:
-            self._sink_task.cancel()
-            await self._sink_task
-
-        self._push_frame_task.cancel()
-        await self._push_frame_task
-
     async def process_frame(self, frame: Frame, direction: FrameDirection):
         await super().process_frame(frame, direction)
 
         #
-        # Out-of-band frames like (CancelFrame or StartInterruptionFrame) are
-        # pushed immediately. Other frames require order so they are put in the
-        # sink queue.
+        # System frames (like StartInterruptionFrame) are pushed
+        # immediately. Other frames require order so they are put in the sink
+        # queue.
         #
-        if isinstance(frame, StartFrame):
-            await self.start(frame)
-            await self.push_frame(frame, direction)
-        # EndFrame is managed in the sink queue handler.
-        elif isinstance(frame, CancelFrame):
-            await self.stop()
+        if isinstance(frame, CancelFrame):
             await self.push_frame(frame, direction)
+            await self.cancel(frame)
         elif isinstance(frame, StartInterruptionFrame) or isinstance(frame, StopInterruptionFrame):
+            await self.push_frame(frame, direction)
             await self._handle_interruptions(frame)
-            await self.push_frame(frame, direction)
         elif isinstance(frame, MetricsFrame):
-            await self.send_metrics(frame)
             await self.push_frame(frame, direction)
+            await self.send_metrics(frame)
         elif isinstance(frame, SystemFrame):
             await self.push_frame(frame, direction)
+        # Control frames.
+        elif isinstance(frame, StartFrame):
+            await self._sink_queue.put(frame)
+            await self.start(frame)
+        elif isinstance(frame, EndFrame):
+            await self._sink_queue.put(frame)
+            await self.stop(frame)
+        # Other frames.
         elif isinstance(frame, AudioRawFrame):
             await self._handle_audio(frame)
+        elif isinstance(frame, ImageRawFrame) or isinstance(frame, SpriteFrame):
+            await self._handle_image(frame)
         else:
             await self._sink_queue.put(frame)
 
-        # If we are finishing, wait here until we have stopped, otherwise we might
-        # close things too early upstream. We need this event because we don't
-        # know when the internal threads will finish.
-        if isinstance(frame, CancelFrame) or isinstance(frame, EndFrame):
-            await self._stopped_event.wait()
-
     async def _handle_interruptions(self, frame: Frame):
         if not self.interruptions_allowed:
             return
@@ -146,11 +169,27 @@ class BaseOutputTransport(FrameProcessor):
             self._create_push_task()
 
     async def _handle_audio(self, frame: AudioRawFrame):
-        audio = frame.audio
-        for i in range(0, len(audio), self._audio_chunk_size):
-            chunk = AudioRawFrame(audio[i: i + self._audio_chunk_size],
-                                  sample_rate=frame.sample_rate, num_channels=frame.num_channels)
-            await self._sink_queue.put(chunk)
+        if not self._params.audio_out_enabled:
+            return
+
+        if self._params.audio_out_is_live:
+            await self._audio_out_queue.put(frame)
+        else:
+            self._audio_buffer.extend(frame.audio)
+            while len(self._audio_buffer) >= self._audio_chunk_size:
+                chunk = AudioRawFrame(bytes(self._audio_buffer[:self._audio_chunk_size]),
+                                      sample_rate=frame.sample_rate, num_channels=frame.num_channels)
+                await self._sink_queue.put(chunk)
+                self._audio_buffer = self._audio_buffer[self._audio_chunk_size:]
+
+    async def _handle_image(self, frame: ImageRawFrame | SpriteFrame):
+        if not self._params.camera_out_enabled:
+            return
+
+        if self._params.camera_out_is_live:
+            await self._camera_out_queue.put(frame)
+        else:
+            await self._sink_queue.put(frame)
 
     def _create_sink_task(self):
         loop = self.get_event_loop()
@@ -158,25 +197,30 @@ class BaseOutputTransport(FrameProcessor):
         self._sink_task = loop.create_task(self._sink_task_handler())
 
     async def _sink_task_handler(self):
-        # Audio accumlation buffer
-        buffer = bytearray()
-        while True:
+        running = True
+        while running:
             try:
                 frame = await self._sink_queue.get()
-                if isinstance(frame, AudioRawFrame) and self._params.audio_out_enabled:
-                    buffer.extend(frame.audio)
-                    buffer = await self._maybe_send_audio(buffer)
-                elif isinstance(frame, ImageRawFrame) and self._params.camera_out_enabled:
+                if isinstance(frame, AudioRawFrame):
+                    await self.write_raw_audio_frames(frame.audio)
+                    await self._internal_push_frame(frame)
+                    await self.push_frame(BotSpeakingFrame(), FrameDirection.UPSTREAM)
+                elif isinstance(frame, ImageRawFrame):
                     await self._set_camera_image(frame)
-                elif isinstance(frame, SpriteFrame) and self._params.camera_out_enabled:
+                elif isinstance(frame, SpriteFrame):
                     await self._set_camera_images(frame.images)
                 elif isinstance(frame, TransportMessageFrame):
                     await self.send_message(frame)
+                elif isinstance(frame, TTSStartedFrame):
+                    await self._internal_push_frame(BotStartedSpeakingFrame(), FrameDirection.UPSTREAM)
+                    await self._internal_push_frame(frame)
+                elif isinstance(frame, TTSStoppedFrame):
+                    await self._internal_push_frame(BotStoppedSpeakingFrame(), FrameDirection.UPSTREAM)
+                    await self._internal_push_frame(frame)
                 else:
                     await self._internal_push_frame(frame)
 
-                if isinstance(frame, EndFrame):
-                    await self.stop()
+                running = not isinstance(frame, EndFrame)
 
                 self._sink_queue.task_done()
             except asyncio.CancelledError:
@@ -200,10 +244,12 @@ class BaseOutputTransport(FrameProcessor):
         await self._push_queue.put((frame, direction))
 
     async def _push_frame_task_handler(self):
-        while True:
+        running = True
+        while running:
             try:
                 (frame, direction) = await self._push_queue.get()
                 await self.push_frame(frame, direction)
+                running = not isinstance(frame, EndFrame)
                 self._push_queue.task_done()
             except asyncio.CancelledError:
                 break
@@ -228,21 +274,20 @@ class BaseOutputTransport(FrameProcessor):
         await self.write_frame_to_camera(frame)
 
     async def _set_camera_image(self, image: ImageRawFrame):
-        if self._params.camera_out_is_live:
-            await self._camera_out_queue.put(image)
-        else:
-            self._camera_images = itertools.cycle([image])
+        self._camera_images = itertools.cycle([image])
 
     async def _set_camera_images(self, images: List[ImageRawFrame]):
         self._camera_images = itertools.cycle(images)
 
     async def _camera_out_task_handler(self):
+        self._camera_out_start_time = None
+        self._camera_out_frame_index = 0
+        self._camera_out_frame_duration = 1 / self._params.camera_out_framerate
+        self._camera_out_frame_reset = self._camera_out_frame_duration * 5
         while True:
             try:
                 if self._params.camera_out_is_live:
-                    image = await self._camera_out_queue.get()
-                    await self._draw_image(image)
-                    self._camera_out_queue.task_done()
+                    await self._camera_out_is_live_handler()
                 elif self._camera_images:
                     image = next(self._camera_images)
                     await self._draw_image(image)
@@ -254,6 +299,31 @@ class BaseOutputTransport(FrameProcessor):
             except Exception as e:
                 logger.exception(f"{self} error writing to camera: {e}")
 
+    async def _camera_out_is_live_handler(self):
+        image = await self._camera_out_queue.get()
+
+        # We get the start time as soon as we get the first image.
+        if not self._camera_out_start_time:
+            self._camera_out_start_time = time.time()
+            self._camera_out_frame_index = 0
+
+        # Calculate how much time we need to wait before rendering next image.
+        real_elapsed_time = time.time() - self._camera_out_start_time
+        real_render_time = self._camera_out_frame_index * self._camera_out_frame_duration
+        delay_time = self._camera_out_frame_duration + real_render_time - real_elapsed_time
+
+        if abs(delay_time) > self._camera_out_frame_reset:
+            self._camera_out_start_time = time.time()
+            self._camera_out_frame_index = 0
+        elif delay_time > 0:
+            await asyncio.sleep(delay_time)
+            self._camera_out_frame_index += 1
+
+        # Render image
+        await self._draw_image(image)
+
+        self._camera_out_queue.task_done()
+
     #
     # Audio out
     #
@@ -261,9 +331,14 @@ class BaseOutputTransport(FrameProcessor):
     async def send_audio(self, frame: AudioRawFrame):
         await self.process_frame(frame, FrameDirection.DOWNSTREAM)
 
-    async def _maybe_send_audio(self, buffer: bytearray) -> bytearray:
-        if len(buffer) >= self._audio_chunk_size:
-            await self.write_raw_audio_frames(bytes(buffer[:self._audio_chunk_size]))
-            buffer = buffer[self._audio_chunk_size:]
-            await self.push_frame(BotSpeakingFrame(), FrameDirection.UPSTREAM)
-        return buffer
+    async def _audio_out_task_handler(self):
+        while True:
+            try:
+                frame = await self._audio_out_queue.get()
+                await self.write_raw_audio_frames(frame.audio)
+                await self._internal_push_frame(frame)
+                await self.push_frame(BotSpeakingFrame(), FrameDirection.UPSTREAM)
+            except asyncio.CancelledError:
+                break
+            except Exception as e:
+                logger.exception(f"{self} error writing to camera: {e}")

src/pipecat/transports/base_transport.py

@@ -29,6 +29,7 @@ class TransportParams(BaseModel):
     camera_out_framerate: int = 30
     camera_out_color_format: str = "RGB"
     audio_out_enabled: bool = False
+    audio_out_is_live: bool = False
     audio_out_sample_rate: int = 16000
     audio_out_channels: int = 1
     audio_in_enabled: bool = False
@@ -60,20 +61,20 @@ class BaseTransport(ABC):
 
     def event_handler(self, event_name: str):
         def decorator(handler):
-            self._add_event_handler(event_name, handler)
+            self.add_event_handler(event_name, handler)
             return handler
         return decorator
 
+    def add_event_handler(self, event_name: str, handler):
+        if event_name not in self._event_handlers:
+            raise Exception(f"Event handler {event_name} not registered")
+        self._event_handlers[event_name].append(handler)
+
     def _register_event_handler(self, event_name: str):
         if event_name in self._event_handlers:
             raise Exception(f"Event handler {event_name} already registered")
         self._event_handlers[event_name] = []
 
-    def _add_event_handler(self, event_name: str, handler):
-        if event_name not in self._event_handlers:
-            raise Exception(f"Event handler {event_name} not registered")
-        self._event_handlers[event_name].append(handler)
-
     async def _call_event_handler(self, event_name: str, *args, **kwargs):
         try:
             for handler in self._event_handlers[event_name]:

src/pipecat/transports/network/fastapi_websocket.py

@@ -12,7 +12,7 @@ import wave
 from typing import Awaitable, Callable
 from pydantic.main import BaseModel
 
-from pipecat.frames.frames import AudioRawFrame, StartFrame
+from pipecat.frames.frames import AudioRawFrame, CancelFrame, EndFrame, StartFrame
 from pipecat.processors.frame_processor import FrameProcessor
 from pipecat.serializers.base_serializer import FrameSerializer
 from pipecat.transports.base_input import BaseInputTransport
@@ -57,14 +57,19 @@ class FastAPIWebsocketInputTransport(BaseInputTransport):
         self._callbacks = callbacks
 
     async def start(self, frame: StartFrame):
-        await self._callbacks.on_client_connected(self._websocket)
         await super().start(frame)
+        await self._callbacks.on_client_connected(self._websocket)
         self._receive_task = self.get_event_loop().create_task(self._receive_messages())
 
-    async def stop(self):
+    async def stop(self, frame: EndFrame):
+        await super().stop(frame)
+        if self._websocket.client_state != WebSocketState.DISCONNECTED:
+            await self._websocket.close()
+
+    async def cancel(self, frame: CancelFrame):
+        await super().cancel(frame)
         if self._websocket.client_state != WebSocketState.DISCONNECTED:
             await self._websocket.close()
-        await super().stop()
 
     async def _receive_messages(self):
         async for message in self._websocket.iter_text():