diff --git a/README.md b/README.md index 554d7b845..9f4535fa7 100644 --- a/README.md +++ b/README.md @@ -6,16 +6,23 @@ [](https://pypi.org/project/pipecat-ai) [](https://discord.gg/pipecat) +[](https://daily-co.github.io/dailyai-docs/docs/intro) -`pipecat` is a framework for building voice (and multimodal) conversational agents. Things like personal coaches, meeting assistants, story-telling toys for kids, customer support bots, and snarky social companions. +`pipecat` is a framework for building voice (and multimodal) conversational agents. Things like personal coaches, meeting assistants, [story-telling toys for kids](https://storytelling-chatbot.fly.dev/), customer support bots, [intake flows](https://www.youtube.com/watch?v=lDevgsp9vn0), and snarky social companions. -Build things like this: +Take a lot at some example apps: -[](https://www.youtube.com/watch?v=lDevgsp9vn0) +
## Getting started with voice agents -You can get started with Pipecat running on your local machine, then move your agent processes to the cloud when you’re ready. You can also add a telephone number, image output, video input, use different LLMs, and more. +You can get started with Pipecat running on your local machine, then move your agent processes to the cloud when you’re ready. You can also add a 📞 telephone number, 🖼️ image output, 📺 video input, use different LLMs, and more. ```shell # install the module @@ -40,14 +47,9 @@ Your project may or may not need these, so they're made available as optional re There are two directories of examples: -- [foundational](https://github.com/pipecat-ai/pipecat/tree/main/examples/foundational) — examples that build on each other, introducing one or two concepts at a time -- [example apps](https://github.com/pipecat-ai/pipecat-examples) — complete applications that you can use as starting points for development +- [foundational](https://github.com/pipecat-ai/pipecat/tree/main/examples/foundational) — small snippets that build on each other, introducing one or two concepts at a time +- [example apps](https://github.com/pipecat-ai/pipecat/tree/main/examples/) — complete applications that you can use as starting points for development -Before running the examples you need to install the dependencies (which will install all the dependencies to run all of the examples): - -``` -pip install -r requirements.txt -``` ## A simple voice agent running locally If you’re doing AI-related stuff, you probably have an OpenAI API key. @@ -57,65 +59,111 @@ To generate voice output, one service that’s easy to get started with is Eleve So let’s run a really simple agent that’s just a GPT-4 prompt, wired up to voice input and speaker output. ```python -TBC +#app.py + +import asyncio +import aiohttp + +from pipecat.frames.frames import EndFrame, TextFrame +from pipecat.pipeline.pipeline import Pipeline +from pipecat.pipeline.task import PipelineTask +from pipecat.pipeline.runner import PipelineRunner +from pipecat.services.elevenlabs import ElevenLabsTTSService +from pipecat.transports.services.daily import DailyParams, DailyTransport + +async def main(): + async with aiohttp.ClientSession() as session: + # Use Daily as a real-time media transport (WebRTC) + daily_url = ... + daily_token = ... + transport = DailyTransport( + daily_url, daily_token, "Bot Name", DailyParams(audio_out_enabled=True)) + + # Use Eleven Labs for Text-to-Speech + tts = ElevenLabsTTSService( + aiohttp_session=session, + api_key=..., + voice_id=..., + ) + + # Simple pipeline that will process tunr text to speech and output the result + pipeline = Pipeline([tts, transport.output()]) + + # Create Pipecat processor that can run one or more pipelines tasks + runner = PipelineRunner() + + # Assign the task callable to run the pipeline + task = PipelineTask(pipeline) + + # Register an event handler to play audio when a + # participant joins the transport WebRTC session + @transport.event_handler("on_participant_joined") + async def on_new_participant_joined(transport, participant): + participant_name = participant["info"]["userName"] or '' + # Queue a TextFrame that will get spoken by the TTS service (Eleven Labs) + await task.queue_frames([TextFrame(f"Hello there, {participant_name}!"), EndFrame()]) + # Run the pipeline task + await runner.run(task) + +if __name__ == "__main__": + asyncio.run(main()) ``` Run it with: ```shell -TBC +python app.py ``` +Daily provides a prebuilt WebRTC user interface. Whilst the app is running, you can visit at `https://
+
+
+
+
+
+
+
+
+This app connects you to a chatbot powered by GPT-4, complete with animations generated by Stable Video Diffusion. The chatbot also has vision powers thanks to [Moondream](https://moondream.ai) so you can ask it, for example, "what do you see?".
+
+ℹ️ The first time, things might take some time to get started since VAD (Voice Activity Detection) and vision models need to be downloaded.
+
+## Get started
+
+```python
+python3 -m venv env
+source env/bin/activate
+pip install -r requirements.txt
+
+cp env.example .env # and add your credentials
+
+```
+
+## Run the server
+
+```bash
+python server.py
+```
+
+Then, visit `http://localhost:7860/start` in your browser to start a chatbot
+session.
+
+## Build and test the Docker image
+
+```
+docker build -t moonbot .
+docker run --env-file .env -p 7860:7860 moonbot
+```
+
+You can try to visit `http://localhost:7860/start` again.
diff --git a/examples/starter-apps/assets/robot01.png b/examples/moondream-chatbot/assets/robot01.png
similarity index 100%
rename from examples/starter-apps/assets/robot01.png
rename to examples/moondream-chatbot/assets/robot01.png
diff --git a/examples/starter-apps/assets/robot010.png b/examples/moondream-chatbot/assets/robot010.png
similarity index 100%
rename from examples/starter-apps/assets/robot010.png
rename to examples/moondream-chatbot/assets/robot010.png
diff --git a/examples/starter-apps/assets/robot011.png b/examples/moondream-chatbot/assets/robot011.png
similarity index 100%
rename from examples/starter-apps/assets/robot011.png
rename to examples/moondream-chatbot/assets/robot011.png
diff --git a/examples/starter-apps/assets/robot012.png b/examples/moondream-chatbot/assets/robot012.png
similarity index 100%
rename from examples/starter-apps/assets/robot012.png
rename to examples/moondream-chatbot/assets/robot012.png
diff --git a/examples/starter-apps/assets/robot013.png b/examples/moondream-chatbot/assets/robot013.png
similarity index 100%
rename from examples/starter-apps/assets/robot013.png
rename to examples/moondream-chatbot/assets/robot013.png
diff --git a/examples/starter-apps/assets/robot014.png b/examples/moondream-chatbot/assets/robot014.png
similarity index 100%
rename from examples/starter-apps/assets/robot014.png
rename to examples/moondream-chatbot/assets/robot014.png
diff --git a/examples/starter-apps/assets/robot015.png b/examples/moondream-chatbot/assets/robot015.png
similarity index 100%
rename from examples/starter-apps/assets/robot015.png
rename to examples/moondream-chatbot/assets/robot015.png
diff --git a/examples/starter-apps/assets/robot016.png b/examples/moondream-chatbot/assets/robot016.png
similarity index 100%
rename from examples/starter-apps/assets/robot016.png
rename to examples/moondream-chatbot/assets/robot016.png
diff --git a/examples/starter-apps/assets/robot017.png b/examples/moondream-chatbot/assets/robot017.png
similarity index 100%
rename from examples/starter-apps/assets/robot017.png
rename to examples/moondream-chatbot/assets/robot017.png
diff --git a/examples/starter-apps/assets/robot018.png b/examples/moondream-chatbot/assets/robot018.png
similarity index 100%
rename from examples/starter-apps/assets/robot018.png
rename to examples/moondream-chatbot/assets/robot018.png
diff --git a/examples/starter-apps/assets/robot019.png b/examples/moondream-chatbot/assets/robot019.png
similarity index 100%
rename from examples/starter-apps/assets/robot019.png
rename to examples/moondream-chatbot/assets/robot019.png
diff --git a/examples/starter-apps/assets/robot02.png b/examples/moondream-chatbot/assets/robot02.png
similarity index 100%
rename from examples/starter-apps/assets/robot02.png
rename to examples/moondream-chatbot/assets/robot02.png
diff --git a/examples/starter-apps/assets/robot020.png b/examples/moondream-chatbot/assets/robot020.png
similarity index 100%
rename from examples/starter-apps/assets/robot020.png
rename to examples/moondream-chatbot/assets/robot020.png
diff --git a/examples/starter-apps/assets/robot021.png b/examples/moondream-chatbot/assets/robot021.png
similarity index 100%
rename from examples/starter-apps/assets/robot021.png
rename to examples/moondream-chatbot/assets/robot021.png
diff --git a/examples/starter-apps/assets/robot022.png b/examples/moondream-chatbot/assets/robot022.png
similarity index 100%
rename from examples/starter-apps/assets/robot022.png
rename to examples/moondream-chatbot/assets/robot022.png
diff --git a/examples/starter-apps/assets/robot023.png b/examples/moondream-chatbot/assets/robot023.png
similarity index 100%
rename from examples/starter-apps/assets/robot023.png
rename to examples/moondream-chatbot/assets/robot023.png
diff --git a/examples/starter-apps/assets/robot024.png b/examples/moondream-chatbot/assets/robot024.png
similarity index 100%
rename from examples/starter-apps/assets/robot024.png
rename to examples/moondream-chatbot/assets/robot024.png
diff --git a/examples/starter-apps/assets/robot025.png b/examples/moondream-chatbot/assets/robot025.png
similarity index 100%
rename from examples/starter-apps/assets/robot025.png
rename to examples/moondream-chatbot/assets/robot025.png
diff --git a/examples/starter-apps/assets/robot03.png b/examples/moondream-chatbot/assets/robot03.png
similarity index 100%
rename from examples/starter-apps/assets/robot03.png
rename to examples/moondream-chatbot/assets/robot03.png
diff --git a/examples/starter-apps/assets/robot04.png b/examples/moondream-chatbot/assets/robot04.png
similarity index 100%
rename from examples/starter-apps/assets/robot04.png
rename to examples/moondream-chatbot/assets/robot04.png
diff --git a/examples/starter-apps/assets/robot05.png b/examples/moondream-chatbot/assets/robot05.png
similarity index 100%
rename from examples/starter-apps/assets/robot05.png
rename to examples/moondream-chatbot/assets/robot05.png
diff --git a/examples/starter-apps/assets/robot06.png b/examples/moondream-chatbot/assets/robot06.png
similarity index 100%
rename from examples/starter-apps/assets/robot06.png
rename to examples/moondream-chatbot/assets/robot06.png
diff --git a/examples/starter-apps/assets/robot07.png b/examples/moondream-chatbot/assets/robot07.png
similarity index 100%
rename from examples/starter-apps/assets/robot07.png
rename to examples/moondream-chatbot/assets/robot07.png
diff --git a/examples/starter-apps/assets/robot08.png b/examples/moondream-chatbot/assets/robot08.png
similarity index 100%
rename from examples/starter-apps/assets/robot08.png
rename to examples/moondream-chatbot/assets/robot08.png
diff --git a/examples/starter-apps/assets/robot09.png b/examples/moondream-chatbot/assets/robot09.png
similarity index 100%
rename from examples/starter-apps/assets/robot09.png
rename to examples/moondream-chatbot/assets/robot09.png
diff --git a/examples/moondream-chatbot/bot.py b/examples/moondream-chatbot/bot.py
new file mode 100644
index 000000000..56d5b72da
--- /dev/null
+++ b/examples/moondream-chatbot/bot.py
@@ -0,0 +1,210 @@
+import asyncio
+
+import aiohttp
+import logging
+import os
+from PIL import Image
+from typing import AsyncGenerator
+
+from dailyai.pipeline.aggregators import (
+ LLMUserResponseAggregator,
+ ParallelPipeline,
+ VisionImageFrameAggregator,
+ SentenceAggregator
+)
+from dailyai.pipeline.frames import (
+ ImageFrame,
+ SpriteFrame,
+ Frame,
+ LLMMessagesFrame,
+ AudioFrame,
+ PipelineStartedFrame,
+ TTSEndFrame,
+ TextFrame,
+ UserImageFrame,
+ UserImageRequestFrame,
+)
+from dailyai.services.moondream_ai_service import MoondreamService
+from dailyai.pipeline.pipeline import FrameProcessor, Pipeline
+from dailyai.transports.daily_transport import DailyTransport
+from dailyai.services.open_ai_services import OpenAILLMService
+from dailyai.services.elevenlabs_ai_service import ElevenLabsTTSService
+
+from runner import configure
+
+from dotenv import load_dotenv
+load_dotenv(override=True)
+
+logging.basicConfig(format=f"%(levelno)s %(asctime)s %(message)s")
+logger = logging.getLogger("dailyai")
+logger.setLevel(logging.DEBUG)
+
+user_request_answer = "Let me take a look."
+
+sprites = []
+
+script_dir = os.path.dirname(__file__)
+
+for i in range(1, 26):
+ # Build the full path to the image file
+ full_path = os.path.join(script_dir, f"assets/robot0{i}.png")
+ # Get the filename without the extension to use as the dictionary key
+ # Open the image and convert it to bytes
+ with Image.open(full_path) as img:
+ sprites.append(img.tobytes())
+
+flipped = sprites[::-1]
+sprites.extend(flipped)
+
+# When the bot isn't talking, show a static image of the cat listening
+quiet_frame = ImageFrame(sprites[0], (1024, 576))
+talking_frame = SpriteFrame(images=sprites)
+
+
+class TalkingAnimation(FrameProcessor):
+ """
+ This class starts a talking animation when it receives an first AudioFrame,
+ and then returns to a "quiet" sprite when it sees a LLMResponseEndFrame.
+ """
+
+ def __init__(self):
+ super().__init__()
+ self._is_talking = False
+
+ async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
+ if isinstance(frame, AudioFrame):
+ if not self._is_talking:
+ yield talking_frame
+ yield frame
+ self._is_talking = True
+ else:
+ yield frame
+ elif isinstance(frame, TTSEndFrame):
+ yield quiet_frame
+ yield frame
+ self._is_talking = False
+ else:
+ yield frame
+
+
+class AnimationInitializer(FrameProcessor):
+ def __init__(self):
+ super().__init__()
+
+ async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
+ if isinstance(frame, PipelineStartedFrame):
+ yield quiet_frame
+ yield frame
+ else:
+ yield frame
+
+
+class UserImageRequester(FrameProcessor):
+ participant_id: str | None
+
+ def __init__(self):
+ super().__init__()
+ self.participant_id = None
+
+ def set_participant_id(self, participant_id: str):
+ self.participant_id = participant_id
+
+ async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
+ if self.participant_id and isinstance(frame, TextFrame):
+ if frame.text == user_request_answer:
+ yield UserImageRequestFrame(self.participant_id)
+ yield TextFrame("Describe the image in a short sentence.")
+ elif isinstance(frame, UserImageFrame):
+ yield frame
+
+
+class TextFilterProcessor(FrameProcessor):
+ text: str
+
+ def __init__(self, text: str):
+ self.text = text
+
+ async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
+ if isinstance(frame, TextFrame):
+ if frame.text != self.text:
+ yield frame
+ else:
+ yield frame
+
+
+class ImageFilterProcessor(FrameProcessor):
+ async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
+ if not isinstance(frame, ImageFrame):
+ yield frame
+
+
+async def main(room_url: str, token):
+ async with aiohttp.ClientSession() as session:
+ transport = DailyTransport(
+ room_url,
+ token,
+ "Chatbot",
+ duration_minutes=5,
+ start_transcription=True,
+ mic_enabled=True,
+ mic_sample_rate=16000,
+ camera_enabled=True,
+ camera_width=1024,
+ camera_height=576,
+ vad_enabled=True,
+ video_rendering_enabled=True
+ )
+
+ tts = ElevenLabsTTSService(
+ aiohttp_session=session,
+ api_key=os.getenv("ELEVENLABS_API_KEY"),
+ voice_id="pNInz6obpgDQGcFmaJgB",
+ )
+
+ llm = OpenAILLMService(
+ api_key=os.getenv("OPENAI_API_KEY"),
+ model="gpt-4-turbo-preview")
+
+ ta = TalkingAnimation()
+ ai = AnimationInitializer()
+
+ sa = SentenceAggregator()
+ ir = UserImageRequester()
+ va = VisionImageFrameAggregator()
+ # If you run into weird description, try with use_cpu=True
+ moondream = MoondreamService()
+
+ tf = TextFilterProcessor(user_request_answer)
+ imgf = ImageFilterProcessor()
+
+ messages = [
+ {
+ "role": "system",
+ "content": f"You are Chatbot, a friendly, helpful robot. Let the user know that you are capable of chatting or describing what you see. Your goal is to demonstrate your capabilities in a succinct way. Reply with only '{user_request_answer}' if the user asks you to describe what you see. Your output will be converted to audio so never include special characters in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by introducing yourself.",
+ },
+ ]
+
+ ura = LLMUserResponseAggregator(messages)
+
+ pipeline = Pipeline([
+ ai, ura, llm, ParallelPipeline(
+ [[sa, ir, va, moondream], [tf, imgf]]
+ ),
+ tts, ta
+ ])
+
+ @transport.event_handler("on_first_other_participant_joined")
+ async def on_first_other_participant_joined(transport, participant):
+ transport.render_participant_video(participant["id"], framerate=0)
+ ir.set_participant_id(participant["id"])
+ await pipeline.queue_frames([LLMMessagesFrame(messages)])
+
+ transport.transcription_settings["extra"]["endpointing"] = True
+ transport.transcription_settings["extra"]["punctuate"] = True
+
+ await asyncio.gather(transport.run(pipeline))
+
+
+if __name__ == "__main__":
+ (url, token) = configure()
+ asyncio.run(main(url, token))
diff --git a/examples/moondream-chatbot/env.example b/examples/moondream-chatbot/env.example
new file mode 100644
index 000000000..cdad19c2f
--- /dev/null
+++ b/examples/moondream-chatbot/env.example
@@ -0,0 +1,4 @@
+DAILY_SAMPLE_ROOM_URL=https://yourdomain.daily.co/yourroom # (for joining the bot to the same room repeatedly for local dev)
+DAILY_API_KEY=7df...
+OPENAI_API_KEY=sk-PL...
+ELEVENLABS_API_KEY=aeb...
diff --git a/examples/moondream-chatbot/image.png b/examples/moondream-chatbot/image.png
new file mode 100644
index 000000000..17ba5401f
Binary files /dev/null and b/examples/moondream-chatbot/image.png differ
diff --git a/examples/moondream-chatbot/requirements.txt b/examples/moondream-chatbot/requirements.txt
new file mode 100644
index 000000000..f0dc27585
--- /dev/null
+++ b/examples/moondream-chatbot/requirements.txt
@@ -0,0 +1,5 @@
+python-dotenv
+requests
+fastapi[all]
+uvicorn
+dailyai[daily,moondream,openai,silero]
diff --git a/examples/starter-apps/runner.py b/examples/moondream-chatbot/runner.py
similarity index 100%
rename from examples/starter-apps/runner.py
rename to examples/moondream-chatbot/runner.py
diff --git a/examples/moondream-chatbot/server.py b/examples/moondream-chatbot/server.py
new file mode 100644
index 000000000..fe383c0aa
--- /dev/null
+++ b/examples/moondream-chatbot/server.py
@@ -0,0 +1,124 @@
+import os
+import argparse
+import subprocess
+import atexit
+
+from fastapi import FastAPI, Request, HTTPException
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import JSONResponse, RedirectResponse
+
+from utils.daily_helpers import create_room as _create_room, get_token, get_name_from_url
+
+MAX_BOTS_PER_ROOM = 1
+
+# Bot sub-process dict for status reporting and concurrency control
+bot_procs = {}
+
+
+def cleanup():
+ # Clean up function, just to be extra safe
+ for proc in bot_procs.values():
+ proc.terminate()
+ proc.wait()
+
+
+atexit.register(cleanup)
+
+
+app = FastAPI()
+
+app.add_middleware(
+ CORSMiddleware,
+ allow_origins=["*"],
+ allow_credentials=True,
+ allow_methods=["*"],
+ allow_headers=["*"],
+)
+
+
+@app.get("/start")
+async def start_agent(request: Request):
+ print(f"!!! Creating room")
+ room_url, room_name = _create_room()
+ print(f"!!! Room URL: {room_url}")
+ # Ensure the room property is present
+ if not room_url:
+ raise HTTPException(
+ status_code=500,
+ detail="Missing 'room' property in request data. Cannot start agent without a target room!")
+
+ # Check if there is already an existing process running in this room
+ num_bots_in_room = sum(
+ 1 for proc in bot_procs.values() if proc[1] == room_url and proc[0].poll() is None)
+ if num_bots_in_room >= MAX_BOTS_PER_ROOM:
+ raise HTTPException(
+ status_code=500, detail=f"Max bot limited reach for room: {room_url}")
+
+ # Get the token for the room
+ token = get_token(room_url)
+
+ if not token:
+ raise HTTPException(
+ status_code=500, detail=f"Failed to get token for room: {room_url}")
+
+ # Spawn a new agent, and join the user session
+ # Note: this is mostly for demonstration purposes (refer to 'deployment' in README)
+ try:
+ proc = subprocess.Popen(
+ [
+ f"python3 -m bot -u {room_url} -t {token}"
+ ],
+ shell=True,
+ bufsize=1,
+ cwd=os.path.dirname(os.path.abspath(__file__))
+ )
+ bot_procs[proc.pid] = (proc, room_url)
+ except Exception as e:
+ raise HTTPException(
+ status_code=500, detail=f"Failed to start subprocess: {e}")
+
+ return RedirectResponse(room_url)
+
+
+@app.get("/status/{pid}")
+def get_status(pid: int):
+ # Look up the subprocess
+ proc = bot_procs.get(pid)
+
+ # If the subprocess doesn't exist, return an error
+ if not proc:
+ raise HTTPException(
+ status_code=404, detail=f"Bot with process id: {pid} not found")
+
+ # Check the status of the subprocess
+ if proc[0].poll() is None:
+ status = "running"
+ else:
+ status = "finished"
+
+ return JSONResponse({"bot_id": pid, "status": status})
+
+
+if __name__ == "__main__":
+ import uvicorn
+
+ default_host = os.getenv("HOST", "0.0.0.0")
+ default_port = int(os.getenv("FAST_API_PORT", "7860"))
+
+ parser = argparse.ArgumentParser(
+ description="Daily Moondream FastAPI server")
+ parser.add_argument("--host", type=str,
+ default=default_host, help="Host address")
+ parser.add_argument("--port", type=int,
+ default=default_port, help="Port number")
+ parser.add_argument("--reload", action="store_true",
+ help="Reload code on change")
+
+ config = parser.parse_args()
+
+ uvicorn.run(
+ "server:app",
+ host=config.host,
+ port=config.port,
+ reload=config.reload,
+ )
diff --git a/examples/moondream-chatbot/utils/daily_helpers.py b/examples/moondream-chatbot/utils/daily_helpers.py
new file mode 100644
index 000000000..140f710e4
--- /dev/null
+++ b/examples/moondream-chatbot/utils/daily_helpers.py
@@ -0,0 +1,109 @@
+
+import urllib.parse
+import os
+import time
+import urllib
+import requests
+
+from dotenv import load_dotenv
+load_dotenv()
+
+
+daily_api_path = os.getenv("DAILY_API_URL") or "api.daily.co/v1"
+daily_api_key = os.getenv("DAILY_API_KEY")
+
+
+def create_room() -> tuple[str, str]:
+ """
+ Helper function to create a Daily room.
+ # See: https://docs.daily.co/reference/rest-api/rooms
+
+ Returns:
+ tuple: A tuple containing the room URL and room name.
+
+ Raises:
+ Exception: If the request to create the room fails or if the response does not contain the room URL or room name.
+ """
+ room_props = {
+ "exp": time.time() + 60 * 60, # 1 hour
+ "enable_chat": True,
+ "enable_emoji_reactions": True,
+ "eject_at_room_exp": True,
+ "enable_prejoin_ui": False, # Important for the bot to be able to join headlessly
+ }
+ res = requests.post(
+ f"https://{daily_api_path}/rooms",
+ headers={"Authorization": f"Bearer {daily_api_key}"},
+ json={
+ "properties": room_props
+ },
+ )
+ if res.status_code != 200:
+ raise Exception(f"Unable to create room: {res.text}")
+
+ data = res.json()
+ room_url: str = data.get("url")
+ room_name: str = data.get("name")
+ if room_url is None or room_name is None:
+ raise Exception("Missing room URL or room name in response")
+
+ return room_url, room_name
+
+
+def get_name_from_url(room_url: str) -> str:
+ """
+ Extracts the name from a given room URL.
+
+ Args:
+ room_url (str): The URL of the room.
+
+ Returns:
+ str: The extracted name from the room URL.
+ """
+ return urllib.parse.urlparse(room_url).path[1:]
+
+
+def get_token(room_url: str) -> str:
+ """
+ Retrieves a meeting token for the specified Daily room URL.
+ # See: https://docs.daily.co/reference/rest-api/meeting-tokens
+
+ Args:
+ room_url (str): The URL of the Daily room.
+
+ Returns:
+ str: The meeting token.
+
+ Raises:
+ Exception: If no room URL is specified or if no Daily API key is specified.
+ Exception: If there is an error creating the meeting token.
+ """
+ if not room_url:
+ raise Exception(
+ "No Daily room specified. You must specify a Daily room in order a token to be generated.")
+
+ if not daily_api_key:
+ raise Exception(
+ "No Daily API key specified. set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.")
+
+ expiration: float = time.time() + 60 * 60
+ room_name = get_name_from_url(room_url)
+
+ res: requests.Response = requests.post(
+ f"https://{daily_api_path}/meeting-tokens",
+ headers={
+ "Authorization": f"Bearer {daily_api_key}"},
+ json={
+ "properties": {
+ "room_name": room_name,
+ "is_owner": True, # Owner tokens required for transcription
+ "exp": expiration}},
+ )
+
+ if res.status_code != 200:
+ raise Exception(
+ f"Failed to create meeting token: {res.status_code} {res.text}")
+
+ token: str = res.json()["token"]
+
+ return token
diff --git a/examples/server/README.md b/examples/server/README.md
deleted file mode 100644
index defd17943..000000000
--- a/examples/server/README.md
+++ /dev/null
@@ -1,34 +0,0 @@
-# Server Example
-
-Use this server app to quickly host a bot on the web:
-
-```
-flask --app daily-bot-manager.py --debug run
-```
-
-It's currently configured to serve example apps defined in the APPS constant in the server file:
-
-```
-chatbot
-patient-intake
-storybot
-translator
-```
-
-Once the server is started, you can create a bot instance by opening `http://127.0.0.1:5000/start/chatbot` in a browser, and the server will do the following:
-
-- Create a new, randomly-named Daily room with `DAILY_API_KEY` from your .env file or environment
-- Start an instance of `chatbot.py` and connect it to that room
-- 301 redirect your browser to the room
-
-### Options
-
-The server supports several options, which can be set in the body of a POST request, or as params in the URL of a GET request.
-
-- `room_url` (default: none): A room URL to join. If empty, the server will create a Daily room and return the URL in the response.
- room_properties (none): A JSON object (URL encoded if included as a GET parameter) for overriding default room creation properties, as described here: https://docs.daily.co/reference/rest-api/rooms/create-room This will be ignored if a room_url is provided.
-- `token_properties` (none): A JSON object (URL encoded if included as a GET parameter) for overriding default token properties. By default, the server creates an owner token with an expiration time of one hour.
-- `duration` (7200 seconds, or two hours): Use this property to set a time limit for the bot, as well as an expiration time for the room (if the server is creating one). This will not add an expiration time to an existing room. Expiration times in `token_properties` or `room_properties` will also take precedence over this value. You can set this property to `0` to disable timeouts, but this isn't recommended.
-- `bot_args` (none): A string containing any additional command-line args to pass to the bot.
-- `wait_for_bot` (true): Whether to wait for the bot to successfully join the room before returning a response from the server. If true, the server will start the bot script, then poll the room for up to 5 seconds to confirm the bot has joined the room. If it doesn't, the server will stop the bot and return a 500 response. If set to `false`, the server will start the bot, but immediately return a 200 response. This can be useful if the server is creating rooms for you, and you need the room URL to join the user to the room.
-- `redirect` (true): Instead of returning a 200 for GET requests, the server will return a 301 redirect to the ROOM_URL. This is handy for testing by creating a bot with a GET request directly in the browser. POST requests will never return redirects. Set to `false` to get 200 responses with info in a JSON object even for GET requests.
diff --git a/examples/server/daily-bot-manager.py b/examples/server/daily-bot-manager.py
deleted file mode 100644
index 6bea559e1..000000000
--- a/examples/server/daily-bot-manager.py
+++ /dev/null
@@ -1,165 +0,0 @@
-import os
-import requests
-import urllib
-import subprocess
-import time
-
-from flask import Flask, jsonify, redirect, request
-from flask_cors import CORS
-
-from dotenv import load_dotenv
-load_dotenv(override=True)
-
-app = Flask(__name__)
-CORS(app)
-
-APPS = {
- "chatbot": "../starter-apps/chatbot.py",
- "patient-intake": "../starter-apps/patient-intake.py",
- "storybot": "../starter-apps/storybot.py",
- "translator": "../starter-apps/translator.py"
-}
-
-daily_api_key = os.getenv("DAILY_API_KEY")
-api_path = os.getenv("DAILY_API_PATH") or "https://api.daily.co/v1"
-
-
-def get_room_name(room_url):
- return urllib.parse.urlparse(room_url).path[1:]
-
-
-def create_room(room_properties, exp):
- room_props = {
- "exp": exp,
- "enable_chat": True,
- "enable_emoji_reactions": True,
- "eject_at_room_exp": True,
- "enable_prejoin_ui": False,
- "enable_recording": "cloud"
- }
- if room_properties:
- room_props |= room_properties
-
- res = requests.post(
- f"{api_path}/rooms",
- headers={"Authorization": f"Bearer {daily_api_key}"},
- json={
- "properties": room_props
- },
- )
- if res.status_code != 200:
- raise Exception(f"Unable to create room: {res.text}")
-
- room_url = res.json()["url"]
- room_name = res.json()["name"]
- return (room_url, room_name)
-
-
-def create_token(room_name, token_properties, exp):
- token_props = {"exp": exp, "is_owner": True}
- if token_properties:
- token_props |= token_properties
- # Force the token to be limited to the room
- token_props |= {"room_name": room_name}
- res = requests.post(
- f'{api_path}/meeting-tokens',
- headers={
- 'Authorization': f'Bearer {daily_api_key}'},
- json={
- 'properties': token_props})
- if res.status_code != 200:
- if res.status_code != 200:
- raise Exception(f"Unable to create meeting token: {res.text}")
-
- meeting_token = res.json()['token']
- return meeting_token
-
-
-def start_bot(*, bot_path, room_url, token, bot_args, wait_for_bot):
-
- room_name = get_room_name(room_url)
- proc = subprocess.Popen(
- [f"python {bot_path} -u {room_url} -t {token} -k {daily_api_key} {bot_args}"],
- shell=True,
- bufsize=1,
- )
-
- if wait_for_bot:
- # Don't return until the bot has joined the room, but wait for at most 5
- # seconds.
- attempts = 0
- while attempts < 50:
- time.sleep(0.1)
- attempts += 1
- res = requests.get(
- f"{api_path}/rooms/{room_name}/get-session-data",
- headers={"Authorization": f"Bearer {daily_api_key}"},
- )
- if res.status_code == 200:
- print(f"Took {attempts} attempts to join room {room_name}")
- return True
-
- # If we don't break from the loop, that means we never found the bot in the room
- raise Exception("The bot was unable to join the room. Please try again.")
-
- return True
-
-
-@app.route("/start/
+
+This app connects you to a chatbot powered by GPT-4, complete with animations generated by Stable Video Diffusion.
+
+See a video of it in action: https://x.com/kwindla/status/1778628911817183509
+
+And a quick video walkthrough of the code: https://www.loom.com/share/13df1967161f4d24ade054e7f8753416
+
+ℹ️ The first time, things might take extra time to get started since VAD (Voice Activity Detection) model needs to be downloaded.
+
+## Get started
+
+```python
+python3 -m venv env
+source env/bin/activate
+pip install -r requirements.txt
+
+cp env.example .env # and add your credentials
+
+```
+
+## Run the server
+
+```bash
+python server.py
+```
+
+Then, visit `http://localhost:7860/start` in your browser to start a chatbot session.
+
+## Build and test the Docker image
+
+```
+docker build -t chatbot .
+docker run --env-file .env -p 7860:7860 chatbot
+```
diff --git a/examples/simple-chatbot/assets/robot01.png b/examples/simple-chatbot/assets/robot01.png
new file mode 100644
index 000000000..3864411dc
Binary files /dev/null and b/examples/simple-chatbot/assets/robot01.png differ
diff --git a/examples/simple-chatbot/assets/robot010.png b/examples/simple-chatbot/assets/robot010.png
new file mode 100644
index 000000000..e389e0933
Binary files /dev/null and b/examples/simple-chatbot/assets/robot010.png differ
diff --git a/examples/simple-chatbot/assets/robot011.png b/examples/simple-chatbot/assets/robot011.png
new file mode 100644
index 000000000..c0f0633f3
Binary files /dev/null and b/examples/simple-chatbot/assets/robot011.png differ
diff --git a/examples/simple-chatbot/assets/robot012.png b/examples/simple-chatbot/assets/robot012.png
new file mode 100644
index 000000000..e5fb2a7d1
Binary files /dev/null and b/examples/simple-chatbot/assets/robot012.png differ
diff --git a/examples/simple-chatbot/assets/robot013.png b/examples/simple-chatbot/assets/robot013.png
new file mode 100644
index 000000000..cd62a2005
Binary files /dev/null and b/examples/simple-chatbot/assets/robot013.png differ
diff --git a/examples/simple-chatbot/assets/robot014.png b/examples/simple-chatbot/assets/robot014.png
new file mode 100644
index 000000000..516ca4e8b
Binary files /dev/null and b/examples/simple-chatbot/assets/robot014.png differ
diff --git a/examples/simple-chatbot/assets/robot015.png b/examples/simple-chatbot/assets/robot015.png
new file mode 100644
index 000000000..9b9242691
Binary files /dev/null and b/examples/simple-chatbot/assets/robot015.png differ
diff --git a/examples/simple-chatbot/assets/robot016.png b/examples/simple-chatbot/assets/robot016.png
new file mode 100644
index 000000000..cbd2d9d6f
Binary files /dev/null and b/examples/simple-chatbot/assets/robot016.png differ
diff --git a/examples/simple-chatbot/assets/robot017.png b/examples/simple-chatbot/assets/robot017.png
new file mode 100644
index 000000000..5780fa27a
Binary files /dev/null and b/examples/simple-chatbot/assets/robot017.png differ
diff --git a/examples/simple-chatbot/assets/robot018.png b/examples/simple-chatbot/assets/robot018.png
new file mode 100644
index 000000000..5c983704d
Binary files /dev/null and b/examples/simple-chatbot/assets/robot018.png differ
diff --git a/examples/simple-chatbot/assets/robot019.png b/examples/simple-chatbot/assets/robot019.png
new file mode 100644
index 000000000..c0f9bef58
Binary files /dev/null and b/examples/simple-chatbot/assets/robot019.png differ
diff --git a/examples/simple-chatbot/assets/robot02.png b/examples/simple-chatbot/assets/robot02.png
new file mode 100644
index 000000000..267969849
Binary files /dev/null and b/examples/simple-chatbot/assets/robot02.png differ
diff --git a/examples/simple-chatbot/assets/robot020.png b/examples/simple-chatbot/assets/robot020.png
new file mode 100644
index 000000000..88bcfa04a
Binary files /dev/null and b/examples/simple-chatbot/assets/robot020.png differ
diff --git a/examples/simple-chatbot/assets/robot021.png b/examples/simple-chatbot/assets/robot021.png
new file mode 100644
index 000000000..5d30e6029
Binary files /dev/null and b/examples/simple-chatbot/assets/robot021.png differ
diff --git a/examples/simple-chatbot/assets/robot022.png b/examples/simple-chatbot/assets/robot022.png
new file mode 100644
index 000000000..0e2d412ed
Binary files /dev/null and b/examples/simple-chatbot/assets/robot022.png differ
diff --git a/examples/simple-chatbot/assets/robot023.png b/examples/simple-chatbot/assets/robot023.png
new file mode 100644
index 000000000..d4bc03938
Binary files /dev/null and b/examples/simple-chatbot/assets/robot023.png differ
diff --git a/examples/simple-chatbot/assets/robot024.png b/examples/simple-chatbot/assets/robot024.png
new file mode 100644
index 000000000..62f60c815
Binary files /dev/null and b/examples/simple-chatbot/assets/robot024.png differ
diff --git a/examples/simple-chatbot/assets/robot025.png b/examples/simple-chatbot/assets/robot025.png
new file mode 100644
index 000000000..c2ac6639e
Binary files /dev/null and b/examples/simple-chatbot/assets/robot025.png differ
diff --git a/examples/simple-chatbot/assets/robot03.png b/examples/simple-chatbot/assets/robot03.png
new file mode 100644
index 000000000..1cb8c76d5
Binary files /dev/null and b/examples/simple-chatbot/assets/robot03.png differ
diff --git a/examples/simple-chatbot/assets/robot04.png b/examples/simple-chatbot/assets/robot04.png
new file mode 100644
index 000000000..155d19f47
Binary files /dev/null and b/examples/simple-chatbot/assets/robot04.png differ
diff --git a/examples/simple-chatbot/assets/robot05.png b/examples/simple-chatbot/assets/robot05.png
new file mode 100644
index 000000000..b5a5c4b79
Binary files /dev/null and b/examples/simple-chatbot/assets/robot05.png differ
diff --git a/examples/simple-chatbot/assets/robot06.png b/examples/simple-chatbot/assets/robot06.png
new file mode 100644
index 000000000..b5733db5f
Binary files /dev/null and b/examples/simple-chatbot/assets/robot06.png differ
diff --git a/examples/simple-chatbot/assets/robot07.png b/examples/simple-chatbot/assets/robot07.png
new file mode 100644
index 000000000..8b5d57655
Binary files /dev/null and b/examples/simple-chatbot/assets/robot07.png differ
diff --git a/examples/simple-chatbot/assets/robot08.png b/examples/simple-chatbot/assets/robot08.png
new file mode 100644
index 000000000..f7600a559
Binary files /dev/null and b/examples/simple-chatbot/assets/robot08.png differ
diff --git a/examples/simple-chatbot/assets/robot09.png b/examples/simple-chatbot/assets/robot09.png
new file mode 100644
index 000000000..16c1a98ba
Binary files /dev/null and b/examples/simple-chatbot/assets/robot09.png differ
diff --git a/examples/starter-apps/chatbot.py b/examples/simple-chatbot/bot.py
similarity index 81%
rename from examples/starter-apps/chatbot.py
rename to examples/simple-chatbot/bot.py
index 19d144f61..39f1b46a4 100644
--- a/examples/starter-apps/chatbot.py
+++ b/examples/simple-chatbot/bot.py
@@ -5,24 +5,24 @@ import os
from PIL import Image
from typing import AsyncGenerator
-from pipecat.pipeline.aggregators import (
+from dailyai.pipeline.aggregators import (
LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
-from pipecat.pipeline.frames import (
+from dailyai.pipeline.frames import (
ImageFrame,
SpriteFrame,
Frame,
- LLMResponseEndFrame,
LLMMessagesFrame,
AudioFrame,
PipelineStartedFrame,
+ TTSEndFrame,
)
-from pipecat.services.ai_services import AIService
-from pipecat.pipeline.pipeline import Pipeline
-from pipecat.transports.daily_transport import DailyTransport
-from pipecat.services.open_ai_services import OpenAILLMService
-from pipecat.services.elevenlabs_ai_services import ElevenLabsTTSService
+from dailyai.services.ai_services import AIService
+from dailyai.pipeline.pipeline import Pipeline
+from dailyai.transports.daily_transport import DailyTransport
+from dailyai.services.open_ai_services import OpenAILLMService
+from dailyai.services.elevenlabs_ai_service import ElevenLabsTTSService
from runner import configure
@@ -30,7 +30,7 @@ from dotenv import load_dotenv
load_dotenv(override=True)
logging.basicConfig(format=f"%(levelno)s %(asctime)s %(message)s")
-logger = logging.getLogger("pipecat")
+logger = logging.getLogger("dailyai")
logger.setLevel(logging.DEBUG)
sprites = []
@@ -47,6 +47,7 @@ for i in range(1, 26):
flipped = sprites[::-1]
sprites.extend(flipped)
+
# When the bot isn't talking, show a static image of the cat listening
quiet_frame = ImageFrame(sprites[0], (1024, 576))
talking_frame = SpriteFrame(images=sprites)
@@ -70,7 +71,7 @@ class TalkingAnimation(AIService):
self._is_talking = True
else:
yield frame
- elif isinstance(frame, LLMResponseEndFrame):
+ elif isinstance(frame, TTSEndFrame):
yield quiet_frame
yield frame
self._is_talking = False
@@ -79,6 +80,8 @@ class TalkingAnimation(AIService):
class AnimationInitializer(AIService):
+ def __init__(self):
+ super().__init__()
async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
if isinstance(frame, PipelineStartedFrame):
@@ -120,7 +123,7 @@ async def main(room_url: str, token):
messages = [
{
"role": "system",
- "content": "You are Chatbot, a friendly, helpful robot. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by introducing yourself.",
+ "content": "You are Chatbot, a friendly, helpful robot. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by introducing yourself.",
},
]
@@ -137,6 +140,8 @@ async def main(room_url: str, token):
pre_processor=LLMUserResponseAggregator(messages),
)
+ transport.transcription_settings["extra"]["endpointing"] = True
+ transport.transcription_settings["extra"]["punctuate"] = True
await asyncio.gather(transport.run(), run_conversation())
diff --git a/examples/simple-chatbot/env.example b/examples/simple-chatbot/env.example
new file mode 100644
index 000000000..d368ae510
--- /dev/null
+++ b/examples/simple-chatbot/env.example
@@ -0,0 +1,4 @@
+DAILY_SAMPLE_ROOM_URL=https://yourdomain.daily.co/yourroom # (for joining the bot to the same room repeatedly for local dev)
+DAILY_API_KEY=7df...
+OPENAI_API_KEY=sk-PL...
+ELEVENLABS_API_KEY=aeb...
\ No newline at end of file
diff --git a/examples/simple-chatbot/image.png b/examples/simple-chatbot/image.png
new file mode 100644
index 000000000..93814fd1e
Binary files /dev/null and b/examples/simple-chatbot/image.png differ
diff --git a/examples/simple-chatbot/requirements.txt b/examples/simple-chatbot/requirements.txt
new file mode 100644
index 000000000..c8cda62cb
--- /dev/null
+++ b/examples/simple-chatbot/requirements.txt
@@ -0,0 +1,5 @@
+python-dotenv
+requests
+fastapi[all]
+uvicorn
+dailyai[daily,openai]
diff --git a/examples/simple-chatbot/runner.py b/examples/simple-chatbot/runner.py
new file mode 100644
index 000000000..6d1a8113d
--- /dev/null
+++ b/examples/simple-chatbot/runner.py
@@ -0,0 +1,58 @@
+import argparse
+import os
+import time
+import urllib
+import requests
+
+
+def configure():
+ parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample")
+ parser.add_argument(
+ "-u",
+ "--url",
+ type=str,
+ required=False,
+ help="URL of the Daily room to join")
+ parser.add_argument(
+ "-k",
+ "--apikey",
+ type=str,
+ required=False,
+ help="Daily API Key (needed to create an owner token for the room)",
+ )
+
+ args, unknown = parser.parse_known_args()
+
+ url = args.url or os.getenv("DAILY_SAMPLE_ROOM_URL")
+ key = args.apikey or os.getenv("DAILY_API_KEY")
+
+ if not url:
+ raise Exception(
+ "No Daily room specified. use the -u/--url option from the command line, or set DAILY_SAMPLE_ROOM_URL in your environment to specify a Daily room URL.")
+
+ if not key:
+ raise Exception("No Daily API key specified. use the -k/--apikey option from the command line, or set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.")
+
+ # Create a meeting token for the given room with an expiration 1 hour in
+ # the future.
+ room_name: str = urllib.parse.urlparse(url).path[1:]
+ expiration: float = time.time() + 60 * 60
+
+ res: requests.Response = requests.post(
+ f"https://api.daily.co/v1/meeting-tokens",
+ headers={
+ "Authorization": f"Bearer {key}"},
+ json={
+ "properties": {
+ "room_name": room_name,
+ "is_owner": True,
+ "exp": expiration}},
+ )
+
+ if res.status_code != 200:
+ raise Exception(
+ f"Failed to create meeting token: {res.status_code} {res.text}")
+
+ token: str = res.json()["token"]
+
+ return (url, token)
diff --git a/examples/simple-chatbot/server.py b/examples/simple-chatbot/server.py
new file mode 100644
index 000000000..1b8928db2
--- /dev/null
+++ b/examples/simple-chatbot/server.py
@@ -0,0 +1,127 @@
+import os
+import argparse
+import subprocess
+import atexit
+from pathlib import Path
+from typing import Optional
+
+from fastapi import FastAPI, Request, HTTPException
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.staticfiles import StaticFiles
+from fastapi.responses import FileResponse, JSONResponse, RedirectResponse
+
+from utils.daily_helpers import create_room as _create_room, get_token, get_name_from_url
+
+MAX_BOTS_PER_ROOM = 1
+
+# Bot sub-process dict for status reporting and concurrency control
+bot_procs = {}
+
+
+def cleanup():
+ # Clean up function, just to be extra safe
+ for proc in bot_procs.values():
+ proc.terminate()
+ proc.wait()
+
+
+atexit.register(cleanup)
+
+
+app = FastAPI()
+
+app.add_middleware(
+ CORSMiddleware,
+ allow_origins=["*"],
+ allow_credentials=True,
+ allow_methods=["*"],
+ allow_headers=["*"],
+)
+
+
+@app.get("/start")
+async def start_agent(request: Request):
+ print(f"!!! Creating room")
+ room_url, room_name = _create_room()
+ print(f"!!! Room URL: {room_url}")
+ # Ensure the room property is present
+ if not room_url:
+ raise HTTPException(
+ status_code=500,
+ detail="Missing 'room' property in request data. Cannot start agent without a target room!")
+
+ # Check if there is already an existing process running in this room
+ num_bots_in_room = sum(
+ 1 for proc in bot_procs.values() if proc[1] == room_url and proc[0].poll() is None)
+ if num_bots_in_room >= MAX_BOTS_PER_ROOM:
+ raise HTTPException(
+ status_code=500, detail=f"Max bot limited reach for room: {room_url}")
+
+ # Get the token for the room
+ token = get_token(room_url)
+
+ if not token:
+ raise HTTPException(
+ status_code=500, detail=f"Failed to get token for room: {room_url}")
+
+ # Spawn a new agent, and join the user session
+ # Note: this is mostly for demonstration purposes (refer to 'deployment' in README)
+ try:
+ proc = subprocess.Popen(
+ [
+ f"python3 -m bot -u {room_url} -t {token}"
+ ],
+ shell=True,
+ bufsize=1,
+ cwd=os.path.dirname(os.path.abspath(__file__))
+ )
+ bot_procs[proc.pid] = (proc, room_url)
+ except Exception as e:
+ raise HTTPException(
+ status_code=500, detail=f"Failed to start subprocess: {e}")
+
+ return RedirectResponse(room_url)
+
+
+@app.get("/status/{pid}")
+def get_status(pid: int):
+ # Look up the subprocess
+ proc = bot_procs.get(pid)
+
+ # If the subprocess doesn't exist, return an error
+ if not proc:
+ raise HTTPException(
+ status_code=404, detail=f"Bot with process id: {pid} not found")
+
+ # Check the status of the subprocess
+ if proc[0].poll() is None:
+ status = "running"
+ else:
+ status = "finished"
+
+ return JSONResponse({"bot_id": pid, "status": status})
+
+
+if __name__ == "__main__":
+ import uvicorn
+
+ default_host = os.getenv("HOST", "0.0.0.0")
+ default_port = int(os.getenv("FAST_API_PORT", "7860"))
+
+ parser = argparse.ArgumentParser(
+ description="Daily Storyteller FastAPI server")
+ parser.add_argument("--host", type=str,
+ default=default_host, help="Host address")
+ parser.add_argument("--port", type=int,
+ default=default_port, help="Port number")
+ parser.add_argument("--reload", action="store_true",
+ help="Reload code on change")
+
+ config = parser.parse_args()
+
+ uvicorn.run(
+ "server:app",
+ host=config.host,
+ port=config.port,
+ reload=config.reload,
+ )
diff --git a/examples/simple-chatbot/utils/daily_helpers.py b/examples/simple-chatbot/utils/daily_helpers.py
new file mode 100644
index 000000000..140f710e4
--- /dev/null
+++ b/examples/simple-chatbot/utils/daily_helpers.py
@@ -0,0 +1,109 @@
+
+import urllib.parse
+import os
+import time
+import urllib
+import requests
+
+from dotenv import load_dotenv
+load_dotenv()
+
+
+daily_api_path = os.getenv("DAILY_API_URL") or "api.daily.co/v1"
+daily_api_key = os.getenv("DAILY_API_KEY")
+
+
+def create_room() -> tuple[str, str]:
+ """
+ Helper function to create a Daily room.
+ # See: https://docs.daily.co/reference/rest-api/rooms
+
+ Returns:
+ tuple: A tuple containing the room URL and room name.
+
+ Raises:
+ Exception: If the request to create the room fails or if the response does not contain the room URL or room name.
+ """
+ room_props = {
+ "exp": time.time() + 60 * 60, # 1 hour
+ "enable_chat": True,
+ "enable_emoji_reactions": True,
+ "eject_at_room_exp": True,
+ "enable_prejoin_ui": False, # Important for the bot to be able to join headlessly
+ }
+ res = requests.post(
+ f"https://{daily_api_path}/rooms",
+ headers={"Authorization": f"Bearer {daily_api_key}"},
+ json={
+ "properties": room_props
+ },
+ )
+ if res.status_code != 200:
+ raise Exception(f"Unable to create room: {res.text}")
+
+ data = res.json()
+ room_url: str = data.get("url")
+ room_name: str = data.get("name")
+ if room_url is None or room_name is None:
+ raise Exception("Missing room URL or room name in response")
+
+ return room_url, room_name
+
+
+def get_name_from_url(room_url: str) -> str:
+ """
+ Extracts the name from a given room URL.
+
+ Args:
+ room_url (str): The URL of the room.
+
+ Returns:
+ str: The extracted name from the room URL.
+ """
+ return urllib.parse.urlparse(room_url).path[1:]
+
+
+def get_token(room_url: str) -> str:
+ """
+ Retrieves a meeting token for the specified Daily room URL.
+ # See: https://docs.daily.co/reference/rest-api/meeting-tokens
+
+ Args:
+ room_url (str): The URL of the Daily room.
+
+ Returns:
+ str: The meeting token.
+
+ Raises:
+ Exception: If no room URL is specified or if no Daily API key is specified.
+ Exception: If there is an error creating the meeting token.
+ """
+ if not room_url:
+ raise Exception(
+ "No Daily room specified. You must specify a Daily room in order a token to be generated.")
+
+ if not daily_api_key:
+ raise Exception(
+ "No Daily API key specified. set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.")
+
+ expiration: float = time.time() + 60 * 60
+ room_name = get_name_from_url(room_url)
+
+ res: requests.Response = requests.post(
+ f"https://{daily_api_path}/meeting-tokens",
+ headers={
+ "Authorization": f"Bearer {daily_api_key}"},
+ json={
+ "properties": {
+ "room_name": room_name,
+ "is_owner": True, # Owner tokens required for transcription
+ "exp": expiration}},
+ )
+
+ if res.status_code != 200:
+ raise Exception(
+ f"Failed to create meeting token: {res.status_code} {res.text}")
+
+ token: str = res.json()["token"]
+
+ return token
diff --git a/examples/starter-apps/assets/clack-short-quiet.wav b/examples/starter-apps/assets/clack-short-quiet.wav
deleted file mode 100644
index f0580d11e..000000000
Binary files a/examples/starter-apps/assets/clack-short-quiet.wav and /dev/null differ
diff --git a/examples/starter-apps/assets/clack-short.wav b/examples/starter-apps/assets/clack-short.wav
deleted file mode 100644
index 864994b28..000000000
Binary files a/examples/starter-apps/assets/clack-short.wav and /dev/null differ
diff --git a/examples/starter-apps/assets/clack.wav b/examples/starter-apps/assets/clack.wav
deleted file mode 100644
index 2f36164b3..000000000
Binary files a/examples/starter-apps/assets/clack.wav and /dev/null differ
diff --git a/examples/starter-apps/assets/ding.wav b/examples/starter-apps/assets/ding.wav
deleted file mode 100644
index b63aa3ada..000000000
Binary files a/examples/starter-apps/assets/ding.wav and /dev/null differ
diff --git a/examples/starter-apps/assets/ding2.wav b/examples/starter-apps/assets/ding2.wav
deleted file mode 100644
index 3b8ab20d5..000000000
Binary files a/examples/starter-apps/assets/ding2.wav and /dev/null differ
diff --git a/examples/starter-apps/assets/grandma-listening.png b/examples/starter-apps/assets/grandma-listening.png
deleted file mode 100644
index 2eb17e698..000000000
Binary files a/examples/starter-apps/assets/grandma-listening.png and /dev/null differ
diff --git a/examples/starter-apps/assets/grandma-writing.png b/examples/starter-apps/assets/grandma-writing.png
deleted file mode 100644
index 6820fabc6..000000000
Binary files a/examples/starter-apps/assets/grandma-writing.png and /dev/null differ
diff --git a/examples/starter-apps/patient-intake.py b/examples/starter-apps/patient-intake.py
deleted file mode 100644
index 2f3823037..000000000
--- a/examples/starter-apps/patient-intake.py
+++ /dev/null
@@ -1,352 +0,0 @@
-import copy
-import aiohttp
-import asyncio
-import json
-import logging
-import os
-import re
-import wave
-from typing import AsyncGenerator, List
-from pipecat.pipeline.opeanai_llm_aggregator import (
- OpenAIAssistantContextAggregator,
- OpenAIUserContextAggregator,
-)
-
-from pipecat.pipeline.pipeline import Pipeline
-from pipecat.transports.daily_transport import DailyTransport
-from pipecat.services.openai_llm_context import OpenAILLMContext
-from pipecat.services.open_ai_services import OpenAILLMService
-# from pipecat.services.deepgram_ai_services import DeepgramTTSService
-from pipecat.services.elevenlabs_ai_services import ElevenLabsTTSService
-from pipecat.services.fireworks_ai_services import FireworksLLMService
-from pipecat.pipeline.frames import (
- Frame,
- LLMFunctionCallFrame,
- LLMFunctionStartFrame,
- AudioFrame,
-)
-from pipecat.pipeline.openai_frames import OpenAILLMContextFrame
-from pipecat.services.ai_services import FrameLogger, AIService
-from openai._types import NotGiven, NOT_GIVEN
-
-from openai.types.chat import (
- ChatCompletionToolParam,
-)
-
-from runner import configure
-
-from dotenv import load_dotenv
-load_dotenv(override=True)
-
-logging.basicConfig(format="%(levelno)s %(asctime)s %(message)s")
-logger = logging.getLogger("pipecat")
-logger.setLevel(logging.DEBUG)
-
-sounds = {}
-sound_files = [
- "clack-short.wav",
- "clack.wav",
- "clack-short-quiet.wav",
- "ding.wav",
- "ding2.wav",
-]
-
-script_dir = os.path.dirname(__file__)
-
-for file in sound_files:
- # Build the full path to the sound file
- full_path = os.path.join(script_dir, "assets", file)
- # Get the filename without the extension to use as the dictionary key
- filename = os.path.splitext(os.path.basename(full_path))[0]
- # Open the sound and convert it to bytes
- with wave.open(full_path) as audio_file:
- sounds[file] = audio_file.readframes(-1)
-
-
-steps = [{"prompt": "Start by introducing yourself. Then, ask the user to confirm their identity by telling you their birthday, including the year. When they answer with their birthday, call the verify_birthday function.",
- "run_async": False,
- "failed": "The user provided an incorrect birthday. Ask them for their birthday again. When they answer, call the verify_birthday function.",
- "tools": [{"type": "function",
- "function": {"name": "verify_birthday",
- "description": "Use this function to verify the user has provided their correct birthday.",
- "parameters": {"type": "object",
- "properties": {"birthday": {"type": "string",
- "description": "The user's birthdate, including the year. The user can provide it in any format, but convert it to YYYY-MM-DD format to call this function.",
- }},
- },
- },
- }],
- },
- {"prompt": "Next, thank the user for confirming their identity, then ask the user to list their current prescriptions. Each prescription needs to have a medication name and a dosage. Do not call the list_prescriptions function with any unknown dosages.",
- "run_async": True,
- "tools": [{"type": "function",
- "function": {"name": "list_prescriptions",
- "description": "Once the user has provided a list of their prescription medications, call this function.",
- "parameters": {"type": "object",
- "properties": {"prescriptions": {"type": "array",
- "items": {"type": "object",
- "properties": {"medication": {"type": "string",
- "description": "The medication's name",
- },
- "dosage": {"type": "string",
- "description": "The prescription's dosage",
- },
- },
- },
- }},
- },
- },
- }],
- },
- {"prompt": "Next, ask the user if they have any allergies. Once they have listed their allergies or confirmed they don't have any, call the list_allergies function.",
- "run_async": True,
- "tools": [{"type": "function",
- "function": {"name": "list_allergies",
- "description": "Once the user has provided a list of their allergies, call this function.",
- "parameters": {"type": "object",
- "properties": {"allergies": {"type": "array",
- "items": {"type": "object",
- "properties": {"name": {"type": "string",
- "description": "What the user is allergic to",
- }},
- },
- }},
- },
- },
- }],
- },
- {"prompt": "Now ask the user if they have any medical conditions the doctor should know about. Once they've answered the question, call the list_conditions function.",
- "run_async": True,
- "tools": [{"type": "function",
- "function": {"name": "list_conditions",
- "description": "Once the user has provided a list of their medical conditions, call this function.",
- "parameters": {"type": "object",
- "properties": {"conditions": {"type": "array",
- "items": {"type": "object",
- "properties": {"name": {"type": "string",
- "description": "The user's medical condition",
- }},
- },
- }},
- },
- },
- },
- ],
- },
- {"prompt": "Finally, ask the user the reason for their doctor visit today. Once they answer, call the list_visit_reasons function.",
- "run_async": True,
- "tools": [{"type": "function",
- "function": {"name": "list_visit_reasons",
- "description": "Once the user has provided a list of the reasons they are visiting a doctor today, call this function.",
- "parameters": {"type": "object",
- "properties": {"visit_reasons": {"type": "array",
- "items": {"type": "object",
- "properties": {"name": {"type": "string",
- "description": "The user's reason for visiting the doctor",
- }},
- },
- }},
- },
- },
- }],
- },
- {"prompt": "Now, thank the user and end the conversation.",
- "run_async": True,
- "tools": [],
- },
- {"prompt": "",
- "run_async": True,
- "tools": []},
- ]
-current_step = 0
-
-
-class ChecklistProcessor(AIService):
-
- def __init__(
- self,
- context: OpenAILLMContext,
- llm: AIService,
- tools: List[ChatCompletionToolParam] | NotGiven = NOT_GIVEN,
- *args,
- **kwargs,
- ):
- super().__init__(*args, **kwargs)
- self._context: OpenAILLMContext = context
- self._llm = llm
- self._id = "You are Jessica, an agent for a company called Tri-County Health Services. Your job is to collect important information from the user before their doctor visit. You're talking to Chad Bailey. You should address the user by their first name and be polite and professional. You're not a medical professional, so you shouldn't provide any advice. Keep your responses short. Your job is to collect information to give to a doctor. Don't make assumptions about what values to plug into functions. Ask for clarification if a user response is ambiguous."
- self._acks = ["One sec.", "Let me confirm that.", "Thanks.", "OK."]
-
- # Create an allowlist of functions that the LLM can call
- self._functions = [
- "verify_birthday",
- "list_prescriptions",
- "list_allergies",
- "list_conditions",
- "list_visit_reasons",
- ]
-
- self._context.add_message(
- {"role": "system", "content": f"{self._id} {steps[0]['prompt']}"}
- )
-
- if tools:
- self._context.set_tools(tools)
-
- def verify_birthday(self, args):
- return args["birthday"] == "1983-01-01"
-
- def list_prescriptions(self, args):
- # print(f"--- Prescriptions: {args['prescriptions']}\n")
- pass
-
- def list_allergies(self, args):
- # print(f"--- Allergies: {args['allergies']}\n")
- pass
-
- def list_conditions(self, args):
- # print(f"--- Medical Conditions: {args['conditions']}")
- pass
-
- def list_visit_reasons(self, args):
- # print(f"Visit Reasons: {args['visit_reasons']}")
- pass
-
- async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
- global current_step
- this_step = steps[current_step]
- self._context.set_tools(this_step["tools"])
- if isinstance(frame, LLMFunctionStartFrame):
- print(f"... Preparing function call: {frame.function_name}")
- self._function_name = frame.function_name
- if this_step["run_async"]:
- # Get the LLM talking about the next step before getting the rest
- # of the function call completion
- current_step += 1
- self._context.add_message(
- {"role": "system", "content": steps[current_step]["prompt"]}
- )
- yield OpenAILLMContextFrame(self._context)
-
- local_context = copy.deepcopy(self._context)
- local_context.set_tool_choice("none")
- async for frame in llm.process_frame(
- OpenAILLMContextFrame(local_context)
- ):
- yield frame
- else:
- # Insert a quick response while we run the function
- yield AudioFrame(sounds["ding2.wav"])
- pass
- elif isinstance(frame, LLMFunctionCallFrame):
-
- if frame.function_name and frame.arguments:
- print(
- f"--> Calling function: {frame.function_name} with arguments:")
- pretty_json = re.sub(
- "\n", "\n ", json.dumps(
- json.loads(
- frame.arguments), indent=2))
- print(f"--> {pretty_json}\n")
- if frame.function_name not in self._functions:
- raise Exception(
- f"Unknown function.")
- fn = getattr(self, frame.function_name)
- result = fn(json.loads(frame.arguments))
-
- if not this_step["run_async"]:
- if result:
- current_step += 1
- self._context.add_message(
- {"role": "system", "content": steps[current_step]["prompt"]}
- )
- yield OpenAILLMContextFrame(self._context)
-
- local_context = copy.deepcopy(self._context)
- local_context.set_tool_choice("none")
- async for frame in llm.process_frame(
- OpenAILLMContextFrame(local_context)
- ):
- yield frame
- else:
- self._context.add_message(
- {"role": "system", "content": this_step["failed"]}
- )
- yield OpenAILLMContextFrame(self._context)
-
- local_context = copy.deepcopy(self._context)
- local_context.set_tool_choice("none")
- async for frame in llm.process_frame(
- OpenAILLMContextFrame(local_context)
- ):
- yield frame
- print(f"<-- Verify result: {result}\n")
-
- else:
- yield frame
-
-
-async def main(room_url: str, token):
- async with aiohttp.ClientSession() as session:
- global transport
- global llm
- global tts
-
- transport = DailyTransport(
- room_url,
- token,
- "Intake Bot",
- 5,
- mic_enabled=True,
- mic_sample_rate=16000,
- camera_enabled=False,
- start_transcription=True,
- vad_enabled=True,
- )
-
- messages = []
-
- llm = FireworksLLMService(
- api_key=os.getenv("FIREWORKS_API_KEY"),
- model="accounts/fireworks/models/firefunction-v1"
- )
- # tts = DeepgramTTSService(
- # aiohttp_session=session,
- # api_key=os.getenv("DEEPGRAM_API_KEY"),
- # voice="aura-asteria-en",
- # )
- tts = ElevenLabsTTSService(
- aiohttp_session=session,
- api_key=os.getenv("ELEVENLABS_API_KEY"),
- voice_id="XrExE9yKIg1WjnnlVkGX",
- )
- context = OpenAILLMContext(
- messages=messages,
- )
-
- checklist = ChecklistProcessor(context, llm)
- fl = FrameLogger("FRAME LOGGER 1:")
- fl2 = FrameLogger("FRAME LOGGER 2:")
- pipeline = Pipeline(processors=[fl, llm, fl2, checklist, tts])
-
- @transport.event_handler("on_first_other_participant_joined")
- async def on_first_other_participant_joined(transport, participant):
- await pipeline.queue_frames([OpenAILLMContextFrame(context)])
-
- async def handle_intake():
- await transport.run_interruptible_pipeline(
- pipeline,
- post_processor=OpenAIAssistantContextAggregator(context),
- pre_processor=OpenAIUserContextAggregator(context),
- )
-
- try:
- await asyncio.gather(transport.run(), handle_intake())
- except (asyncio.CancelledError, KeyboardInterrupt):
- print("whoops")
- transport.stop()
-
-
-if __name__ == "__main__":
- (url, token) = configure()
- asyncio.run(main(url, token))
diff --git a/examples/starter-apps/storybot.py b/examples/starter-apps/storybot.py
deleted file mode 100644
index af23c7ec9..000000000
--- a/examples/starter-apps/storybot.py
+++ /dev/null
@@ -1,294 +0,0 @@
-import aiohttp
-import asyncio
-import json
-import random
-import logging
-import os
-import re
-import wave
-from typing import AsyncGenerator
-from PIL import Image
-
-from pipecat.pipeline.pipeline import Pipeline
-from pipecat.pipeline.frame_processor import FrameProcessor
-from pipecat.services.live_stream import LiveStream
-from pipecat.transports.daily_transport import DailyTransport
-from pipecat.services.azure_ai_services import AzureLLMService, AzureTTSService
-from pipecat.services.fal_ai_services import FalImageGenService
-from pipecat.services.open_ai_services import OpenAILLMService
-from pipecat.services.deepgram_ai_services import DeepgramTTSService
-from pipecat.services.elevenlabs_ai_services import ElevenLabsTTSService
-from pipecat.pipeline.aggregators import (
- LLMAssistantContextAggregator,
- LLMAssistantResponseAggregator,
- LLMUserResponseAggregator,
-)
-from pipecat.pipeline.frames import (
- EndPipeFrame,
- LLMMessagesFrame,
- Frame,
- TextFrame,
- LLMResponseEndFrame,
- AudioFrame,
- ImageFrame,
- UserStoppedSpeakingFrame,
-)
-from pipecat.services.ai_services import FrameLogger, AIService
-
-from runner import configure
-
-from dotenv import load_dotenv
-load_dotenv(override=True)
-
-logging.basicConfig(format=f"%(levelno)s %(asctime)s %(message)s")
-logger = logging.getLogger("pipecat")
-logger.setLevel(logging.DEBUG)
-
-sounds = {}
-images = {}
-sound_files = ["talking.wav", "listening.wav", "ding3.wav"]
-image_files = ["grandma-writing.png", "grandma-listening.png"]
-script_dir = os.path.dirname(__file__)
-
-for file in sound_files:
- # Build the full path to the sound file
- full_path = os.path.join(script_dir, "assets", file)
- # Get the filename without the extension to use as the dictionary key
- filename = os.path.splitext(os.path.basename(full_path))[0]
- # Open the sound and convert it to bytes
- with wave.open(full_path) as audio_file:
- sounds[file] = audio_file.readframes(-1)
-
-for file in image_files:
- # Build the full path to the image file
- full_path = os.path.join(script_dir, "assets", file)
- # Get the filename without the extension to use as the dictionary key
- filename = os.path.splitext(os.path.basename(full_path))[0]
- # Open the image and convert it to bytes
- with Image.open(full_path) as img:
- images[file] = img.tobytes()
-
-
-class StoryStartFrame(TextFrame):
- pass
-
-
-class StoryPageFrame(TextFrame):
- pass
-
-
-class StoryPromptFrame(TextFrame):
- pass
-
-
-class StoryProcessor(FrameProcessor):
- def __init__(self, messages, story):
- self._messages = messages
- self._text = ""
- self._story = story
-
- async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
- """
- The response from the LLM service looks like:
- A comment about the user's choice
- [start] (when the cat starts telling parts of the story)
- A sentence of the story
- [break] (between each sentence/'page' of the story)
- [prompt] (when the cat asks the user to make a decision)
- Question about the next part of the story
-
- 1. Catch the frames that are generated by the LLM service
- """
- if isinstance(frame, UserStoppedSpeakingFrame):
- yield ImageFrame(images["grandma-writing.png"], (1024, 1024))
- yield AudioFrame(sounds["talking.wav"])
-
- elif isinstance(frame, TextFrame):
- self._text += frame.text
-
- if re.findall(r".*\[[sS]tart\].*", self._text):
- # Then we have the intro. Send it to speech ASAP
- self._text = self._text.replace("[Start]", "")
- self._text = self._text.replace("[start]", "")
-
- self._text = self._text.replace("\n", " ")
- if len(self._text) > 2:
- yield ImageFrame(images["grandma-writing.png"], (1024, 1024))
- yield StoryStartFrame(self._text)
- yield AudioFrame(sounds["ding3.wav"])
- self._text = ""
-
- elif re.findall(r".*\[[bB]reak\].*", self._text):
- # Then it's a page of the story. Get an image too
- self._text = self._text.replace("[Break]", "")
- self._text = self._text.replace("[break]", "")
- self._text = self._text.replace("\n", " ")
- if len(self._text) > 2:
- self._story.append(self._text)
- yield StoryPageFrame(self._text)
- yield AudioFrame(sounds["ding3.wav"])
-
- self._text = ""
- elif re.findall(r".*\[[pP]rompt\].*", self._text):
- # Then it's question time. Flush any
- # text here as a story page, then set
- # the var to get to prompt mode
- # cb: trying scene now
- # self.handle_chunk(self._text)
- self._text = self._text.replace("[Prompt]", "")
- self._text = self._text.replace("[prompt]", "")
-
- self._text = self._text.replace("\n", " ")
- if len(self._text) > 2:
- self._story.append(self._text)
- yield StoryPageFrame(self._text)
- else:
- # After the prompt thing, we'll catch an LLM end to get the
- # last bit
- pass
- elif isinstance(frame, LLMResponseEndFrame):
- yield ImageFrame(images["grandma-writing.png"], (1024, 1024))
- yield StoryPromptFrame(self._text)
- self._text = ""
- yield frame
- yield ImageFrame(images["grandma-listening.png"], (1024, 1024))
- yield AudioFrame(sounds["listening.wav"])
-
- else:
- # pass through everything that's not a TextFrame
- yield frame
-
-
-class StoryImageGenerator(FrameProcessor):
- def __init__(self, story, llm, img):
- self._story = story
- self._llm = llm
- self._img = img
-
- async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
- if isinstance(frame, StoryPageFrame):
- if len(self._story) == 1:
- prompt = f'You are an illustrator for a children\'s story book. Generate a prompt for DALL-E to create an illustration for the first page of the book, which reads: "{self._story[0]}"\n\n Your response should start with the phrase "Children\'s book illustration of".'
- else:
- prompt = f"You are an illustrator for a children's story book. Here is the story so far:\n\n\"{' '.join(self._story[:-1])}\"\n\nGenerate a prompt for DALL-E to create an illustration for the next page. Here's the sentence for the next page:\n\n\"{self._story[-1:][0]}\"\n\n Your response should start with the phrase \"Children's book illustration of\"."
- msgs = [{"role": "system", "content": prompt}]
- image_prompt = ""
- async for f in self._llm.process_frame(LLMMessagesFrame(msgs)):
- if isinstance(f, TextFrame):
- image_prompt += f.text
- async for f in self._img.process_frame(TextFrame(image_prompt)):
- yield f
- # Yield the original StoryPageFrame for basic image/audio sync
- yield frame
- else:
- yield frame
-
-
-async def main(room_url: str, token):
- async with aiohttp.ClientSession() as session:
- messages = [
- {
- "role": "system",
- "content": "You are a storytelling grandma who loves to make up fantastic, fun, and educational stories for children between the ages of 5 and 10 years old. Your stories are full of friendly, magical creatures. Your stories are never scary. Each sentence of your story will become a page in a storybook. Stop after 3-4 sentences and give the child a choice to make that will influence the next part of the story. Once the child responds, start by saying something nice about the choice they made, then include [start] in your response. Include [break] after each sentence of the story. Include [prompt] between the story and the prompt.",
- }
- ]
-
- story = []
-
- llm = OpenAILLMService(
- api_key=os.getenv("OPENAI_API_KEY"),
- model="gpt-4-1106-preview",
- ) # gpt-4-1106-preview
- tts = ElevenLabsTTSService(
- aiohttp_session=session,
- api_key=os.getenv("ELEVENLABS_API_KEY"),
- voice_id="Xb7hH8MSUJpSbSDYk0k2",
- ) # matilda
- img = FalImageGenService(
- params={
- image_size = "1024x1024",
- },
- aiohttp_session=session,
- key=os.getenv("FAL_KEY"),
- )
- lra = LLMAssistantResponseAggregator(messages)
- ura = LLMUserResponseAggregator(messages)
- sp = StoryProcessor(messages, story)
- sig = StoryImageGenerator(story, llm, img)
-
- transport = DailyTransport(
- room_url,
- token,
- "Storybot",
- 5,
- mic_enabled=True,
- mic_sample_rate=16000,
- camera_enabled=True,
- camera_width=1024,
- camera_height=1024,
- start_transcription=True,
- vad_enabled=True,
- vad_stop_s=1.5,
- )
-
- start_story_event = asyncio.Event()
-
- @transport.event_handler("on_first_other_participant_joined")
- async def on_first_other_participant_joined(transport, participant):
- start_story_event.set()
-
- async def storytime():
- await start_story_event.wait()
-
- # We're being a bit tricky here by using a special system prompt to
- # ask the user for a story topic. After their intial response, we'll
- # use a different system prompt to create story pages.
- intro_messages = [
- {
- "role": "system",
- "content": "You are a storytelling grandma who loves to make up fantastic, fun, and educational stories for children between the ages of 5 and 10 years old. Your stories are full of friendly, magical creatures. Your stories are never scary. Begin by asking what a child wants you to tell a story about. Keep your reponse to only a few sentences.",
- }
- ]
- lca = LLMAssistantContextAggregator(messages)
- local_pipeline = Pipeline(
- [llm, lca, tts], sink=transport.send_queue)
- await local_pipeline.queue_frames(
- [
- ImageFrame(images["grandma-listening.png"], (1024, 1024)),
- LLMMessagesFrame(intro_messages),
- AudioFrame(sounds["listening.wav"]),
- EndPipeFrame(),
- ]
- )
- await local_pipeline.run_pipeline()
-
- pipeline = Pipeline([llm, lca, tts, ls_sink])
- pipeline.queue_frames([...])
- pipeline.run()
-
- fl = FrameLogger("### After Image Generation")
- pipeline = Pipeline(
- processors=[
- ura,
- llm,
- sp,
- sig,
- fl,
- tts,
- lra,
- ]
- )
- await transport.run_pipeline(
- pipeline,
- )
-
- try:
- await asyncio.gather(transport.run(), storytime())
- except (asyncio.CancelledError, KeyboardInterrupt):
- print("whoops")
- transport.stop()
-
-
-if __name__ == "__main__":
- (url, token) = configure()
- asyncio.run(main(url, token))
diff --git a/examples/storytelling-chatbot/.dockerignore b/examples/storytelling-chatbot/.dockerignore
new file mode 100644
index 000000000..a6b5f4231
--- /dev/null
+++ b/examples/storytelling-chatbot/.dockerignore
@@ -0,0 +1,2 @@
+frontend/node_modules
+frontend/out
\ No newline at end of file
diff --git a/examples/storytelling-chatbot/.gitignore b/examples/storytelling-chatbot/.gitignore
new file mode 100644
index 000000000..ba9cd4106
--- /dev/null
+++ b/examples/storytelling-chatbot/.gitignore
@@ -0,0 +1,158 @@
+node_modules/
+.idea/
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+.DS_Store
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+# Usually these files are written by a python script from a template
+# before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+# For a library or package, you might want to ignore these files since the code is
+# intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+# However, in case of collaboration, if having platform-specific dependencies or dependencies
+# having no cross-platform support, pipenv may install dependencies that don't work, or not
+# install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+# This is especially recommended for binary packages to ensure reproducibility, and is more
+# commonly ignored for libraries.
+# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+# JetBrains specific template is maintainted in a separate JetBrains.gitignore that can
+# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+# and can be added to the global gitignore or merged into this file. For a more nuclear
+# option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+read.html
\ No newline at end of file
diff --git a/examples/storytelling-chatbot/Dockerfile b/examples/storytelling-chatbot/Dockerfile
new file mode 100644
index 000000000..b84ba5431
--- /dev/null
+++ b/examples/storytelling-chatbot/Dockerfile
@@ -0,0 +1,54 @@
+FROM python:3.11-bullseye
+
+ARG DEBIAN_FRONTEND=noninteractive
+ARG USE_PERSISTENT_DATA
+ENV PYTHONUNBUFFERED=1
+ENV NODE_MAJOR=20
+
+# Expose FastAPI port
+ENV FAST_API_PORT=7860
+EXPOSE 7860
+
+# Install system dependencies
+RUN apt-get update && apt-get install --no-install-recommends -y \
+ build-essential \
+ git \
+ ffmpeg \
+ google-perftools \
+ ca-certificates curl gnupg \
+ && apt-get clean && rm -rf /var/lib/apt/lists/*
+
+# Install Node.js
+RUN mkdir -p /etc/apt/keyrings
+RUN curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg
+RUN echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_${NODE_MAJOR}.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list > /dev/null
+RUN apt-get update && apt-get install nodejs -y
+
+# Set up a new user named "user" with user ID 1000
+RUN useradd -m -u 1000 user
+
+# Set home to the user's home directory
+ENV HOME=/home/user \
+ PATH=/home/user/.local/bin:$PATH \
+ PYTHONPATH=$HOME/app \
+ PYTHONUNBUFFERED=1
+
+# Switch to the "user" user
+USER user
+
+# Set the working directory to the user's home directory
+WORKDIR $HOME/app
+
+# Install Python dependencies
+COPY ./requirements.txt requirements.txt
+RUN pip3 install --no-cache-dir --upgrade -r requirements.txt
+
+# Copy everything else
+COPY --chown=user ./src/ src/
+
+# Copy frontend app and build
+COPY --chown=user ./frontend/ frontend/
+RUN cd frontend && npm install && npm run build
+
+# Start the FastAPI server
+CMD python3 src/server.py --port ${FAST_API_PORT}
\ No newline at end of file
diff --git a/examples/storytelling-chatbot/README.md b/examples/storytelling-chatbot/README.md
new file mode 100644
index 000000000..543ee6d46
--- /dev/null
+++ b/examples/storytelling-chatbot/README.md
@@ -0,0 +1,83 @@
+[](https://storytelling-chatbot.fly.dev)
+
+# Storytelling Chatbot
+
+
+
+This example shows how to build a voice-driven interactive storytelling experience.
+It periodically prompts the user for input for a 'choose your own adventure' style experience.
+
+We add visual elements to the story by generating images at lightning speed using Fal.
+
+
+---
+
+### It uses the following AI services:
+
+**Deepgram - Speech-to-Text**
+
+Transcribes inbound participant voice media to text.
+
+**OpenAI (GPT4) - LLM**
+
+Our creative writer LLM. You can see the context used to prompt it [here](src/prompts.py)
+
+**ElevenLabs - Text-to-Speech**
+
+Converts and streams the LLM response from text to audio
+
+**Fal.ai - Image Generation**
+
+Adds pictures to our story (really fast!) Prompting is quite key for style consistency, so we task the LLM to turn each story page into a short image prompt.
+
+---
+
+## Setup
+
+**Install requirements**
+
+```shell
+pip install -r requirements.txt
+```
+
+**Create environment file and set variables:**
+
+```shell
+mv env.example .env
+```
+
+**Build the frontend:**
+
+This project uses a custom frontend, which needs to built. Note: this is done automatically as part of the Docker deployment.
+
+```shell
+cd frontend/
+npm install / yarn
+npm run build
+```
+
+The build UI files can be found in `frontend/out`
+
+## Running it locally
+
+Start the API / bot manager:
+
+`python src/server.py`
+
+If you'd like to run a custom domain or port:
+
+`python src/server.py --host somehost --p 7777`
+
+➡️ Open the host URL in your browser
+
+> [!IMPORTANT]
+> Whilst working on the frontend code, please `yarn run dev`
+> and open the NextJS hosted service vs. the Python server.
+> (Usually localhost:3000.)
+
+---
+
+## Improvements to make
+
+- Wait for track_started event to avoid rushed intro
+- Show 5 minute timer on the UI
diff --git a/examples/storytelling-chatbot/env.example b/examples/storytelling-chatbot/env.example
new file mode 100644
index 000000000..e32c9fe23
--- /dev/null
+++ b/examples/storytelling-chatbot/env.example
@@ -0,0 +1,6 @@
+ELEVENLABS_API_KEY=
+ELEVENLABS_VOICE_ID=
+FAL_KEY=
+DAILY_API_URL=api.daily.co/v1
+DAILY_API_KEY=
+OPENAI_API_KEY=
\ No newline at end of file
diff --git a/examples/storytelling-chatbot/frontend/.eslintrc.json b/examples/storytelling-chatbot/frontend/.eslintrc.json
new file mode 100644
index 000000000..bffb357a7
--- /dev/null
+++ b/examples/storytelling-chatbot/frontend/.eslintrc.json
@@ -0,0 +1,3 @@
+{
+ "extends": "next/core-web-vitals"
+}
diff --git a/examples/storytelling-chatbot/frontend/.gitignore b/examples/storytelling-chatbot/frontend/.gitignore
new file mode 100644
index 000000000..fd3dbb571
--- /dev/null
+++ b/examples/storytelling-chatbot/frontend/.gitignore
@@ -0,0 +1,36 @@
+# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
+
+# dependencies
+/node_modules
+/.pnp
+.pnp.js
+.yarn/install-state.gz
+
+# testing
+/coverage
+
+# next.js
+/.next/
+/out/
+
+# production
+/build
+
+# misc
+.DS_Store
+*.pem
+
+# debug
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# local env files
+.env*.local
+
+# vercel
+.vercel
+
+# typescript
+*.tsbuildinfo
+next-env.d.ts
diff --git a/examples/storytelling-chatbot/frontend/README.md b/examples/storytelling-chatbot/frontend/README.md
new file mode 100644
index 000000000..c4033664f
--- /dev/null
+++ b/examples/storytelling-chatbot/frontend/README.md
@@ -0,0 +1,36 @@
+This is a [Next.js](https://nextjs.org/) project bootstrapped with [`create-next-app`](https://github.com/vercel/next.js/tree/canary/packages/create-next-app).
+
+## Getting Started
+
+First, run the development server:
+
+```bash
+npm run dev
+# or
+yarn dev
+# or
+pnpm dev
+# or
+bun dev
+```
+
+Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
+
+You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file.
+
+This project uses [`next/font`](https://nextjs.org/docs/basic-features/font-optimization) to automatically optimize and load Inter, a custom Google Font.
+
+## Learn More
+
+To learn more about Next.js, take a look at the following resources:
+
+- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
+- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.
+
+You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js/) - your feedback and contributions are welcome!
+
+## Deploy on Vercel
+
+The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.
+
+Check out our [Next.js deployment documentation](https://nextjs.org/docs/deployment) for more details.
diff --git a/examples/storytelling-chatbot/frontend/app/favicon.ico b/examples/storytelling-chatbot/frontend/app/favicon.ico
new file mode 100644
index 000000000..8c49042df
Binary files /dev/null and b/examples/storytelling-chatbot/frontend/app/favicon.ico differ
diff --git a/examples/storytelling-chatbot/frontend/app/globals.css b/examples/storytelling-chatbot/frontend/app/globals.css
new file mode 100644
index 000000000..4ea7806f0
--- /dev/null
+++ b/examples/storytelling-chatbot/frontend/app/globals.css
@@ -0,0 +1,109 @@
+@tailwind base;
+@tailwind components;
+@tailwind utilities;
+
+@layer base {
+ :root {
+ --background: 0 0% 100%;
+ --foreground: 224 71.4% 4.1%;
+
+ --card: 0 0% 100%;
+ --card-foreground: 224 71.4% 4.1%;
+
+ --popover: 0 0% 100%;
+ --popover-foreground: 224 71.4% 4.1%;
+
+ --primary: 220.9 39.3% 11%;
+ --primary-foreground: 210 20% 98%;
+
+ --secondary: 220 14.3% 95.9%;
+ --secondary-foreground: 220.9 39.3% 11%;
+
+ --muted: 220 14.3% 95.9%;
+ --muted-foreground: 220 8.9% 46.1%;
+
+ --accent: 220 14.3% 95.9%;
+ --accent-foreground: 220.9 39.3% 11%;
+
+ --destructive: 0 84.2% 60.2%;
+ --destructive-foreground: 210 20% 98%;
+
+ --border: 220 13% 91%;
+ --input: 220 13% 91%;
+ --ring: 224 71.4% 4.1%;
+
+ --radius: 0.5rem;
+ }
+
+ .dark {
+ --background: 224 71.4% 4.1%;
+ --foreground: 210 20% 98%;
+
+ --card: 224 71.4% 4.1%;
+ --card-foreground: 210 20% 98%;
+
+ --popover: 224 71.4% 4.1%;
+ --popover-foreground: 210 20% 98%;
+
+ --primary: 210 20% 98%;
+ --primary-foreground: 220.9 39.3% 11%;
+
+ --secondary: 215 27.9% 16.9%;
+ --secondary-foreground: 210 20% 98%;
+
+ --muted: 215 27.9% 16.9%;
+ --muted-foreground: 217.9 10.6% 64.9%;
+
+ --accent: 215 27.9% 16.9%;
+ --accent-foreground: 210 20% 98%;
+
+ --destructive: 0 62.8% 30.6%;
+ --destructive-foreground: 210 20% 98%;
+
+ --border: 215 27.9% 16.9%;
+ --input: 215 27.9% 16.9%;
+ --ring: 216 12.2% 83.9%;
+ }
+}
+
+@layer base {
+ * {
+ @apply border-border;
+ }
+ body {
+ @apply bg-background text-foreground;
+ }
+}
+
+body{
+ background: url("/bg.jpg") no-repeat center center;
+ background-size: cover;
+}
+
+.cardShadow{
+ box-shadow: 0px 124px 35px 0px rgba(0, 0, 0, 0.00), 0px 79px 32px 0px rgba(0, 0, 0, 0.01), 0px 45px 27px 0px rgba(0, 0, 0, 0.05), 0px 20px 20px 0px rgba(0, 0, 0, 0.09), 0px 5px 11px 0px rgba(0, 0, 0, 0.10);
+}
+
+@keyframes fadeInSlideUp {
+ 0% {
+ opacity: 0;
+ transform: translateY(50px);
+ }
+ 100% {
+ opacity: 1;
+ transform: translateY(0);
+ }
+}
+
+.fade-in {
+ animation: fadeIn 1s ease-out;
+}
+
+@keyframes fadeIn {
+ 0% {
+ opacity: 0;
+ }
+ 100% {
+ opacity: 1;
+ }
+}
\ No newline at end of file
diff --git a/examples/storytelling-chatbot/frontend/app/layout.tsx b/examples/storytelling-chatbot/frontend/app/layout.tsx
new file mode 100644
index 000000000..e7ca11512
--- /dev/null
+++ b/examples/storytelling-chatbot/frontend/app/layout.tsx
@@ -0,0 +1,46 @@
+import React from "react";
+
+import "./globals.css";
+import type { Metadata } from "next";
+import { Space_Grotesk, Space_Mono } from "next/font/google";
+
+import { cn } from "@/app/utils";
+
+// Font
+const sans = Space_Grotesk({
+ subsets: ["latin"],
+ weight: ["400", "500", "600"],
+ variable: "--font-sans",
+});
+
+const mono = Space_Mono({
+ subsets: ["latin"],
+ weight: ["400", "700"],
+ variable: "--font-mono",
+});
+
+export const metadata: Metadata = {
+ title: "Storytelling Chatbot - Daily AI",
+ description: "Built with git.new/ai",
+ metadataBase: new URL(process.env.SITE_URL || "http://localhost:3000"),
+};
+
+export default function RootLayout({
+ children,
+}: Readonly<{
+ children: React.ReactNode;
+}>) {
+ return (
+
+
+ + Please check your browser and system permissions. Make sure that + this app is allowed to access your microphone. +
+ ) : micState === "in-use" ? ( ++ Your microphone is being used by another app. Please close any + other apps using your microphone and restart this app. +
+ ) : micState === "not-found" ? ( ++ No microphone seems to be connected. Please connect a microphone. +
+ ) : micState === "not-supported" ? ( ++ This app is not supported on your device. Please update your + software or use a different device. +
+ ) : ( ++ There seems to be an issue accessing your microphone. Try + restarting the app or consult a system administrator. +
+ )} ++ This app demos a voice-controlled storytelling chatbot. It will + start with the bot asking you what kind of story you'd like + to hear (e.g. a fairy tale, a mystery, etc.). After each scene, + the bot will pause to ask for your input. Direct the story any + way you choose! +
+
+
+ Since you'll be talking to Storybot, we need to make sure + it can hear you! Please configure your microphone and speakers + below. +
++ {sentence} +
+ ))} + {partialText && ( ++ {partialText} +
+ )} +
+
+This app listens for user speech, then translates that speech to Spanish and speaks the translation back to the user using text-to-speech. It's probably most useful with multiple users talking to each other, along with some manual track subscription management in the Daily call.
+
+See a quick video walkthrough of the code here: https://www.loom.com/share/59fdddf129534dc2be4dde3cc6ebe8de
+
+## Get started
+
+```python
+python3 -m venv env
+source env/bin/activate
+pip install -r requirements.txt
+
+cp env.example .env # and add your credentials
+
+```
+
+## Run the server
+
+```bash
+python server.py
+```
+
+Then, visit `http://localhost:7860/start` in your browser to start a translatorbot session.
+
+## Build and test the Docker image
+
+```
+docker build -t chatbot .
+docker run --env-file .env -p 7860:7860 chatbot
+```
diff --git a/examples/starter-apps/translator.py b/examples/translation-chatbot/bot.py
similarity index 82%
rename from examples/starter-apps/translator.py
rename to examples/translation-chatbot/bot.py
index 94a0fe1e9..5b8c3286b 100644
--- a/examples/starter-apps/translator.py
+++ b/examples/translation-chatbot/bot.py
@@ -4,21 +4,21 @@ import logging
import os
from typing import AsyncGenerator
-from pipecat.pipeline.aggregators import (
+from dailyai.pipeline.aggregators import (
SentenceAggregator,
)
-from pipecat.pipeline.frames import (
+from dailyai.pipeline.frames import (
Frame,
LLMMessagesFrame,
TextFrame,
SendAppMessageFrame,
)
-from pipecat.pipeline.frame_processor import FrameProcessor
-from pipecat.pipeline.pipeline import Pipeline
-from pipecat.transports.daily_transport import DailyTransport
-from pipecat.services.azure_ai_services import AzureTTSService
-from pipecat.services.open_ai_services import OpenAILLMService
-from pipecat.pipeline.aggregators import LLMFullResponseAggregator
+from dailyai.pipeline.frame_processor import FrameProcessor
+from dailyai.pipeline.pipeline import Pipeline
+from dailyai.transports.daily_transport import DailyTransport
+from dailyai.services.azure_ai_services import AzureTTSService
+from dailyai.services.open_ai_services import OpenAILLMService
+from dailyai.pipeline.aggregators import LLMFullResponseAggregator
from runner import configure
@@ -28,7 +28,7 @@ from dotenv import load_dotenv
load_dotenv(override=True)
logging.basicConfig(format=f"%(levelno)s %(asctime)s %(message)s")
-logger = logging.getLogger("pipecat")
+logger = logging.getLogger("dailyai")
logger.setLevel(logging.DEBUG)
"""
@@ -99,6 +99,8 @@ async def main(room_url: str, token):
ts = TranslationSubtitles("spanish")
pipeline = Pipeline([sa, tp, llm, lfra, ts, tts])
+ transport.transcription_settings["extra"]["endpointing"] = True
+ transport.transcription_settings["extra"]["punctuate"] = True
await transport.run(pipeline)
diff --git a/examples/translation-chatbot/env.example b/examples/translation-chatbot/env.example
new file mode 100644
index 000000000..42c1083bf
--- /dev/null
+++ b/examples/translation-chatbot/env.example
@@ -0,0 +1,5 @@
+DAILY_SAMPLE_ROOM_URL=https://yourdomain.daily.co/yourroom # (for joining the bot to the same room repeatedly for local dev)
+DAILY_API_KEY=7df...
+OPENAI_API_KEY=sk-PL...
+AZURE_SPEECH_REGION=westus...
+AZURE_SPEECH_API_KEY=febf...
\ No newline at end of file
diff --git a/examples/translation-chatbot/image.png b/examples/translation-chatbot/image.png
new file mode 100644
index 000000000..af7ebe0bb
Binary files /dev/null and b/examples/translation-chatbot/image.png differ
diff --git a/examples/translation-chatbot/requirements.txt b/examples/translation-chatbot/requirements.txt
new file mode 100644
index 000000000..cb7ac4232
--- /dev/null
+++ b/examples/translation-chatbot/requirements.txt
@@ -0,0 +1,4 @@
+python-dotenv
+requests
+fastapi[all]
+dailyai[daily,openai,azure]
diff --git a/examples/translation-chatbot/runner.py b/examples/translation-chatbot/runner.py
new file mode 100644
index 000000000..6d1a8113d
--- /dev/null
+++ b/examples/translation-chatbot/runner.py
@@ -0,0 +1,58 @@
+import argparse
+import os
+import time
+import urllib
+import requests
+
+
+def configure():
+ parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample")
+ parser.add_argument(
+ "-u",
+ "--url",
+ type=str,
+ required=False,
+ help="URL of the Daily room to join")
+ parser.add_argument(
+ "-k",
+ "--apikey",
+ type=str,
+ required=False,
+ help="Daily API Key (needed to create an owner token for the room)",
+ )
+
+ args, unknown = parser.parse_known_args()
+
+ url = args.url or os.getenv("DAILY_SAMPLE_ROOM_URL")
+ key = args.apikey or os.getenv("DAILY_API_KEY")
+
+ if not url:
+ raise Exception(
+ "No Daily room specified. use the -u/--url option from the command line, or set DAILY_SAMPLE_ROOM_URL in your environment to specify a Daily room URL.")
+
+ if not key:
+ raise Exception("No Daily API key specified. use the -k/--apikey option from the command line, or set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.")
+
+ # Create a meeting token for the given room with an expiration 1 hour in
+ # the future.
+ room_name: str = urllib.parse.urlparse(url).path[1:]
+ expiration: float = time.time() + 60 * 60
+
+ res: requests.Response = requests.post(
+ f"https://api.daily.co/v1/meeting-tokens",
+ headers={
+ "Authorization": f"Bearer {key}"},
+ json={
+ "properties": {
+ "room_name": room_name,
+ "is_owner": True,
+ "exp": expiration}},
+ )
+
+ if res.status_code != 200:
+ raise Exception(
+ f"Failed to create meeting token: {res.status_code} {res.text}")
+
+ token: str = res.json()["token"]
+
+ return (url, token)
diff --git a/examples/translation-chatbot/server.py b/examples/translation-chatbot/server.py
new file mode 100644
index 000000000..1b8928db2
--- /dev/null
+++ b/examples/translation-chatbot/server.py
@@ -0,0 +1,127 @@
+import os
+import argparse
+import subprocess
+import atexit
+from pathlib import Path
+from typing import Optional
+
+from fastapi import FastAPI, Request, HTTPException
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.staticfiles import StaticFiles
+from fastapi.responses import FileResponse, JSONResponse, RedirectResponse
+
+from utils.daily_helpers import create_room as _create_room, get_token, get_name_from_url
+
+MAX_BOTS_PER_ROOM = 1
+
+# Bot sub-process dict for status reporting and concurrency control
+bot_procs = {}
+
+
+def cleanup():
+ # Clean up function, just to be extra safe
+ for proc in bot_procs.values():
+ proc.terminate()
+ proc.wait()
+
+
+atexit.register(cleanup)
+
+
+app = FastAPI()
+
+app.add_middleware(
+ CORSMiddleware,
+ allow_origins=["*"],
+ allow_credentials=True,
+ allow_methods=["*"],
+ allow_headers=["*"],
+)
+
+
+@app.get("/start")
+async def start_agent(request: Request):
+ print(f"!!! Creating room")
+ room_url, room_name = _create_room()
+ print(f"!!! Room URL: {room_url}")
+ # Ensure the room property is present
+ if not room_url:
+ raise HTTPException(
+ status_code=500,
+ detail="Missing 'room' property in request data. Cannot start agent without a target room!")
+
+ # Check if there is already an existing process running in this room
+ num_bots_in_room = sum(
+ 1 for proc in bot_procs.values() if proc[1] == room_url and proc[0].poll() is None)
+ if num_bots_in_room >= MAX_BOTS_PER_ROOM:
+ raise HTTPException(
+ status_code=500, detail=f"Max bot limited reach for room: {room_url}")
+
+ # Get the token for the room
+ token = get_token(room_url)
+
+ if not token:
+ raise HTTPException(
+ status_code=500, detail=f"Failed to get token for room: {room_url}")
+
+ # Spawn a new agent, and join the user session
+ # Note: this is mostly for demonstration purposes (refer to 'deployment' in README)
+ try:
+ proc = subprocess.Popen(
+ [
+ f"python3 -m bot -u {room_url} -t {token}"
+ ],
+ shell=True,
+ bufsize=1,
+ cwd=os.path.dirname(os.path.abspath(__file__))
+ )
+ bot_procs[proc.pid] = (proc, room_url)
+ except Exception as e:
+ raise HTTPException(
+ status_code=500, detail=f"Failed to start subprocess: {e}")
+
+ return RedirectResponse(room_url)
+
+
+@app.get("/status/{pid}")
+def get_status(pid: int):
+ # Look up the subprocess
+ proc = bot_procs.get(pid)
+
+ # If the subprocess doesn't exist, return an error
+ if not proc:
+ raise HTTPException(
+ status_code=404, detail=f"Bot with process id: {pid} not found")
+
+ # Check the status of the subprocess
+ if proc[0].poll() is None:
+ status = "running"
+ else:
+ status = "finished"
+
+ return JSONResponse({"bot_id": pid, "status": status})
+
+
+if __name__ == "__main__":
+ import uvicorn
+
+ default_host = os.getenv("HOST", "0.0.0.0")
+ default_port = int(os.getenv("FAST_API_PORT", "7860"))
+
+ parser = argparse.ArgumentParser(
+ description="Daily Storyteller FastAPI server")
+ parser.add_argument("--host", type=str,
+ default=default_host, help="Host address")
+ parser.add_argument("--port", type=int,
+ default=default_port, help="Port number")
+ parser.add_argument("--reload", action="store_true",
+ help="Reload code on change")
+
+ config = parser.parse_args()
+
+ uvicorn.run(
+ "server:app",
+ host=config.host,
+ port=config.port,
+ reload=config.reload,
+ )
diff --git a/examples/translation-chatbot/utils/daily_helpers.py b/examples/translation-chatbot/utils/daily_helpers.py
new file mode 100644
index 000000000..140f710e4
--- /dev/null
+++ b/examples/translation-chatbot/utils/daily_helpers.py
@@ -0,0 +1,109 @@
+
+import urllib.parse
+import os
+import time
+import urllib
+import requests
+
+from dotenv import load_dotenv
+load_dotenv()
+
+
+daily_api_path = os.getenv("DAILY_API_URL") or "api.daily.co/v1"
+daily_api_key = os.getenv("DAILY_API_KEY")
+
+
+def create_room() -> tuple[str, str]:
+ """
+ Helper function to create a Daily room.
+ # See: https://docs.daily.co/reference/rest-api/rooms
+
+ Returns:
+ tuple: A tuple containing the room URL and room name.
+
+ Raises:
+ Exception: If the request to create the room fails or if the response does not contain the room URL or room name.
+ """
+ room_props = {
+ "exp": time.time() + 60 * 60, # 1 hour
+ "enable_chat": True,
+ "enable_emoji_reactions": True,
+ "eject_at_room_exp": True,
+ "enable_prejoin_ui": False, # Important for the bot to be able to join headlessly
+ }
+ res = requests.post(
+ f"https://{daily_api_path}/rooms",
+ headers={"Authorization": f"Bearer {daily_api_key}"},
+ json={
+ "properties": room_props
+ },
+ )
+ if res.status_code != 200:
+ raise Exception(f"Unable to create room: {res.text}")
+
+ data = res.json()
+ room_url: str = data.get("url")
+ room_name: str = data.get("name")
+ if room_url is None or room_name is None:
+ raise Exception("Missing room URL or room name in response")
+
+ return room_url, room_name
+
+
+def get_name_from_url(room_url: str) -> str:
+ """
+ Extracts the name from a given room URL.
+
+ Args:
+ room_url (str): The URL of the room.
+
+ Returns:
+ str: The extracted name from the room URL.
+ """
+ return urllib.parse.urlparse(room_url).path[1:]
+
+
+def get_token(room_url: str) -> str:
+ """
+ Retrieves a meeting token for the specified Daily room URL.
+ # See: https://docs.daily.co/reference/rest-api/meeting-tokens
+
+ Args:
+ room_url (str): The URL of the Daily room.
+
+ Returns:
+ str: The meeting token.
+
+ Raises:
+ Exception: If no room URL is specified or if no Daily API key is specified.
+ Exception: If there is an error creating the meeting token.
+ """
+ if not room_url:
+ raise Exception(
+ "No Daily room specified. You must specify a Daily room in order a token to be generated.")
+
+ if not daily_api_key:
+ raise Exception(
+ "No Daily API key specified. set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.")
+
+ expiration: float = time.time() + 60 * 60
+ room_name = get_name_from_url(room_url)
+
+ res: requests.Response = requests.post(
+ f"https://{daily_api_path}/meeting-tokens",
+ headers={
+ "Authorization": f"Bearer {daily_api_key}"},
+ json={
+ "properties": {
+ "room_name": room_name,
+ "is_owner": True, # Owner tokens required for transcription
+ "exp": expiration}},
+ )
+
+ if res.status_code != 200:
+ raise Exception(
+ f"Failed to create meeting token: {res.status_code} {res.text}")
+
+ token: str = res.json()["token"]
+
+ return token