wx44wx/pipecat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added examples back

Browse Source
This commit is contained in:
Jon Taylor
2024-05-13 17:09:46 +01:00
parent 27ba50cbbf
commit f1c02f8554
154 changed files with 7918 additions and 1287 deletions
Download Patch File Download Diff File Expand all files Collapse all files

122
README.md
View File

@@ -6,16 +6,23 @@
[![PyPI](https://img.shields.io/pypi/v/pipecat-ai)](https://pypi.org/project/pipecat-ai) [![Discord](https://img.shields.io/discord/1239284677165056021
)](https://discord.gg/pipecat)
[![Docs](https://img.shields.io/badge/docs-docusaurus)](https://daily-co.github.io/dailyai-docs/docs/intro)
`pipecat` is a framework for building voice (and multimodal) conversational agents. Things like personal coaches, meeting assistants, story-telling toys for kids, customer support bots, and snarky social companions.
`pipecat` is a framework for building voice (and multimodal) conversational agents. Things like personal coaches, meeting assistants, [story-telling toys for kids](https://storytelling-chatbot.fly.dev/), customer support bots, [intake flows](https://www.youtube.com/watch?v=lDevgsp9vn0), and snarky social companions.
Build things like this:
Take a lot at some example apps:
[![AI-powered voice patient intake for healthcare](https://img.youtube.com/vi/lDevgsp9vn0/0.jpg)](https://www.youtube.com/watch?v=lDevgsp9vn0)
<p float="left">
<a href="https://github.com/pipecat-ai/pipecat/tree/main/examples/simple-chatbot"><img src="examples/simple-chatbot/image.png" width="280" /></a>&nbsp;
<a href="https://github.com/pipecat-ai/pipecat/tree/main/examples/storytelling-chatbot"><img src="examples/storytelling-chatbot/image.png" width="280" /></a>
<br/>
<a href="https://github.com/pipecat-ai/pipecat/tree/main/examples/translation-chatbot"><img src="examples/translation-chatbot/image.png" width="280" /></a>&nbsp;
<a href="https://github.com/pipecat-ai/pipecat/tree/main/examples/moondream-chatbot"><img src="examples/moondream-chatbot/image.png" width="280" /></a>
</p>
## Getting started with voice agents
You can get started with Pipecat running on your local machine, then move your agent processes to the cloud when you’re ready. You can also add a telephone number, image output, video input, use different LLMs, and more.
You can get started with Pipecat running on your local machine, then move your agent processes to the cloud when you’re ready. You can also add a 📞 telephone number, 🖼️ image output, 📺 video input, use different LLMs, and more.
```shell
# install the module
@@ -40,14 +47,9 @@ Your project may or may not need these, so they're made available as optional re
There are two directories of examples:
- [foundational](https://github.com/pipecat-ai/pipecat/tree/main/examples/foundational) — examples that build on each other, introducing one or two concepts at a time
- [example apps](https://github.com/pipecat-ai/pipecat-examples) — complete applications that you can use as starting points for development
- [foundational](https://github.com/pipecat-ai/pipecat/tree/main/examples/foundational) — small snippets that build on each other, introducing one or two concepts at a time
- [example apps](https://github.com/pipecat-ai/pipecat/tree/main/examples/) — complete applications that you can use as starting points for development
Before running the examples you need to install the dependencies (which will install all the dependencies to run all of the examples):
```
pip install -r requirements.txt
```
## A simple voice agent running locally
If you’re doing AI-related stuff, you probably have an OpenAI API key.
@@ -57,65 +59,111 @@ To generate voice output, one service that’s easy to get started with is Eleve
So let’s run a really simple agent that’s just a GPT-4 prompt, wired up to voice input and speaker output.
```python
TBC
#app.py
import asyncio
import aiohttp
from pipecat.frames.frames import EndFrame, TextFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.task import PipelineTask
from pipecat.pipeline.runner import PipelineRunner
from pipecat.services.elevenlabs import ElevenLabsTTSService
from pipecat.transports.services.daily import DailyParams, DailyTransport
async def main():
async with aiohttp.ClientSession() as session:
# Use Daily as a real-time media transport (WebRTC)
daily_url = ...
daily_token = ...
transport = DailyTransport(
daily_url, daily_token, "Bot Name", DailyParams(audio_out_enabled=True))
# Use Eleven Labs for Text-to-Speech
tts = ElevenLabsTTSService(
aiohttp_session=session,
api_key=...,
voice_id=...,
)
# Simple pipeline that will process tunr text to speech and output the result
pipeline = Pipeline([tts, transport.output()])
# Create Pipecat processor that can run one or more pipelines tasks
runner = PipelineRunner()
# Assign the task callable to run the pipeline
task = PipelineTask(pipeline)
# Register an event handler to play audio when a
# participant joins the transport WebRTC session
@transport.event_handler("on_participant_joined")
async def on_new_participant_joined(transport, participant):
participant_name = participant["info"]["userName"] or ''
# Queue a TextFrame that will get spoken by the TTS service (Eleven Labs)
await task.queue_frames([TextFrame(f"Hello there, {participant_name}!"), EndFrame()])
# Run the pipeline task
await runner.run(task)
if __name__ == "__main__":
asyncio.run(main())
```
Run it with:
```shell
TBC
python app.py
```
Daily provides a prebuilt WebRTC user interface. Whilst the app is running, you can visit at `https://<yourdomain>.daily.co/<room_url>` and listen to the bot say hello!
## Example projects
We've created a seperate repo [here](https://github.com/pipecat-ai/pipecat-examples) that have fully featured example projects to help you get started.
```shell
git@github.com:pipecat-ai/pipecat-examples.git
```
## WebSockets instead of pipes
To run your agent in the cloud, you can switch the Pipecat transport layer to use a WebSocket instead of Unix pipes.
```shell
TBC
```
## WebRTC for production use
WebSockets are fine for server-to-server communication or for initial development. But for production use, you’ll need client-server audio to use a protocol designed for real-time media transport. (For an explanation of the difference between WebSockets and WebRTC, see [this post.])
WebSockets are fine for server-to-server communication or for initial development. But for production use, you’ll need client-server audio to use a protocol designed for real-time media transport. (For an explanation of the difference between WebSockets and WebRTC, see [this post.](https://www.daily.co/blog/how-to-talk-to-an-llm-with-your-voice/#webrtc))
One way to get up and running quickly with WebRTC is to sign up for a Daily developer account. Daily gives you SDKs and global infrastructure for audio (and video) routing. Every account gets 10,000 audio/video/transcription minutes free each month.
Sign up [here](https://dashboard.daily.co/u/signup) and [create a room](https://docs.daily.co/reference/rest-api/rooms) in the developer Dashboard. Then run the examples, this time connecting via WebRTC instead of a WebSocket.
Sign up [here](https://dashboard.daily.co/u/signup) and [create a room](https://docs.daily.co/reference/rest-api/rooms) in the developer Dashboard.
### What is VAD?
Voice Activity Detection &mdash; very important for knowing when a user has finished speaking to your bot. If you are not using press-to-talk, and want Pipecat to detect when the user has finished talking, VAD is an essential component for a natural feeling conversation.
Pipecast makes use of WebRTC VAD by default when using a WebRTC transport layer. Optionally, you can use Silero VAD for improved accuracy at the cost of higher CPU usage.
```shell
pip install pipecat-ai[silero]
```
The first time your run your bot with Silero, startup may take a while whilst it downloads and caches the model in the background. You can check the progress of this in the console.
## Hacking on the framework itself
_Note that you may need to set up a virtual environment before following the instructions below. For instance, you might need to run the following from the root of the repo:_
```
```shell
python3 -m venv venv
source venv/bin/activate
```
From the root of this repo, run the following:
```
```shell
pip install -r dev-requirements.txt -r {env}-requirements.txt
python -m build
```
This builds the package. To use the package locally (eg to run sample files), run
```
```shell
pip install --editable .
```
If you want to use this package from another directory, you can run:
```
```shell
pip install path_to_this_repo
```
@@ -123,7 +171,7 @@ pip install path_to_this_repo
From the root directory, run:
```
```shell
pytest --doctest-modules --ignore-glob="*to_be_updated*" src tests
```
@@ -170,3 +218,9 @@ Install the
"--max-line-length=100"
],
```
## Getting help
➡️ [Join our Discord](https://discord.gg/pipecat)
➡️ [Reach us on Twitter](https://x.com/pipecat_ai)

7
docs/README.md
View File

@@ -8,10 +8,3 @@ Learn about the thinking behind the framework's design.
See how a Frame is processed through a Transport, a Pipeline, and a series of Frame Processors.
## [Example Code](examples/)
The repository includes several example apps in the `examples` directory. The docs explain how they work.
## [API Reference](api/)
Complete documentation of the available classes and methods in the SDK.

119
docs/examples/01-say-one-thing.md
View File

@@ -1,119 +0,0 @@
# 01: Say One Thing
_video here - youtube?_
This example uses a text-to-speech (TTS) service to say one predefined sentence. But first, a quick overview of the general structure of these examples.
## Running the demos
All of the demos have something like this at the bottom of the file:
```python
if __name__ == "__main__":
(url, token) = configure()
asyncio.run(main(url, token))
```
### `configure()`
The `configure()` function comes from `examples/foundational/support/runner.py`, and it allows you to configure the examples from the command line directly, or using environment variables:
```bash
python 01-say-one-thing.py -u https://YOUR_DOMAIN.daily.co/YOUR_ROOM -k YOUR_API_KEY
# or
DAILY_ROOM_URL=https://YOUR_DOMAIN.daily.co/YOUR_ROOM DAILY_API_KEY=YOUR_API_KEY python 01-say-one-thing.py
# or set DAILY_ROOM_URL and DAILY_API_KEY in a .env file
python 01-say-one-thing.py
```
You'll need a Daily account to run these demos. You can sign up for free at [daily.co](https://daily.co). Once you've signed up you can create a room from the [Dashboard](https://dashboard.daily.co/rooms), and grab [your API key](https://dashboard.daily.co/developers) while you're there.
Some functionality (such as transcription) requires the bot to have owner privileges in the room. `runner.py` uses the Daily REST API to create a meeting token with owner privileges. You can learn more about meeting tokens in the [Daily docs](https://docs.daily.co/reference/rest-api/meeting-tokens).
### `asyncio.run()`
The AI SDK makes heavy use of Python's `asyncio` module. [This is a reasonable intro to the topic](https://builtin.com/data-science/asyncio) if you haven't worked with `asyncio` and coroutines before.
You can learn a bit more about the specifics of how the Daily AI SDK uses coroutines in the [Architecture Guide](../architecture.md).
## The `main()` function
All of the examples have a `main()` function with a similar structure:
- Configure the transport
- Configure the AI service(s) used in the demo
- Configure any event listeners
- Define a processing pipeline
- Run the example's coroutine(s)
### Configuring the transport
The first section of the `main()` function configures the transport object:
```python
meeting_duration_minutes = 5
transport = DailyTransportService(
room_url,
None,
"Say One Thing",
meeting_duration_minutes,
)
transport.mic_enabled = True
```
The [Architecture Guide](../architecture.md) explains the transport object in more detail. In this case, we're configuring a Daily transport object and enabling the virtual microphone, so our bot can play audio.
### Configuring the services
As described in the [Architecture Guide](../architecture.md), 'a 'Service' is a class that processes 'Frames' as part of a 'Pipeline'. In this demo app, we'll only need one service: a text-to-speech generator. We can create an instance of the `ElevenLabsTTSService` class with this line of code:
```python
tts = ElevenLabsTTSService(aiohttp_session=session, api_key=os.getenv("ELEVENLABS_API_KEY"), voice_id=os.getenv("ELEVENLABS_VOICE_ID"))
```
You'll need to make sure and set those environment variables somewhere. The easiest way to do that is to copy the `example.env` file in the repo and rename it to `.env`, and then add your credentials to that file. `runner.py` loads the `python-dotenv` module and initializes it, making the values in that file available in the environment.
### Configuring event listeners
This part isn't strictly necessary for an app like this. You could include the contents of the `on_participant_joined` function directly in the body of the `main()` function, and it would run as soon as you started the script from the command line.
Instead, we can use an event handler to wait to run that code until someone else joins the meeting. We'll define a function called `greet_user()`, and use the `@transport.event_handler("on_participant_joined")` decorator to tell the SDK that we want to run that function whenever a user joins the room.
```python
@transport.event_handler("on_participant_joined")
async def greet_user(transport, participant):
if participant["info"]["isLocal"]:
return
await tts.say(
"Hello there, " + participant["info"]["userName"] + "!",
transport.send_queue,
)
# wait for the output queue to be empty, then leave the meeting
await transport.stop_when_done()
```
### Defining a processing pipeline
In this example, we don't actually have much of a processing pipeline! In fact, we're doing the whole thing inside the `greet_user()` function already.
Pipelines usually look like a bunch of nested calls to the `run()` or `run_to_queue()` function from different Services. In this example, we're using the `say()` function from the TTS service. This is effectively a convenience wrapper around the `run_to_queue()` function, which we'll discuss more later. It's important to `await` this function to ensure that the speech frames are queued for playback before the next line of code, because of the `stop_when_done()` function being called immediately afterward.
The output of the `say()` function goes to the transport's `send_queue`. This queue is the all-important connection between the world of the Services pipeline that's generating frames asynchronously and the ordered playback of audio and visual media in the WebRTC call.
### Running the coroutines
In this example, we don't actually have any separate processing pipelines—everything happens as a result of an event from the transport. So we only need to run the transport's coroutine, and await its completion:
```python
await transport.run()
```
In future examples, we'll run more processes in parallel. For now, this script can run until the transport exits—which will happen based on calling `stop_when_done()` in the `greet_user()` function.
## Next Steps
Next, we'll start connecting multiple AI services together by building a service pipeline.
## [02 - LLM Say One Thing »](02-llm-say-one-thing.md)

5
docs/examples/README.md
View File

@@ -1,5 +0,0 @@
# Daily AI SDK Examples
The docs in this folder pair with the example apps located in `examples/foundational`. They are designed to serve as a quick references for building different kinds of AI apps. But the examples also build on one another, so it can be really helpful to walk through them in order.
To start, you can learn about the overall structure of the examples in [01 - Say One Thing](01-say-one-thing.md).

93
examples/README.md Normal file
View File

@@ -0,0 +1,93 @@
# Pipecat &mdash; Examples
## Foundational snippets
Small snippets that build on each other, introducing one or two concepts at a time.
➡️ [Take a look](https://github.com/pipecat-ai/pipecat/tree/main/examples/foundational)
## Chatbot examples
Collection of self-contained real-time voice and video AI demo applications built with [pipecat](https://github.com/pipecat-ai/pipecat/).
<p float="left">
<a href="https://github.com/pipecat-ai/pipecat-examples/tree/main/examples/simple-chatbot"><img src="simple-chatbot/image.png" width="280" /></a>&nbsp;
<a href="https://github.com/pipecat-ai/pipecat-examples/tree/main/examples/storytelling-chatbot"><img src="storytelling-chatbot/image.png" width="280" /></a>
<br/>
<a href="https://github.com/pipecat-ai/pipecat-examples/tree/main/examples/translation-chatbot"><img src="translation-chatbot/image.png" width="280" />&nbsp;
<a href="https://github.com/pipecat-ai/pipecat-examples/tree/main/examples/moondream-chatbot"><img src="moondream-chatbot/image.png" width="280" /></a>
</p>
### Quickstart
Each project has its own set of dependencies and configuration variables. They intentionally avoids shared code across projects &mdash; you can grab whichever demo folder you want to work with as a starting point.
We recommend you start with a virtual environment:
```shell
cd pipecat-ai/examples/simple-chatbot
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
```
Next, follow the steps in the README for each demo.
ℹ️ Make sure you `pip install -r requirements.txt` for each demo project, so you can be sure to have the necessary service dependencies that extend the functionality of Pipecat. You can read more about the framework architecture [here](https://github.com/pipecat-ai/pipecat/tree/main/docs).
## Projects:
| Project | Description | Services |
| -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------- |
| [Simple Chatbot](simple-chatbot) | Basic voice-driven conversational bot. A good starting point for learning the flow of the framework. | Deepgram, OpenAI, Daily, Daily Prebuilt UI |
| [Storytelling Chatbot](storytelling-chatbot) | Stitches together multiple third-party services to create a collaborative storytime experience. | Deepgram, ElevenLabs, Open AI, Fal, Daily, Custom UI |
| [Translation Chatbot](translation-chatbot) | Listens for user speech, then translates that speech to Spanish and speaks the translation back. Demonstrates multi-participant use-cases. | Deepgram, Azure, OpenAI, Daily, Daily Prebuilt UI |
| [Moondream Chatbot](moondream-chatbot) | Demonstrates how to add vision capabilities to GPT4. **Note: works best with a GPU** | Deepgram, OpenAI, Moondream, Daily, Daily Prebuilt UI |
| Function-calling Chatbot (TBC) | A chatbot that can call functions in response to user input | Deepgram, OpenAI, Fireworks, Daily, Daily Prebuilt UI |
> [!IMPORTANT]
> These example projects use Daily as a WebRTC transport and can be joined using their hosted Prebuilt UI.
> It provides a quick way to join a real-time session with your bot and test your ideas without building any frontend code. If you'd like to see an example of a custom UI, try Storybot.
## FAQ
### Deployment
For each of these demos we've included a `Dockerfile`. Out of the box, this should provide everything needed to get the respective demo running on a VM:
```shell
docker build username/app:tag .
docker run -p 7860:7860 --env-file ./.env username/app:tag
docker push ...
```
### SSL
If you're working with a custom UI (such as with the Storytelling Chatbot), it's important to ensure your deployment platform supports HTTPS, as accessing user devices such as mics and webcams requires SSL.
If you try to run a custom UI without SSL, you may see an error in the console telling you that `navigator` is undefined, or no devices are available.
### Are these examples production ready?
Yes, kind of.
These demos attempt to keep things simple and are unopinionated regarding environment or scalability.
We're using FastAPI to spawn a subprocess for the bots / agents &mdash; useful for small tests, but not so great for production grade apps with many concurrent users. You can see how this works in each project's `start` endpoint in `server.py`.
Creating virtualized worker pools and on-demand instances is out of scope for these examples, but we hope to add some examples to this repo soon!
For projects that have CUDA as a requirement, such as Moondream Chatbot, be sure to deploy to a GPU-powered platform (such as [fly.io](https://fly.io) or [Runpod](https://runpod.io).)
## Getting help
➡️ [Join our Discord](https://discord.gg/pipecat)
➡️ [Reach us on Twitter](https://x.com/pipecat_ai)

122
examples/image-gen.py
View File

@@ -1,122 +0,0 @@
import argparse
import asyncio
import requests
import time
import urllib.parse
import random
from pipecat.transports.daily_transport import DailyTransport
from pipecat.services.azure_ai_services import AzureLLMService, AzureTTSService
from pipecat.pipeline.frames import Frame
from pipecat.services.fal_ai_services import FalImageGenService
async def main(room_url: str, token):
global transport
global llm
global tts
transport = DailyTransport(
room_url,
token,
"Imagebot",
1,
)
transport._mic_enabled = True
transport._camera_enabled = True
transport._mic_sample_rate = 16000
transport._camera_width = 1024
transport._camera_height = 1024
llm = AzureLLMService()
tts = AzureTTSService()
img = FalImageGenService()
async def handle_transcriptions():
print("handle_transcriptions got called")
sentence = ""
async for message in transport.get_transcriptions():
print(f"transcription message: {message}")
if message["session_id"] == transport._my_participant_id:
continue
finder = message["text"].find("start over")
print(f"finder: {finder}")
if finder >= 0:
async for audio in tts.run_tts(f"Resetting."):
transport.output_queue.put(
Frame(FrameType.AUDIO_FRAME, audio))
sentence = ""
continue
# todo: we could differentiate between transcriptions from
# different participants
sentence += f" {message['text']}"
print(f"sentence is now: {sentence}")
# TODO: Cache this audio
phrase = random.choice(
["OK.", "Got it.", "Sure.", "You bet.", "Sure thing."])
async for audio in tts.run_tts(phrase):
transport.output_queue.put(Frame(FrameType.AUDIO_FRAME, audio))
img_result = img.run_image_gen(sentence, "1024x1024")
awaited_img = await asyncio.gather(img_result)
transport.output_queue.put(
[
Frame(FrameType.IMAGE_FRAME, awaited_img[0][1]),
]
)
@transport.event_handler("on_participant_joined")
async def on_participant_joined(transport, participant):
print(f"participant joined: {participant['info']['userName']}")
if participant["info"]["isLocal"]:
return
async for audio in tts.run_tts("Describe an image, and I'll create it."):
audio_generator = tts.run_tts(
f"Hello, {participant['info']['userName']}! Describe an image and I'll create it. To start over, just say 'start over'.")
async for audio in audio_generator:
transport.output_queue.put(Frame(FrameType.AUDIO_FRAME, audio))
await asyncio.gather(transport.run(), handle_transcriptions())
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Simple Daily Bot Sample")
parser.add_argument(
"-u",
"--url",
type=str,
required=True,
help="URL of the Daily room to join")
parser.add_argument(
"-k",
"--apikey",
type=str,
required=True,
help="Daily API Key (needed to create token)",
)
args, unknown = parser.parse_known_args()
# Create a meeting token for the given room with an expiration 1 hour in
# the future.
room_name: str = urllib.parse.urlparse(args.url).path[1:]
expiration: float = time.time() + 60 * 60
res: requests.Response = requests.post(
f"https://api.daily.co/v1/meeting-tokens",
headers={
"Authorization": f"Bearer {args.apikey}"},
json={
"properties": {
"room_name": room_name,
"is_owner": True,
"exp": expiration}},
)
if res.status_code != 200:
raise Exception(
f"Failed to create meeting token: {res.status_code} {res.text}")
token: str = res.json()["token"]
asyncio.run(main(args.url, token))

135
examples/internal/11a-dial-out.py
View File

@@ -1,135 +0,0 @@
import aiohttp
import asyncio
import os
import wave
from pipecat.transports.daily_transport import DailyTransport
from pipecat.services.azure_ai_services import AzureLLMService, AzureTTSService
from pipecat.pipeline.aggregators import LLMContextAggregator
from pipecat.services.ai_services import AIService, FrameLogger
from pipecat.pipeline.frames import Frame, AudioFrame, LLMResponseEndFrame, LLMMessagesFrame
from typing import AsyncGenerator
from runner import configure
from dotenv import load_dotenv
load_dotenv(override=True)
sounds = {}
sound_files = [
'ding1.wav',
'ding2.wav'
]
script_dir = os.path.dirname(__file__)
for file in sound_files:
# Build the full path to the image file
full_path = os.path.join(script_dir, "assets", file)
# Get the filename without the extension to use as the dictionary key
filename = os.path.splitext(os.path.basename(full_path))[0]
# Open the image and convert it to bytes
with wave.open(full_path) as audio_file:
sounds[file] = audio_file.readframes(-1)
class OutboundSoundEffectWrapper(AIService):
def __init__(self):
pass
async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
if isinstance(frame, LLMResponseEndFrame):
yield AudioFrame(sounds["ding1.wav"])
# In case anything else up the stack needs it
yield frame
else:
yield frame
class InboundSoundEffectWrapper(AIService):
def __init__(self):
pass
async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
if isinstance(frame, LLMMessagesFrame):
yield AudioFrame(sounds["ding2.wav"])
# In case anything else up the stack needs it
yield frame
else:
yield frame
async def main(room_url: str, token, phone):
async with aiohttp.ClientSession() as session:
global transport
global llm
global tts
transport = DailyTransport(
room_url,
token,
"Respond bot",
300,
)
transport._mic_enabled = True
transport._mic_sample_rate = 16000
transport._camera_enabled = False
llm = AzureLLMService()
tts = AzureTTSService()
@transport.event_handler("on_first_other_participant_joined")
async def on_first_other_participant_joined(transport, participant):
await tts.say("Hi, I'm listening!", transport.send_queue)
await transport.send_queue.put(AudioFrame(sounds["ding1.wav"]))
async def handle_transcriptions():
messages = [
{"role": "system", "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio. Respond to what the user said in a creative and helpful way."},
]
tma_in = LLMContextAggregator(
messages, "user", transport._my_participant_id
)
tma_out = LLMContextAggregator(
messages, "assistant", transport._my_participant_id
)
out_sound = OutboundSoundEffectWrapper()
in_sound = InboundSoundEffectWrapper()
fl = FrameLogger("LLM Out")
fl2 = FrameLogger("Transcription In")
await out_sound.run_to_queue(
transport.send_queue,
tts.run(
tma_out.run(
llm.run(
fl2.run(
in_sound.run(
tma_in.run(
transport.get_receive_frames()
)
)
)
)
)
)
)
@transport.event_handler("on_participant_joined")
async def pax_joined(transport, pax):
print(f"PARTICIPANT JOINED: {pax}")
@transport.event_handler("on_call_state_updated")
async def on_call_state_updated(transport, state):
if (state == "joined"):
if (phone):
transport.start_recording()
transport.dialout(phone)
await asyncio.gather(transport.run(), handle_transcriptions())
if __name__ == "__main__":
(url, token) = configure()
asyncio.run(main(url, token))

163
examples/moondream-chatbot/.dockerignore Normal file
View File

@@ -0,0 +1,163 @@
# flyctl launch added from .gitignore
# Byte-compiled / optimized / DLL files
**/__pycache__
**/*.py[cod]
**/*$py.class
# C extensions
**/*.so
# Distribution / packaging
**/.Python
**/build
**/develop-eggs
**/dist
**/downloads
**/eggs
**/.eggs
**/lib
**/lib64
**/parts
**/sdist
**/var
**/wheels
**/share/python-wheels
**/*.egg-info
**/.installed.cfg
**/*.egg
**/MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
**/*.manifest
**/*.spec
# Installer logs
**/pip-log.txt
**/pip-delete-this-directory.txt
# Unit test / coverage reports
**/htmlcov
**/.tox
**/.nox
**/.coverage
**/.coverage.*
**/.cache
**/nosetests.xml
**/coverage.xml
**/*.cover
**/*.py,cover
**/.hypothesis
**/.pytest_cache
**/cover
# Translations
**/*.mo
**/*.pot
# Django stuff:
**/*.log
**/local_settings.py
**/db.sqlite3
**/db.sqlite3-journal
# Flask stuff:
**/instance
**/.webassets-cache
# Scrapy stuff:
**/.scrapy
# Sphinx documentation
**/docs/_build
# PyBuilder
**/.pybuilder
**/target
# Jupyter Notebook
**/.ipynb_checkpoints
# IPython
**/profile_default
**/ipython_config.py
# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock
# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
**/.pdm.toml
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
**/__pypackages__
# Celery stuff
**/celerybeat-schedule
**/celerybeat.pid
# SageMath parsed files
**/*.sage.py
# Environments
**/.env
**/.venv
**/env
**/venv
**/ENV
**/env.bak
**/venv.bak
# Spyder project settings
**/.spyderproject
**/.spyproject
# Rope project settings
**/.ropeproject
# mkdocs documentation
site
# mypy
**/.mypy_cache
**/.dmypy.json
**/dmypy.json
# Pyre type checker
**/.pyre
# pytype static type analyzer
**/.pytype
# Cython debug symbols
**/cython_debug
# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
**/runpod.toml
fly.toml

161
examples/moondream-chatbot/.gitignore vendored Normal file
View File

@@ -0,0 +1,161 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
.pybuilder/
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock
# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat.pid
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
# pytype static type analyzer
.pytype/
# Cython debug symbols
cython_debug/
# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
runpod.toml

25
examples/moondream-chatbot/Dockerfile Normal file
View File

@@ -0,0 +1,25 @@
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y wget
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
RUN dpkg -i cuda-keyring_1.1-1_all.deb
RUN echo "deb [signed-by=/usr/share/keyrings/cuda-archive-keyring.gpg] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/ /" > /etc/apt/sources.list.d/cuda-ubuntu2204-x86_64.list
RUN apt-get update && apt-get install -y python3 python3-pip
RUN apt-get install -y cuda-nvcc-12-4 libcublas-12-4 libcudnn8
RUN mkdir /app
RUN mkdir /app/assets
RUN mkdir /app/utils
COPY *.py /app/
COPY requirements.txt /app/
copy assets/* /app/assets/
copy utils/* /app/utils/
WORKDIR /app
RUN pip3 install -r requirements.txt
EXPOSE 7860
CMD ["python3", "server.py"]

37
examples/moondream-chatbot/README.md Normal file
View File

@@ -0,0 +1,37 @@
# Moondream Chatbot
<img src="image.png" width="420px">
This app connects you to a chatbot powered by GPT-4, complete with animations generated by Stable Video Diffusion. The chatbot also has vision powers thanks to [Moondream](https://moondream.ai) so you can ask it, for example, "what do you see?".
ℹ️ The first time, things might take some time to get started since VAD (Voice Activity Detection) and vision models need to be downloaded.
## Get started
```python
python3 -m venv env
source env/bin/activate
pip install -r requirements.txt
cp env.example .env # and add your credentials
```
## Run the server
```bash
python server.py
```
Then, visit `http://localhost:7860/start` in your browser to start a chatbot
session.
## Build and test the Docker image
```
docker build -t moonbot .
docker run --env-file .env -p 7860:7860 moonbot
```
You can try to visit `http://localhost:7860/start` again.

0
examples/starter-apps/assets/robot01.png → examples/moondream-chatbot/assets/robot01.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 759 KiB

After

Width:  |  Height:  |  Size: 759 KiB

0
examples/starter-apps/assets/robot010.png → examples/moondream-chatbot/assets/robot010.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 884 KiB

After

Width:  |  Height:  |  Size: 884 KiB

0
examples/starter-apps/assets/robot011.png → examples/moondream-chatbot/assets/robot011.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 876 KiB

After

Width:  |  Height:  |  Size: 876 KiB

0
examples/starter-apps/assets/robot012.png → examples/moondream-chatbot/assets/robot012.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 881 KiB

After

Width:  |  Height:  |  Size: 881 KiB

0
examples/starter-apps/assets/robot013.png → examples/moondream-chatbot/assets/robot013.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 866 KiB

After

Width:  |  Height:  |  Size: 866 KiB

0
examples/starter-apps/assets/robot014.png → examples/moondream-chatbot/assets/robot014.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 874 KiB

After

Width:  |  Height:  |  Size: 874 KiB

0
examples/starter-apps/assets/robot015.png → examples/moondream-chatbot/assets/robot015.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 882 KiB

After

Width:  |  Height:  |  Size: 882 KiB

0
examples/starter-apps/assets/robot016.png → examples/moondream-chatbot/assets/robot016.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 885 KiB

After

Width:  |  Height:  |  Size: 885 KiB

0
examples/starter-apps/assets/robot017.png → examples/moondream-chatbot/assets/robot017.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 888 KiB

After

Width:  |  Height:  |  Size: 888 KiB

0
examples/starter-apps/assets/robot018.png → examples/moondream-chatbot/assets/robot018.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 890 KiB

After

Width:  |  Height:  |  Size: 890 KiB

0
examples/starter-apps/assets/robot019.png → examples/moondream-chatbot/assets/robot019.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 898 KiB

After

Width:  |  Height:  |  Size: 898 KiB

0
examples/starter-apps/assets/robot02.png → examples/moondream-chatbot/assets/robot02.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 836 KiB

After

Width:  |  Height:  |  Size: 836 KiB

0
examples/starter-apps/assets/robot020.png → examples/moondream-chatbot/assets/robot020.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 903 KiB

After

Width:  |  Height:  |  Size: 903 KiB

0
examples/starter-apps/assets/robot021.png → examples/moondream-chatbot/assets/robot021.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 908 KiB

After

Width:  |  Height:  |  Size: 908 KiB

0
examples/starter-apps/assets/robot022.png → examples/moondream-chatbot/assets/robot022.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 908 KiB

After

Width:  |  Height:  |  Size: 908 KiB

0
examples/starter-apps/assets/robot023.png → examples/moondream-chatbot/assets/robot023.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 905 KiB

After

Width:  |  Height:  |  Size: 905 KiB

0
examples/starter-apps/assets/robot024.png → examples/moondream-chatbot/assets/robot024.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 903 KiB

After

Width:  |  Height:  |  Size: 903 KiB

0
examples/starter-apps/assets/robot025.png → examples/moondream-chatbot/assets/robot025.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 866 KiB

After

Width:  |  Height:  |  Size: 866 KiB

0
examples/starter-apps/assets/robot03.png → examples/moondream-chatbot/assets/robot03.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 849 KiB

After

Width:  |  Height:  |  Size: 849 KiB

0
examples/starter-apps/assets/robot04.png → examples/moondream-chatbot/assets/robot04.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 866 KiB

After

Width:  |  Height:  |  Size: 866 KiB

0
examples/starter-apps/assets/robot05.png → examples/moondream-chatbot/assets/robot05.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 866 KiB

After

Width:  |  Height:  |  Size: 866 KiB

0
examples/starter-apps/assets/robot06.png → examples/moondream-chatbot/assets/robot06.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 864 KiB

After

Width:  |  Height:  |  Size: 864 KiB

0
examples/starter-apps/assets/robot07.png → examples/moondream-chatbot/assets/robot07.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 858 KiB

After

Width:  |  Height:  |  Size: 858 KiB

0
examples/starter-apps/assets/robot08.png → examples/moondream-chatbot/assets/robot08.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 875 KiB

After

Width:  |  Height:  |  Size: 875 KiB

0
examples/starter-apps/assets/robot09.png → examples/moondream-chatbot/assets/robot09.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 881 KiB

After

Width:  |  Height:  |  Size: 881 KiB

210
examples/moondream-chatbot/bot.py Normal file
View File

@@ -0,0 +1,210 @@
import asyncio
import aiohttp
import logging
import os
from PIL import Image
from typing import AsyncGenerator
from dailyai.pipeline.aggregators import (
LLMUserResponseAggregator,
ParallelPipeline,
VisionImageFrameAggregator,
SentenceAggregator
)
from dailyai.pipeline.frames import (
ImageFrame,
SpriteFrame,
Frame,
LLMMessagesFrame,
AudioFrame,
PipelineStartedFrame,
TTSEndFrame,
TextFrame,
UserImageFrame,
UserImageRequestFrame,
)
from dailyai.services.moondream_ai_service import MoondreamService
from dailyai.pipeline.pipeline import FrameProcessor, Pipeline
from dailyai.transports.daily_transport import DailyTransport
from dailyai.services.open_ai_services import OpenAILLMService
from dailyai.services.elevenlabs_ai_service import ElevenLabsTTSService
from runner import configure
from dotenv import load_dotenv
load_dotenv(override=True)
logging.basicConfig(format=f"%(levelno)s %(asctime)s %(message)s")
logger = logging.getLogger("dailyai")
logger.setLevel(logging.DEBUG)
user_request_answer = "Let me take a look."
sprites = []
script_dir = os.path.dirname(__file__)
for i in range(1, 26):
# Build the full path to the image file
full_path = os.path.join(script_dir, f"assets/robot0{i}.png")
# Get the filename without the extension to use as the dictionary key
# Open the image and convert it to bytes
with Image.open(full_path) as img:
sprites.append(img.tobytes())
flipped = sprites[::-1]
sprites.extend(flipped)
# When the bot isn't talking, show a static image of the cat listening
quiet_frame = ImageFrame(sprites[0], (1024, 576))
talking_frame = SpriteFrame(images=sprites)
class TalkingAnimation(FrameProcessor):
"""
This class starts a talking animation when it receives an first AudioFrame,
and then returns to a "quiet" sprite when it sees a LLMResponseEndFrame.
"""
def __init__(self):
super().__init__()
self._is_talking = False
async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
if isinstance(frame, AudioFrame):
if not self._is_talking:
yield talking_frame
yield frame
self._is_talking = True
else:
yield frame
elif isinstance(frame, TTSEndFrame):
yield quiet_frame
yield frame
self._is_talking = False
else:
yield frame
class AnimationInitializer(FrameProcessor):
def __init__(self):
super().__init__()
async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
if isinstance(frame, PipelineStartedFrame):
yield quiet_frame
yield frame
else:
yield frame
class UserImageRequester(FrameProcessor):
participant_id: str | None
def __init__(self):
super().__init__()
self.participant_id = None
def set_participant_id(self, participant_id: str):
self.participant_id = participant_id
async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
if self.participant_id and isinstance(frame, TextFrame):
if frame.text == user_request_answer:
yield UserImageRequestFrame(self.participant_id)
yield TextFrame("Describe the image in a short sentence.")
elif isinstance(frame, UserImageFrame):
yield frame
class TextFilterProcessor(FrameProcessor):
text: str
def __init__(self, text: str):
self.text = text
async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
if isinstance(frame, TextFrame):
if frame.text != self.text:
yield frame
else:
yield frame
class ImageFilterProcessor(FrameProcessor):
async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
if not isinstance(frame, ImageFrame):
yield frame
async def main(room_url: str, token):
async with aiohttp.ClientSession() as session:
transport = DailyTransport(
room_url,
token,
"Chatbot",
duration_minutes=5,
start_transcription=True,
mic_enabled=True,
mic_sample_rate=16000,
camera_enabled=True,
camera_width=1024,
camera_height=576,
vad_enabled=True,
video_rendering_enabled=True
)
tts = ElevenLabsTTSService(
aiohttp_session=session,
api_key=os.getenv("ELEVENLABS_API_KEY"),
voice_id="pNInz6obpgDQGcFmaJgB",
)
llm = OpenAILLMService(
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4-turbo-preview")
ta = TalkingAnimation()
ai = AnimationInitializer()
sa = SentenceAggregator()
ir = UserImageRequester()
va = VisionImageFrameAggregator()
# If you run into weird description, try with use_cpu=True
moondream = MoondreamService()
tf = TextFilterProcessor(user_request_answer)
imgf = ImageFilterProcessor()
messages = [
{
"role": "system",
"content": f"You are Chatbot, a friendly, helpful robot. Let the user know that you are capable of chatting or describing what you see. Your goal is to demonstrate your capabilities in a succinct way. Reply with only '{user_request_answer}' if the user asks you to describe what you see. Your output will be converted to audio so never include special characters in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by introducing yourself.",
},
]
ura = LLMUserResponseAggregator(messages)
pipeline = Pipeline([
ai, ura, llm, ParallelPipeline(
[[sa, ir, va, moondream], [tf, imgf]]
),
tts, ta
])
@transport.event_handler("on_first_other_participant_joined")
async def on_first_other_participant_joined(transport, participant):
transport.render_participant_video(participant["id"], framerate=0)
ir.set_participant_id(participant["id"])
await pipeline.queue_frames([LLMMessagesFrame(messages)])
transport.transcription_settings["extra"]["endpointing"] = True
transport.transcription_settings["extra"]["punctuate"] = True
await asyncio.gather(transport.run(pipeline))
if __name__ == "__main__":
(url, token) = configure()
asyncio.run(main(url, token))

4
examples/moondream-chatbot/env.example Normal file
View File

@@ -0,0 +1,4 @@
DAILY_SAMPLE_ROOM_URL=https://yourdomain.daily.co/yourroom # (for joining the bot to the same room repeatedly for local dev)
DAILY_API_KEY=7df...
OPENAI_API_KEY=sk-PL...
ELEVENLABS_API_KEY=aeb...

BIN
examples/moondream-chatbot/image.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.1 MiB

5
examples/moondream-chatbot/requirements.txt Normal file
View File

@@ -0,0 +1,5 @@
python-dotenv
requests
fastapi[all]
uvicorn
dailyai[daily,moondream,openai,silero]

0
examples/starter-apps/runner.py → examples/moondream-chatbot/runner.py
View File

124
examples/moondream-chatbot/server.py Normal file
View File

@@ -0,0 +1,124 @@
import os
import argparse
import subprocess
import atexit
from fastapi import FastAPI, Request, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import JSONResponse, RedirectResponse
from utils.daily_helpers import create_room as _create_room, get_token, get_name_from_url
MAX_BOTS_PER_ROOM = 1
# Bot sub-process dict for status reporting and concurrency control
bot_procs = {}
def cleanup():
# Clean up function, just to be extra safe
for proc in bot_procs.values():
proc.terminate()
proc.wait()
atexit.register(cleanup)
app = FastAPI()
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
@app.get("/start")
async def start_agent(request: Request):
print(f"!!! Creating room")
room_url, room_name = _create_room()
print(f"!!! Room URL: {room_url}")
# Ensure the room property is present
if not room_url:
raise HTTPException(
status_code=500,
detail="Missing 'room' property in request data. Cannot start agent without a target room!")
# Check if there is already an existing process running in this room
num_bots_in_room = sum(
1 for proc in bot_procs.values() if proc[1] == room_url and proc[0].poll() is None)
if num_bots_in_room >= MAX_BOTS_PER_ROOM:
raise HTTPException(
status_code=500, detail=f"Max bot limited reach for room: {room_url}")
# Get the token for the room
token = get_token(room_url)
if not token:
raise HTTPException(
status_code=500, detail=f"Failed to get token for room: {room_url}")
# Spawn a new agent, and join the user session
# Note: this is mostly for demonstration purposes (refer to 'deployment' in README)
try:
proc = subprocess.Popen(
[
f"python3 -m bot -u {room_url} -t {token}"
],
shell=True,
bufsize=1,
cwd=os.path.dirname(os.path.abspath(__file__))
)
bot_procs[proc.pid] = (proc, room_url)
except Exception as e:
raise HTTPException(
status_code=500, detail=f"Failed to start subprocess: {e}")
return RedirectResponse(room_url)
@app.get("/status/{pid}")
def get_status(pid: int):
# Look up the subprocess
proc = bot_procs.get(pid)
# If the subprocess doesn't exist, return an error
if not proc:
raise HTTPException(
status_code=404, detail=f"Bot with process id: {pid} not found")
# Check the status of the subprocess
if proc[0].poll() is None:
status = "running"
else:
status = "finished"
return JSONResponse({"bot_id": pid, "status": status})
if __name__ == "__main__":
import uvicorn
default_host = os.getenv("HOST", "0.0.0.0")
default_port = int(os.getenv("FAST_API_PORT", "7860"))
parser = argparse.ArgumentParser(
description="Daily Moondream FastAPI server")
parser.add_argument("--host", type=str,
default=default_host, help="Host address")
parser.add_argument("--port", type=int,
default=default_port, help="Port number")
parser.add_argument("--reload", action="store_true",
help="Reload code on change")
config = parser.parse_args()
uvicorn.run(
"server:app",
host=config.host,
port=config.port,
reload=config.reload,
)

109
examples/moondream-chatbot/utils/daily_helpers.py Normal file
View File

@@ -0,0 +1,109 @@
import urllib.parse
import os
import time
import urllib
import requests
from dotenv import load_dotenv
load_dotenv()
daily_api_path = os.getenv("DAILY_API_URL") or "api.daily.co/v1"
daily_api_key = os.getenv("DAILY_API_KEY")
def create_room() -> tuple[str, str]:
"""
Helper function to create a Daily room.
# See: https://docs.daily.co/reference/rest-api/rooms
Returns:
tuple: A tuple containing the room URL and room name.
Raises:
Exception: If the request to create the room fails or if the response does not contain the room URL or room name.
"""
room_props = {
"exp": time.time() + 60 * 60, # 1 hour
"enable_chat": True,
"enable_emoji_reactions": True,
"eject_at_room_exp": True,
"enable_prejoin_ui": False, # Important for the bot to be able to join headlessly
}
res = requests.post(
f"https://{daily_api_path}/rooms",
headers={"Authorization": f"Bearer {daily_api_key}"},
json={
"properties": room_props
},
)
if res.status_code != 200:
raise Exception(f"Unable to create room: {res.text}")
data = res.json()
room_url: str = data.get("url")
room_name: str = data.get("name")
if room_url is None or room_name is None:
raise Exception("Missing room URL or room name in response")
return room_url, room_name
def get_name_from_url(room_url: str) -> str:
"""
Extracts the name from a given room URL.
Args:
room_url (str): The URL of the room.
Returns:
str: The extracted name from the room URL.
"""
return urllib.parse.urlparse(room_url).path[1:]
def get_token(room_url: str) -> str:
"""
Retrieves a meeting token for the specified Daily room URL.
# See: https://docs.daily.co/reference/rest-api/meeting-tokens
Args:
room_url (str): The URL of the Daily room.
Returns:
str: The meeting token.
Raises:
Exception: If no room URL is specified or if no Daily API key is specified.
Exception: If there is an error creating the meeting token.
"""
if not room_url:
raise Exception(
"No Daily room specified. You must specify a Daily room in order a token to be generated.")
if not daily_api_key:
raise Exception(
"No Daily API key specified. set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.")
expiration: float = time.time() + 60 * 60
room_name = get_name_from_url(room_url)
res: requests.Response = requests.post(
f"https://{daily_api_path}/meeting-tokens",
headers={
"Authorization": f"Bearer {daily_api_key}"},
json={
"properties": {
"room_name": room_name,
"is_owner": True, # Owner tokens required for transcription
"exp": expiration}},
)
if res.status_code != 200:
raise Exception(
f"Failed to create meeting token: {res.status_code} {res.text}")
token: str = res.json()["token"]
return token

34
examples/server/README.md
View File

@@ -1,34 +0,0 @@
# Server Example
Use this server app to quickly host a bot on the web:
```
flask --app daily-bot-manager.py --debug run
```
It's currently configured to serve example apps defined in the APPS constant in the server file:
```
chatbot
patient-intake
storybot
translator
```
Once the server is started, you can create a bot instance by opening `http://127.0.0.1:5000/start/chatbot` in a browser, and the server will do the following:
- Create a new, randomly-named Daily room with `DAILY_API_KEY` from your .env file or environment
- Start an instance of `chatbot.py` and connect it to that room
- 301 redirect your browser to the room
### Options
The server supports several options, which can be set in the body of a POST request, or as params in the URL of a GET request.
- `room_url` (default: none): A room URL to join. If empty, the server will create a Daily room and return the URL in the response.
room_properties (none): A JSON object (URL encoded if included as a GET parameter) for overriding default room creation properties, as described here: https://docs.daily.co/reference/rest-api/rooms/create-room This will be ignored if a room_url is provided.
- `token_properties` (none): A JSON object (URL encoded if included as a GET parameter) for overriding default token properties. By default, the server creates an owner token with an expiration time of one hour.
- `duration` (7200 seconds, or two hours): Use this property to set a time limit for the bot, as well as an expiration time for the room (if the server is creating one). This will not add an expiration time to an existing room. Expiration times in `token_properties` or `room_properties` will also take precedence over this value. You can set this property to `0` to disable timeouts, but this isn't recommended.
- `bot_args` (none): A string containing any additional command-line args to pass to the bot.
- `wait_for_bot` (true): Whether to wait for the bot to successfully join the room before returning a response from the server. If true, the server will start the bot script, then poll the room for up to 5 seconds to confirm the bot has joined the room. If it doesn't, the server will stop the bot and return a 500 response. If set to `false`, the server will start the bot, but immediately return a 200 response. This can be useful if the server is creating rooms for you, and you need the room URL to join the user to the room.
- `redirect` (true): Instead of returning a 200 for GET requests, the server will return a 301 redirect to the ROOM_URL. This is handy for testing by creating a bot with a GET request directly in the browser. POST requests will never return redirects. Set to `false` to get 200 responses with info in a JSON object even for GET requests.

165
examples/server/daily-bot-manager.py
View File

@@ -1,165 +0,0 @@
import os
import requests
import urllib
import subprocess
import time
from flask import Flask, jsonify, redirect, request
from flask_cors import CORS
from dotenv import load_dotenv
load_dotenv(override=True)
app = Flask(__name__)
CORS(app)
APPS = {
"chatbot": "../starter-apps/chatbot.py",
"patient-intake": "../starter-apps/patient-intake.py",
"storybot": "../starter-apps/storybot.py",
"translator": "../starter-apps/translator.py"
}
daily_api_key = os.getenv("DAILY_API_KEY")
api_path = os.getenv("DAILY_API_PATH") or "https://api.daily.co/v1"
def get_room_name(room_url):
return urllib.parse.urlparse(room_url).path[1:]
def create_room(room_properties, exp):
room_props = {
"exp": exp,
"enable_chat": True,
"enable_emoji_reactions": True,
"eject_at_room_exp": True,
"enable_prejoin_ui": False,
"enable_recording": "cloud"
}
if room_properties:
room_props |= room_properties
res = requests.post(
f"{api_path}/rooms",
headers={"Authorization": f"Bearer {daily_api_key}"},
json={
"properties": room_props
},
)
if res.status_code != 200:
raise Exception(f"Unable to create room: {res.text}")
room_url = res.json()["url"]
room_name = res.json()["name"]
return (room_url, room_name)
def create_token(room_name, token_properties, exp):
token_props = {"exp": exp, "is_owner": True}
if token_properties:
token_props |= token_properties
# Force the token to be limited to the room
token_props |= {"room_name": room_name}
res = requests.post(
f'{api_path}/meeting-tokens',
headers={
'Authorization': f'Bearer {daily_api_key}'},
json={
'properties': token_props})
if res.status_code != 200:
if res.status_code != 200:
raise Exception(f"Unable to create meeting token: {res.text}")
meeting_token = res.json()['token']
return meeting_token
def start_bot(*, bot_path, room_url, token, bot_args, wait_for_bot):
room_name = get_room_name(room_url)
proc = subprocess.Popen(
[f"python {bot_path} -u {room_url} -t {token} -k {daily_api_key} {bot_args}"],
shell=True,
bufsize=1,
)
if wait_for_bot:
# Don't return until the bot has joined the room, but wait for at most 5
# seconds.
attempts = 0
while attempts < 50:
time.sleep(0.1)
attempts += 1
res = requests.get(
f"{api_path}/rooms/{room_name}/get-session-data",
headers={"Authorization": f"Bearer {daily_api_key}"},
)
if res.status_code == 200:
print(f"Took {attempts} attempts to join room {room_name}")
return True
# If we don't break from the loop, that means we never found the bot in the room
raise Exception("The bot was unable to join the room. Please try again.")
return True
@app.route("/start/<string:botname>", methods=["GET", "POST"])
def start(botname):
try:
if botname not in APPS:
raise Exception(f"Bot '{botname}' is not in the allowlist.")
bot_path = APPS[botname]
props = {
"room_url": None,
"room_properties": None,
"token_properties": None,
"bot_args": None,
"wait_for_bot": True,
"duration": None,
"redirect": True
}
props |= request.values.to_dict() # gets URL params as well as plaintext POST body
try:
props |= request.json
except BaseException:
pass
if props['redirect'] == "false":
props['redirect'] = False
if props['wait_for_bot'] == "false":
props['wait_for_bot'] = False
duration = int(os.getenv("DAILY_BOT_DURATION") or 7200)
if props['duration']:
duration = props['duration']
exp = time.time() + duration
if (props['room_url']):
room_url = props['room_url']
try:
room_name = get_room_name(room_url)
except ValueError:
raise Exception(
"There was a problem detecting the room name. Please double-check the value of room_url.")
else:
room_url, room_name = create_room(props['room_properties'], exp)
token = create_token(room_name, props['token_properties'], exp)
bot = start_bot(
room_url=room_url,
bot_path=bot_path,
token=token,
bot_args=props['bot_args'],
wait_for_bot=props['wait_for_bot'])
if props['redirect'] and request.method == "GET":
return redirect(room_url, 302)
else:
return jsonify({"room_url": room_url, "token": token})
except BaseException as e:
return f"There was a problem starting the bot: {e}", 500
@app.route("/healthz")
def health_check():
return "ok", 200

161
examples/simple-chatbot/.gitignore vendored Normal file
View File

@@ -0,0 +1,161 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
.pybuilder/
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock
# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat.pid
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
# pytype static type analyzer
.pytype/
# Cython debug symbols
cython_debug/
# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
runpod.toml

16
examples/simple-chatbot/Dockerfile Normal file
View File

@@ -0,0 +1,16 @@
FROM python:3.10-bullseye
RUN mkdir /app
RUN mkdir /app/assets
RUN mkdir /app/utils
COPY *.py /app/
COPY requirements.txt /app/
copy assets/* /app/assets/
copy utils/* /app/utils/
WORKDIR /app
RUN pip3 install -r requirements.txt
EXPOSE 7860
CMD ["python3", "server.py"]

37
examples/simple-chatbot/README.md Normal file
View File

@@ -0,0 +1,37 @@
# Simple Chatbot
<img src="image.png" width="420px">
This app connects you to a chatbot powered by GPT-4, complete with animations generated by Stable Video Diffusion.
See a video of it in action: https://x.com/kwindla/status/1778628911817183509
And a quick video walkthrough of the code: https://www.loom.com/share/13df1967161f4d24ade054e7f8753416
ℹ️ The first time, things might take extra time to get started since VAD (Voice Activity Detection) model needs to be downloaded.
## Get started
```python
python3 -m venv env
source env/bin/activate
pip install -r requirements.txt
cp env.example .env # and add your credentials
```
## Run the server
```bash
python server.py
```
Then, visit `http://localhost:7860/start` in your browser to start a chatbot session.
## Build and test the Docker image
```
docker build -t chatbot .
docker run --env-file .env -p 7860:7860 chatbot
```

BIN
examples/simple-chatbot/assets/robot01.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 759 KiB

BIN
examples/simple-chatbot/assets/robot010.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 884 KiB

BIN
examples/simple-chatbot/assets/robot011.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 876 KiB

BIN
examples/simple-chatbot/assets/robot012.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 881 KiB

BIN
examples/simple-chatbot/assets/robot013.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 866 KiB

BIN
examples/simple-chatbot/assets/robot014.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 874 KiB

BIN
examples/simple-chatbot/assets/robot015.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 882 KiB

BIN
examples/simple-chatbot/assets/robot016.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 885 KiB

BIN
examples/simple-chatbot/assets/robot017.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 888 KiB

BIN
examples/simple-chatbot/assets/robot018.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 890 KiB

BIN
examples/simple-chatbot/assets/robot019.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 898 KiB

BIN
examples/simple-chatbot/assets/robot02.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 836 KiB

BIN
examples/simple-chatbot/assets/robot020.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 903 KiB

BIN
examples/simple-chatbot/assets/robot021.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 908 KiB

BIN
examples/simple-chatbot/assets/robot022.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 908 KiB

BIN
examples/simple-chatbot/assets/robot023.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 905 KiB

BIN
examples/simple-chatbot/assets/robot024.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 903 KiB

BIN
examples/simple-chatbot/assets/robot025.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 866 KiB

BIN
examples/simple-chatbot/assets/robot03.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 849 KiB

BIN
examples/simple-chatbot/assets/robot04.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 866 KiB

BIN
examples/simple-chatbot/assets/robot05.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 866 KiB

BIN
examples/simple-chatbot/assets/robot06.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 864 KiB

BIN
examples/simple-chatbot/assets/robot07.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 858 KiB

BIN
examples/simple-chatbot/assets/robot08.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 875 KiB

BIN
examples/simple-chatbot/assets/robot09.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 881 KiB

27
examples/starter-apps/chatbot.py → examples/simple-chatbot/bot.py
View File

@@ -5,24 +5,24 @@ import os
from PIL import Image
from typing import AsyncGenerator
from pipecat.pipeline.aggregators import (
from dailyai.pipeline.aggregators import (
LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.pipeline.frames import (
from dailyai.pipeline.frames import (
ImageFrame,
SpriteFrame,
Frame,
LLMResponseEndFrame,
LLMMessagesFrame,
AudioFrame,
PipelineStartedFrame,
TTSEndFrame,
)
from pipecat.services.ai_services import AIService
from pipecat.pipeline.pipeline import Pipeline
from pipecat.transports.daily_transport import DailyTransport
from pipecat.services.open_ai_services import OpenAILLMService
from pipecat.services.elevenlabs_ai_services import ElevenLabsTTSService
from dailyai.services.ai_services import AIService
from dailyai.pipeline.pipeline import Pipeline
from dailyai.transports.daily_transport import DailyTransport
from dailyai.services.open_ai_services import OpenAILLMService
from dailyai.services.elevenlabs_ai_service import ElevenLabsTTSService
from runner import configure
@@ -30,7 +30,7 @@ from dotenv import load_dotenv
load_dotenv(override=True)
logging.basicConfig(format=f"%(levelno)s %(asctime)s %(message)s")
logger = logging.getLogger("pipecat")
logger = logging.getLogger("dailyai")
logger.setLevel(logging.DEBUG)
sprites = []
@@ -47,6 +47,7 @@ for i in range(1, 26):
flipped = sprites[::-1]
sprites.extend(flipped)
# When the bot isn't talking, show a static image of the cat listening
quiet_frame = ImageFrame(sprites[0], (1024, 576))
talking_frame = SpriteFrame(images=sprites)
@@ -70,7 +71,7 @@ class TalkingAnimation(AIService):
self._is_talking = True
else:
yield frame
elif isinstance(frame, LLMResponseEndFrame):
elif isinstance(frame, TTSEndFrame):
yield quiet_frame
yield frame
self._is_talking = False
@@ -79,6 +80,8 @@ class TalkingAnimation(AIService):
class AnimationInitializer(AIService):
def __init__(self):
super().__init__()
async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
if isinstance(frame, PipelineStartedFrame):
@@ -120,7 +123,7 @@ async def main(room_url: str, token):
messages = [
{
"role": "system",
"content": "You are Chatbot, a friendly, helpful robot. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by introducing yourself.",
"content": "You are Chatbot, a friendly, helpful robot. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by introducing yourself.",
},
]
@@ -137,6 +140,8 @@ async def main(room_url: str, token):
pre_processor=LLMUserResponseAggregator(messages),
)
transport.transcription_settings["extra"]["endpointing"] = True
transport.transcription_settings["extra"]["punctuate"] = True
await asyncio.gather(transport.run(), run_conversation())

4
examples/simple-chatbot/env.example Normal file
View File

@@ -0,0 +1,4 @@
DAILY_SAMPLE_ROOM_URL=https://yourdomain.daily.co/yourroom # (for joining the bot to the same room repeatedly for local dev)
DAILY_API_KEY=7df...
OPENAI_API_KEY=sk-PL...
ELEVENLABS_API_KEY=aeb...

BIN
examples/simple-chatbot/image.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 733 KiB

5
examples/simple-chatbot/requirements.txt Normal file
View File

@@ -0,0 +1,5 @@
python-dotenv
requests
fastapi[all]
uvicorn
dailyai[daily,openai]

58
examples/simple-chatbot/runner.py Normal file
View File

@@ -0,0 +1,58 @@
import argparse
import os
import time
import urllib
import requests
def configure():
parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample")
parser.add_argument(
"-u",
"--url",
type=str,
required=False,
help="URL of the Daily room to join")
parser.add_argument(
"-k",
"--apikey",
type=str,
required=False,
help="Daily API Key (needed to create an owner token for the room)",
)
args, unknown = parser.parse_known_args()
url = args.url or os.getenv("DAILY_SAMPLE_ROOM_URL")
key = args.apikey or os.getenv("DAILY_API_KEY")
if not url:
raise Exception(
"No Daily room specified. use the -u/--url option from the command line, or set DAILY_SAMPLE_ROOM_URL in your environment to specify a Daily room URL.")
if not key:
raise Exception("No Daily API key specified. use the -k/--apikey option from the command line, or set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.")
# Create a meeting token for the given room with an expiration 1 hour in
# the future.
room_name: str = urllib.parse.urlparse(url).path[1:]
expiration: float = time.time() + 60 * 60
res: requests.Response = requests.post(
f"https://api.daily.co/v1/meeting-tokens",
headers={
"Authorization": f"Bearer {key}"},
json={
"properties": {
"room_name": room_name,
"is_owner": True,
"exp": expiration}},
)
if res.status_code != 200:
raise Exception(
f"Failed to create meeting token: {res.status_code} {res.text}")
token: str = res.json()["token"]
return (url, token)

127
examples/simple-chatbot/server.py Normal file
View File

@@ -0,0 +1,127 @@
import os
import argparse
import subprocess
import atexit
from pathlib import Path
from typing import Optional
from fastapi import FastAPI, Request, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from fastapi.staticfiles import StaticFiles
from fastapi.responses import FileResponse, JSONResponse, RedirectResponse
from utils.daily_helpers import create_room as _create_room, get_token, get_name_from_url
MAX_BOTS_PER_ROOM = 1
# Bot sub-process dict for status reporting and concurrency control
bot_procs = {}
def cleanup():
# Clean up function, just to be extra safe
for proc in bot_procs.values():
proc.terminate()
proc.wait()
atexit.register(cleanup)
app = FastAPI()
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
@app.get("/start")
async def start_agent(request: Request):
print(f"!!! Creating room")
room_url, room_name = _create_room()
print(f"!!! Room URL: {room_url}")
# Ensure the room property is present
if not room_url:
raise HTTPException(
status_code=500,
detail="Missing 'room' property in request data. Cannot start agent without a target room!")
# Check if there is already an existing process running in this room
num_bots_in_room = sum(
1 for proc in bot_procs.values() if proc[1] == room_url and proc[0].poll() is None)
if num_bots_in_room >= MAX_BOTS_PER_ROOM:
raise HTTPException(
status_code=500, detail=f"Max bot limited reach for room: {room_url}")
# Get the token for the room
token = get_token(room_url)
if not token:
raise HTTPException(
status_code=500, detail=f"Failed to get token for room: {room_url}")
# Spawn a new agent, and join the user session
# Note: this is mostly for demonstration purposes (refer to 'deployment' in README)
try:
proc = subprocess.Popen(
[
f"python3 -m bot -u {room_url} -t {token}"
],
shell=True,
bufsize=1,
cwd=os.path.dirname(os.path.abspath(__file__))
)
bot_procs[proc.pid] = (proc, room_url)
except Exception as e:
raise HTTPException(
status_code=500, detail=f"Failed to start subprocess: {e}")
return RedirectResponse(room_url)
@app.get("/status/{pid}")
def get_status(pid: int):
# Look up the subprocess
proc = bot_procs.get(pid)
# If the subprocess doesn't exist, return an error
if not proc:
raise HTTPException(
status_code=404, detail=f"Bot with process id: {pid} not found")
# Check the status of the subprocess
if proc[0].poll() is None:
status = "running"
else:
status = "finished"
return JSONResponse({"bot_id": pid, "status": status})
if __name__ == "__main__":
import uvicorn
default_host = os.getenv("HOST", "0.0.0.0")
default_port = int(os.getenv("FAST_API_PORT", "7860"))
parser = argparse.ArgumentParser(
description="Daily Storyteller FastAPI server")
parser.add_argument("--host", type=str,
default=default_host, help="Host address")
parser.add_argument("--port", type=int,
default=default_port, help="Port number")
parser.add_argument("--reload", action="store_true",
help="Reload code on change")
config = parser.parse_args()
uvicorn.run(
"server:app",
host=config.host,
port=config.port,
reload=config.reload,
)

109
examples/simple-chatbot/utils/daily_helpers.py Normal file
View File

@@ -0,0 +1,109 @@
import urllib.parse
import os
import time
import urllib
import requests
from dotenv import load_dotenv
load_dotenv()
daily_api_path = os.getenv("DAILY_API_URL") or "api.daily.co/v1"
daily_api_key = os.getenv("DAILY_API_KEY")
def create_room() -> tuple[str, str]:
"""
Helper function to create a Daily room.
# See: https://docs.daily.co/reference/rest-api/rooms
Returns:
tuple: A tuple containing the room URL and room name.
Raises:
Exception: If the request to create the room fails or if the response does not contain the room URL or room name.
"""
room_props = {
"exp": time.time() + 60 * 60, # 1 hour
"enable_chat": True,
"enable_emoji_reactions": True,
"eject_at_room_exp": True,
"enable_prejoin_ui": False, # Important for the bot to be able to join headlessly
}
res = requests.post(
f"https://{daily_api_path}/rooms",
headers={"Authorization": f"Bearer {daily_api_key}"},
json={
"properties": room_props
},
)
if res.status_code != 200:
raise Exception(f"Unable to create room: {res.text}")
data = res.json()
room_url: str = data.get("url")
room_name: str = data.get("name")
if room_url is None or room_name is None:
raise Exception("Missing room URL or room name in response")
return room_url, room_name
def get_name_from_url(room_url: str) -> str:
"""
Extracts the name from a given room URL.
Args:
room_url (str): The URL of the room.
Returns:
str: The extracted name from the room URL.
"""
return urllib.parse.urlparse(room_url).path[1:]
def get_token(room_url: str) -> str:
"""
Retrieves a meeting token for the specified Daily room URL.
# See: https://docs.daily.co/reference/rest-api/meeting-tokens
Args:
room_url (str): The URL of the Daily room.
Returns:
str: The meeting token.
Raises:
Exception: If no room URL is specified or if no Daily API key is specified.
Exception: If there is an error creating the meeting token.
"""
if not room_url:
raise Exception(
"No Daily room specified. You must specify a Daily room in order a token to be generated.")
if not daily_api_key:
raise Exception(
"No Daily API key specified. set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.")
expiration: float = time.time() + 60 * 60
room_name = get_name_from_url(room_url)
res: requests.Response = requests.post(
f"https://{daily_api_path}/meeting-tokens",
headers={
"Authorization": f"Bearer {daily_api_key}"},
json={
"properties": {
"room_name": room_name,
"is_owner": True, # Owner tokens required for transcription
"exp": expiration}},
)
if res.status_code != 200:
raise Exception(
f"Failed to create meeting token: {res.status_code} {res.text}")
token: str = res.json()["token"]
return token

BIN
examples/starter-apps/assets/clack-short-quiet.wav
View File

Binary file not shown.

BIN
examples/starter-apps/assets/clack-short.wav
View File

Binary file not shown.

BIN
examples/starter-apps/assets/clack.wav
View File

Binary file not shown.

BIN
examples/starter-apps/assets/ding.wav
View File

Binary file not shown.

BIN
examples/starter-apps/assets/ding2.wav
View File

Binary file not shown.

BIN
examples/starter-apps/assets/grandma-listening.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.1 MiB

BIN
examples/starter-apps/assets/grandma-writing.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.1 MiB

352
examples/starter-apps/patient-intake.py
View File

@@ -1,352 +0,0 @@
import copy
import aiohttp
import asyncio
import json
import logging
import os
import re
import wave
from typing import AsyncGenerator, List
from pipecat.pipeline.opeanai_llm_aggregator import (
OpenAIAssistantContextAggregator,
OpenAIUserContextAggregator,
)
from pipecat.pipeline.pipeline import Pipeline
from pipecat.transports.daily_transport import DailyTransport
from pipecat.services.openai_llm_context import OpenAILLMContext
from pipecat.services.open_ai_services import OpenAILLMService
# from pipecat.services.deepgram_ai_services import DeepgramTTSService
from pipecat.services.elevenlabs_ai_services import ElevenLabsTTSService
from pipecat.services.fireworks_ai_services import FireworksLLMService
from pipecat.pipeline.frames import (
Frame,
LLMFunctionCallFrame,
LLMFunctionStartFrame,
AudioFrame,
)
from pipecat.pipeline.openai_frames import OpenAILLMContextFrame
from pipecat.services.ai_services import FrameLogger, AIService
from openai._types import NotGiven, NOT_GIVEN
from openai.types.chat import (
ChatCompletionToolParam,
)
from runner import configure
from dotenv import load_dotenv
load_dotenv(override=True)
logging.basicConfig(format="%(levelno)s %(asctime)s %(message)s")
logger = logging.getLogger("pipecat")
logger.setLevel(logging.DEBUG)
sounds = {}
sound_files = [
"clack-short.wav",
"clack.wav",
"clack-short-quiet.wav",
"ding.wav",
"ding2.wav",
]
script_dir = os.path.dirname(__file__)
for file in sound_files:
# Build the full path to the sound file
full_path = os.path.join(script_dir, "assets", file)
# Get the filename without the extension to use as the dictionary key
filename = os.path.splitext(os.path.basename(full_path))[0]
# Open the sound and convert it to bytes
with wave.open(full_path) as audio_file:
sounds[file] = audio_file.readframes(-1)
steps = [{"prompt": "Start by introducing yourself. Then, ask the user to confirm their identity by telling you their birthday, including the year. When they answer with their birthday, call the verify_birthday function.",
"run_async": False,
"failed": "The user provided an incorrect birthday. Ask them for their birthday again. When they answer, call the verify_birthday function.",
"tools": [{"type": "function",
"function": {"name": "verify_birthday",
"description": "Use this function to verify the user has provided their correct birthday.",
"parameters": {"type": "object",
"properties": {"birthday": {"type": "string",
"description": "The user's birthdate, including the year. The user can provide it in any format, but convert it to YYYY-MM-DD format to call this function.",
}},
},
},
}],
},
{"prompt": "Next, thank the user for confirming their identity, then ask the user to list their current prescriptions. Each prescription needs to have a medication name and a dosage. Do not call the list_prescriptions function with any unknown dosages.",
"run_async": True,
"tools": [{"type": "function",
"function": {"name": "list_prescriptions",
"description": "Once the user has provided a list of their prescription medications, call this function.",
"parameters": {"type": "object",
"properties": {"prescriptions": {"type": "array",
"items": {"type": "object",
"properties": {"medication": {"type": "string",
"description": "The medication's name",
},
"dosage": {"type": "string",
"description": "The prescription's dosage",
},
},
},
}},
},
},
}],
},
{"prompt": "Next, ask the user if they have any allergies. Once they have listed their allergies or confirmed they don't have any, call the list_allergies function.",
"run_async": True,
"tools": [{"type": "function",
"function": {"name": "list_allergies",
"description": "Once the user has provided a list of their allergies, call this function.",
"parameters": {"type": "object",
"properties": {"allergies": {"type": "array",
"items": {"type": "object",
"properties": {"name": {"type": "string",
"description": "What the user is allergic to",
}},
},
}},
},
},
}],
},
{"prompt": "Now ask the user if they have any medical conditions the doctor should know about. Once they've answered the question, call the list_conditions function.",
"run_async": True,
"tools": [{"type": "function",
"function": {"name": "list_conditions",
"description": "Once the user has provided a list of their medical conditions, call this function.",
"parameters": {"type": "object",
"properties": {"conditions": {"type": "array",
"items": {"type": "object",
"properties": {"name": {"type": "string",
"description": "The user's medical condition",
}},
},
}},
},
},
},
],
},
{"prompt": "Finally, ask the user the reason for their doctor visit today. Once they answer, call the list_visit_reasons function.",
"run_async": True,
"tools": [{"type": "function",
"function": {"name": "list_visit_reasons",
"description": "Once the user has provided a list of the reasons they are visiting a doctor today, call this function.",
"parameters": {"type": "object",
"properties": {"visit_reasons": {"type": "array",
"items": {"type": "object",
"properties": {"name": {"type": "string",
"description": "The user's reason for visiting the doctor",
}},
},
}},
},
},
}],
},
{"prompt": "Now, thank the user and end the conversation.",
"run_async": True,
"tools": [],
},
{"prompt": "",
"run_async": True,
"tools": []},
]
current_step = 0
class ChecklistProcessor(AIService):
def __init__(
self,
context: OpenAILLMContext,
llm: AIService,
tools: List[ChatCompletionToolParam] | NotGiven = NOT_GIVEN,
*args,
**kwargs,
):
super().__init__(*args, **kwargs)
self._context: OpenAILLMContext = context
self._llm = llm
self._id = "You are Jessica, an agent for a company called Tri-County Health Services. Your job is to collect important information from the user before their doctor visit. You're talking to Chad Bailey. You should address the user by their first name and be polite and professional. You're not a medical professional, so you shouldn't provide any advice. Keep your responses short. Your job is to collect information to give to a doctor. Don't make assumptions about what values to plug into functions. Ask for clarification if a user response is ambiguous."
self._acks = ["One sec.", "Let me confirm that.", "Thanks.", "OK."]
# Create an allowlist of functions that the LLM can call
self._functions = [
"verify_birthday",
"list_prescriptions",
"list_allergies",
"list_conditions",
"list_visit_reasons",
]
self._context.add_message(
{"role": "system", "content": f"{self._id} {steps[0]['prompt']}"}
)
if tools:
self._context.set_tools(tools)
def verify_birthday(self, args):
return args["birthday"] == "1983-01-01"
def list_prescriptions(self, args):
# print(f"--- Prescriptions: {args['prescriptions']}\n")
pass
def list_allergies(self, args):
# print(f"--- Allergies: {args['allergies']}\n")
pass
def list_conditions(self, args):
# print(f"--- Medical Conditions: {args['conditions']}")
pass
def list_visit_reasons(self, args):
# print(f"Visit Reasons: {args['visit_reasons']}")
pass
async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
global current_step
this_step = steps[current_step]
self._context.set_tools(this_step["tools"])
if isinstance(frame, LLMFunctionStartFrame):
print(f"... Preparing function call: {frame.function_name}")
self._function_name = frame.function_name
if this_step["run_async"]:
# Get the LLM talking about the next step before getting the rest
# of the function call completion
current_step += 1
self._context.add_message(
{"role": "system", "content": steps[current_step]["prompt"]}
)
yield OpenAILLMContextFrame(self._context)
local_context = copy.deepcopy(self._context)
local_context.set_tool_choice("none")
async for frame in llm.process_frame(
OpenAILLMContextFrame(local_context)
):
yield frame
else:
# Insert a quick response while we run the function
yield AudioFrame(sounds["ding2.wav"])
pass
elif isinstance(frame, LLMFunctionCallFrame):
if frame.function_name and frame.arguments:
print(
f"--> Calling function: {frame.function_name} with arguments:")
pretty_json = re.sub(
"\n", "\n ", json.dumps(
json.loads(
frame.arguments), indent=2))
print(f"--> {pretty_json}\n")
if frame.function_name not in self._functions:
raise Exception(
f"Unknown function.")
fn = getattr(self, frame.function_name)
result = fn(json.loads(frame.arguments))
if not this_step["run_async"]:
if result:
current_step += 1
self._context.add_message(
{"role": "system", "content": steps[current_step]["prompt"]}
)
yield OpenAILLMContextFrame(self._context)
local_context = copy.deepcopy(self._context)
local_context.set_tool_choice("none")
async for frame in llm.process_frame(
OpenAILLMContextFrame(local_context)
):
yield frame
else:
self._context.add_message(
{"role": "system", "content": this_step["failed"]}
)
yield OpenAILLMContextFrame(self._context)
local_context = copy.deepcopy(self._context)
local_context.set_tool_choice("none")
async for frame in llm.process_frame(
OpenAILLMContextFrame(local_context)
):
yield frame
print(f"<-- Verify result: {result}\n")
else:
yield frame
async def main(room_url: str, token):
async with aiohttp.ClientSession() as session:
global transport
global llm
global tts
transport = DailyTransport(
room_url,
token,
"Intake Bot",
5,
mic_enabled=True,
mic_sample_rate=16000,
camera_enabled=False,
start_transcription=True,
vad_enabled=True,
)
messages = []
llm = FireworksLLMService(
api_key=os.getenv("FIREWORKS_API_KEY"),
model="accounts/fireworks/models/firefunction-v1"
)
# tts = DeepgramTTSService(
# aiohttp_session=session,
# api_key=os.getenv("DEEPGRAM_API_KEY"),
# voice="aura-asteria-en",
# )
tts = ElevenLabsTTSService(
aiohttp_session=session,
api_key=os.getenv("ELEVENLABS_API_KEY"),
voice_id="XrExE9yKIg1WjnnlVkGX",
)
context = OpenAILLMContext(
messages=messages,
)
checklist = ChecklistProcessor(context, llm)
fl = FrameLogger("FRAME LOGGER 1:")
fl2 = FrameLogger("FRAME LOGGER 2:")
pipeline = Pipeline(processors=[fl, llm, fl2, checklist, tts])
@transport.event_handler("on_first_other_participant_joined")
async def on_first_other_participant_joined(transport, participant):
await pipeline.queue_frames([OpenAILLMContextFrame(context)])
async def handle_intake():
await transport.run_interruptible_pipeline(
pipeline,
post_processor=OpenAIAssistantContextAggregator(context),
pre_processor=OpenAIUserContextAggregator(context),
)
try:
await asyncio.gather(transport.run(), handle_intake())
except (asyncio.CancelledError, KeyboardInterrupt):
print("whoops")
transport.stop()
if __name__ == "__main__":
(url, token) = configure()
asyncio.run(main(url, token))

294
examples/starter-apps/storybot.py
View File

@@ -1,294 +0,0 @@
import aiohttp
import asyncio
import json
import random
import logging
import os
import re
import wave
from typing import AsyncGenerator
from PIL import Image
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.frame_processor import FrameProcessor
from pipecat.services.live_stream import LiveStream
from pipecat.transports.daily_transport import DailyTransport
from pipecat.services.azure_ai_services import AzureLLMService, AzureTTSService
from pipecat.services.fal_ai_services import FalImageGenService
from pipecat.services.open_ai_services import OpenAILLMService
from pipecat.services.deepgram_ai_services import DeepgramTTSService
from pipecat.services.elevenlabs_ai_services import ElevenLabsTTSService
from pipecat.pipeline.aggregators import (
LLMAssistantContextAggregator,
LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.pipeline.frames import (
EndPipeFrame,
LLMMessagesFrame,
Frame,
TextFrame,
LLMResponseEndFrame,
AudioFrame,
ImageFrame,
UserStoppedSpeakingFrame,
)
from pipecat.services.ai_services import FrameLogger, AIService
from runner import configure
from dotenv import load_dotenv
load_dotenv(override=True)
logging.basicConfig(format=f"%(levelno)s %(asctime)s %(message)s")
logger = logging.getLogger("pipecat")
logger.setLevel(logging.DEBUG)
sounds = {}
images = {}
sound_files = ["talking.wav", "listening.wav", "ding3.wav"]
image_files = ["grandma-writing.png", "grandma-listening.png"]
script_dir = os.path.dirname(__file__)
for file in sound_files:
# Build the full path to the sound file
full_path = os.path.join(script_dir, "assets", file)
# Get the filename without the extension to use as the dictionary key
filename = os.path.splitext(os.path.basename(full_path))[0]
# Open the sound and convert it to bytes
with wave.open(full_path) as audio_file:
sounds[file] = audio_file.readframes(-1)
for file in image_files:
# Build the full path to the image file
full_path = os.path.join(script_dir, "assets", file)
# Get the filename without the extension to use as the dictionary key
filename = os.path.splitext(os.path.basename(full_path))[0]
# Open the image and convert it to bytes
with Image.open(full_path) as img:
images[file] = img.tobytes()
class StoryStartFrame(TextFrame):
pass
class StoryPageFrame(TextFrame):
pass
class StoryPromptFrame(TextFrame):
pass
class StoryProcessor(FrameProcessor):
def __init__(self, messages, story):
self._messages = messages
self._text = ""
self._story = story
async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
"""
The response from the LLM service looks like:
A comment about the user's choice
[start] (when the cat starts telling parts of the story)
A sentence of the story
[break] (between each sentence/'page' of the story)
[prompt] (when the cat asks the user to make a decision)
Question about the next part of the story
1. Catch the frames that are generated by the LLM service
"""
if isinstance(frame, UserStoppedSpeakingFrame):
yield ImageFrame(images["grandma-writing.png"], (1024, 1024))
yield AudioFrame(sounds["talking.wav"])
elif isinstance(frame, TextFrame):
self._text += frame.text
if re.findall(r".*\[[sS]tart\].*", self._text):
# Then we have the intro. Send it to speech ASAP
self._text = self._text.replace("[Start]", "")
self._text = self._text.replace("[start]", "")
self._text = self._text.replace("\n", " ")
if len(self._text) > 2:
yield ImageFrame(images["grandma-writing.png"], (1024, 1024))
yield StoryStartFrame(self._text)
yield AudioFrame(sounds["ding3.wav"])
self._text = ""
elif re.findall(r".*\[[bB]reak\].*", self._text):
# Then it's a page of the story. Get an image too
self._text = self._text.replace("[Break]", "")
self._text = self._text.replace("[break]", "")
self._text = self._text.replace("\n", " ")
if len(self._text) > 2:
self._story.append(self._text)
yield StoryPageFrame(self._text)
yield AudioFrame(sounds["ding3.wav"])
self._text = ""
elif re.findall(r".*\[[pP]rompt\].*", self._text):
# Then it's question time. Flush any
# text here as a story page, then set
# the var to get to prompt mode
# cb: trying scene now
# self.handle_chunk(self._text)
self._text = self._text.replace("[Prompt]", "")
self._text = self._text.replace("[prompt]", "")
self._text = self._text.replace("\n", " ")
if len(self._text) > 2:
self._story.append(self._text)
yield StoryPageFrame(self._text)
else:
# After the prompt thing, we'll catch an LLM end to get the
# last bit
pass
elif isinstance(frame, LLMResponseEndFrame):
yield ImageFrame(images["grandma-writing.png"], (1024, 1024))
yield StoryPromptFrame(self._text)
self._text = ""
yield frame
yield ImageFrame(images["grandma-listening.png"], (1024, 1024))
yield AudioFrame(sounds["listening.wav"])
else:
# pass through everything that's not a TextFrame
yield frame
class StoryImageGenerator(FrameProcessor):
def __init__(self, story, llm, img):
self._story = story
self._llm = llm
self._img = img
async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
if isinstance(frame, StoryPageFrame):
if len(self._story) == 1:
prompt = f'You are an illustrator for a children\'s story book. Generate a prompt for DALL-E to create an illustration for the first page of the book, which reads: "{self._story[0]}"\n\n Your response should start with the phrase "Children\'s book illustration of".'
else:
prompt = f"You are an illustrator for a children's story book. Here is the story so far:\n\n\"{' '.join(self._story[:-1])}\"\n\nGenerate a prompt for DALL-E to create an illustration for the next page. Here's the sentence for the next page:\n\n\"{self._story[-1:][0]}\"\n\n Your response should start with the phrase \"Children's book illustration of\"."
msgs = [{"role": "system", "content": prompt}]
image_prompt = ""
async for f in self._llm.process_frame(LLMMessagesFrame(msgs)):
if isinstance(f, TextFrame):
image_prompt += f.text
async for f in self._img.process_frame(TextFrame(image_prompt)):
yield f
# Yield the original StoryPageFrame for basic image/audio sync
yield frame
else:
yield frame
async def main(room_url: str, token):
async with aiohttp.ClientSession() as session:
messages = [
{
"role": "system",
"content": "You are a storytelling grandma who loves to make up fantastic, fun, and educational stories for children between the ages of 5 and 10 years old. Your stories are full of friendly, magical creatures. Your stories are never scary. Each sentence of your story will become a page in a storybook. Stop after 3-4 sentences and give the child a choice to make that will influence the next part of the story. Once the child responds, start by saying something nice about the choice they made, then include [start] in your response. Include [break] after each sentence of the story. Include [prompt] between the story and the prompt.",
}
]
story = []
llm = OpenAILLMService(
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-4-1106-preview",
) # gpt-4-1106-preview
tts = ElevenLabsTTSService(
aiohttp_session=session,
api_key=os.getenv("ELEVENLABS_API_KEY"),
voice_id="Xb7hH8MSUJpSbSDYk0k2",
) # matilda
img = FalImageGenService(
params={
image_size = "1024x1024",
},
aiohttp_session=session,
key=os.getenv("FAL_KEY"),
)
lra = LLMAssistantResponseAggregator(messages)
ura = LLMUserResponseAggregator(messages)
sp = StoryProcessor(messages, story)
sig = StoryImageGenerator(story, llm, img)
transport = DailyTransport(
room_url,
token,
"Storybot",
5,
mic_enabled=True,
mic_sample_rate=16000,
camera_enabled=True,
camera_width=1024,
camera_height=1024,
start_transcription=True,
vad_enabled=True,
vad_stop_s=1.5,
)
start_story_event = asyncio.Event()
@transport.event_handler("on_first_other_participant_joined")
async def on_first_other_participant_joined(transport, participant):
start_story_event.set()
async def storytime():
await start_story_event.wait()
# We're being a bit tricky here by using a special system prompt to
# ask the user for a story topic. After their intial response, we'll
# use a different system prompt to create story pages.
intro_messages = [
{
"role": "system",
"content": "You are a storytelling grandma who loves to make up fantastic, fun, and educational stories for children between the ages of 5 and 10 years old. Your stories are full of friendly, magical creatures. Your stories are never scary. Begin by asking what a child wants you to tell a story about. Keep your reponse to only a few sentences.",
}
]
lca = LLMAssistantContextAggregator(messages)
local_pipeline = Pipeline(
[llm, lca, tts], sink=transport.send_queue)
await local_pipeline.queue_frames(
[
ImageFrame(images["grandma-listening.png"], (1024, 1024)),
LLMMessagesFrame(intro_messages),
AudioFrame(sounds["listening.wav"]),
EndPipeFrame(),
]
)
await local_pipeline.run_pipeline()
pipeline = Pipeline([llm, lca, tts, ls_sink])
pipeline.queue_frames([...])
pipeline.run()
fl = FrameLogger("### After Image Generation")
pipeline = Pipeline(
processors=[
ura,
llm,
sp,
sig,
fl,
tts,
lra,
]
)
await transport.run_pipeline(
pipeline,
)
try:
await asyncio.gather(transport.run(), storytime())
except (asyncio.CancelledError, KeyboardInterrupt):
print("whoops")
transport.stop()
if __name__ == "__main__":
(url, token) = configure()
asyncio.run(main(url, token))

2
examples/storytelling-chatbot/.dockerignore Normal file
View File

@@ -0,0 +1,2 @@
frontend/node_modules
frontend/out

158
examples/storytelling-chatbot/.gitignore vendored Normal file
View File

@@ -0,0 +1,158 @@
node_modules/
.idea/
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
.DS_Store
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
.pybuilder/
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat.pid
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
# pytype static type analyzer
.pytype/
# Cython debug symbols
cython_debug/
# PyCharm
# JetBrains specific template is maintainted in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
read.html

54
examples/storytelling-chatbot/Dockerfile Normal file
View File

@@ -0,0 +1,54 @@
FROM python:3.11-bullseye
ARG DEBIAN_FRONTEND=noninteractive
ARG USE_PERSISTENT_DATA
ENV PYTHONUNBUFFERED=1
ENV NODE_MAJOR=20
# Expose FastAPI port
ENV FAST_API_PORT=7860
EXPOSE 7860
# Install system dependencies
RUN apt-get update && apt-get install --no-install-recommends -y \
build-essential \
git \
ffmpeg \
google-perftools \
ca-certificates curl gnupg \
&& apt-get clean && rm -rf /var/lib/apt/lists/*
# Install Node.js
RUN mkdir -p /etc/apt/keyrings
RUN curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg
RUN echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_${NODE_MAJOR}.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list > /dev/null
RUN apt-get update && apt-get install nodejs -y
# Set up a new user named "user" with user ID 1000
RUN useradd -m -u 1000 user
# Set home to the user's home directory
ENV HOME=/home/user \
PATH=/home/user/.local/bin:$PATH \
PYTHONPATH=$HOME/app \
PYTHONUNBUFFERED=1
# Switch to the "user" user
USER user
# Set the working directory to the user's home directory
WORKDIR $HOME/app
# Install Python dependencies
COPY ./requirements.txt requirements.txt
RUN pip3 install --no-cache-dir --upgrade -r requirements.txt
# Copy everything else
COPY --chown=user ./src/ src/
# Copy frontend app and build
COPY --chown=user ./frontend/ frontend/
RUN cd frontend && npm install && npm run build
# Start the FastAPI server
CMD python3 src/server.py --port ${FAST_API_PORT}

83
examples/storytelling-chatbot/README.md Normal file
View File

@@ -0,0 +1,83 @@
[![Try](https://img.shields.io/badge/try_it-here-blue)](https://storytelling-chatbot.fly.dev)
# Storytelling Chatbot
<img src="image.png" width="420px">
This example shows how to build a voice-driven interactive storytelling experience.
It periodically prompts the user for input for a 'choose your own adventure' style experience.
We add visual elements to the story by generating images at lightning speed using Fal.
---
### It uses the following AI services:
**Deepgram - Speech-to-Text**
Transcribes inbound participant voice media to text.
**OpenAI (GPT4) - LLM**
Our creative writer LLM. You can see the context used to prompt it [here](src/prompts.py)
**ElevenLabs - Text-to-Speech**
Converts and streams the LLM response from text to audio
**Fal.ai - Image Generation**
Adds pictures to our story (really fast!) Prompting is quite key for style consistency, so we task the LLM to turn each story page into a short image prompt.
---
## Setup
**Install requirements**
```shell
pip install -r requirements.txt
```
**Create environment file and set variables:**
```shell
mv env.example .env
```
**Build the frontend:**
This project uses a custom frontend, which needs to built. Note: this is done automatically as part of the Docker deployment.
```shell
cd frontend/
npm install / yarn
npm run build
```
The build UI files can be found in `frontend/out`
## Running it locally
Start the API / bot manager:
`python src/server.py`
If you'd like to run a custom domain or port:
`python src/server.py --host somehost --p 7777`
➡️ Open the host URL in your browser
> [!IMPORTANT]
> Whilst working on the frontend code, please `yarn run dev`
> and open the NextJS hosted service vs. the Python server.
> (Usually localhost:3000.)
---
## Improvements to make
- Wait for track_started event to avoid rushed intro
- Show 5 minute timer on the UI

6
examples/storytelling-chatbot/env.example Normal file
View File

@@ -0,0 +1,6 @@
ELEVENLABS_API_KEY=
ELEVENLABS_VOICE_ID=
FAL_KEY=
DAILY_API_URL=api.daily.co/v1
DAILY_API_KEY=
OPENAI_API_KEY=

3
examples/storytelling-chatbot/frontend/.eslintrc.json Normal file
View File

@@ -0,0 +1,3 @@
{
"extends": "next/core-web-vitals"
}

36
examples/storytelling-chatbot/frontend/.gitignore vendored Normal file
View File

@@ -0,0 +1,36 @@
# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
# dependencies
/node_modules
/.pnp
.pnp.js
.yarn/install-state.gz
# testing
/coverage
# next.js
/.next/
/out/
# production
/build
# misc
.DS_Store
*.pem
# debug
npm-debug.log*
yarn-debug.log*
yarn-error.log*
# local env files
.env*.local
# vercel
.vercel
# typescript
*.tsbuildinfo
next-env.d.ts

36
examples/storytelling-chatbot/frontend/README.md Normal file
View File

@@ -0,0 +1,36 @@
This is a [Next.js](https://nextjs.org/) project bootstrapped with [`create-next-app`](https://github.com/vercel/next.js/tree/canary/packages/create-next-app).
## Getting Started
First, run the development server:
```bash
npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev
```
Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file.
This project uses [`next/font`](https://nextjs.org/docs/basic-features/font-optimization) to automatically optimize and load Inter, a custom Google Font.
## Learn More
To learn more about Next.js, take a look at the following resources:
- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.
You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js/) - your feedback and contributions are welcome!
## Deploy on Vercel
The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.
Check out our [Next.js deployment documentation](https://nextjs.org/docs/deployment) for more details.

BIN
examples/storytelling-chatbot/frontend/app/favicon.ico Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.1 KiB

109
examples/storytelling-chatbot/frontend/app/globals.css Normal file
View File

@@ -0,0 +1,109 @@
@tailwind base;
@tailwind components;
@tailwind utilities;
@layer base {
:root {
--background: 0 0% 100%;
--foreground: 224 71.4% 4.1%;
--card: 0 0% 100%;
--card-foreground: 224 71.4% 4.1%;
--popover: 0 0% 100%;
--popover-foreground: 224 71.4% 4.1%;
--primary: 220.9 39.3% 11%;
--primary-foreground: 210 20% 98%;
--secondary: 220 14.3% 95.9%;
--secondary-foreground: 220.9 39.3% 11%;
--muted: 220 14.3% 95.9%;
--muted-foreground: 220 8.9% 46.1%;
--accent: 220 14.3% 95.9%;
--accent-foreground: 220.9 39.3% 11%;
--destructive: 0 84.2% 60.2%;
--destructive-foreground: 210 20% 98%;
--border: 220 13% 91%;
--input: 220 13% 91%;
--ring: 224 71.4% 4.1%;
--radius: 0.5rem;
}
.dark {
--background: 224 71.4% 4.1%;
--foreground: 210 20% 98%;
--card: 224 71.4% 4.1%;
--card-foreground: 210 20% 98%;
--popover: 224 71.4% 4.1%;
--popover-foreground: 210 20% 98%;
--primary: 210 20% 98%;
--primary-foreground: 220.9 39.3% 11%;
--secondary: 215 27.9% 16.9%;
--secondary-foreground: 210 20% 98%;
--muted: 215 27.9% 16.9%;
--muted-foreground: 217.9 10.6% 64.9%;
--accent: 215 27.9% 16.9%;
--accent-foreground: 210 20% 98%;
--destructive: 0 62.8% 30.6%;
--destructive-foreground: 210 20% 98%;
--border: 215 27.9% 16.9%;
--input: 215 27.9% 16.9%;
--ring: 216 12.2% 83.9%;
}
}
@layer base {
* {
@apply border-border;
}
body {
@apply bg-background text-foreground;
}
}
body{
background: url("/bg.jpg") no-repeat center center;
background-size: cover;
}
.cardShadow{
box-shadow: 0px 124px 35px 0px rgba(0, 0, 0, 0.00), 0px 79px 32px 0px rgba(0, 0, 0, 0.01), 0px 45px 27px 0px rgba(0, 0, 0, 0.05), 0px 20px 20px 0px rgba(0, 0, 0, 0.09), 0px 5px 11px 0px rgba(0, 0, 0, 0.10);
}
@keyframes fadeInSlideUp {
0% {
opacity: 0;
transform: translateY(50px);
}
100% {
opacity: 1;
transform: translateY(0);
}
}
.fade-in {
animation: fadeIn 1s ease-out;
}
@keyframes fadeIn {
0% {
opacity: 0;
}
100% {
opacity: 1;
}
}

46
examples/storytelling-chatbot/frontend/app/layout.tsx Normal file
View File

@@ -0,0 +1,46 @@
import React from "react";
import "./globals.css";
import type { Metadata } from "next";
import { Space_Grotesk, Space_Mono } from "next/font/google";
import { cn } from "@/app/utils";
// Font
const sans = Space_Grotesk({
subsets: ["latin"],
weight: ["400", "500", "600"],
variable: "--font-sans",
});
const mono = Space_Mono({
subsets: ["latin"],
weight: ["400", "700"],
variable: "--font-mono",
});
export const metadata: Metadata = {
title: "Storytelling Chatbot - Daily AI",
description: "Built with git.new/ai",
metadataBase: new URL(process.env.SITE_URL || "http://localhost:3000"),
};
export default function RootLayout({
children,
}: Readonly<{
children: React.ReactNode;
}>) {
return (
<html lang="en">
<body
className={cn(
"min-h-screen bg-background font-sans antialiased flex flex-col",
sans.variable,
mono.variable
)}
>
<main className="flex flex-1">{children}</main>
</body>
</html>
);
}

Some files were not shown because too many files have changed in this diff Show More