moved expiry time to create_room

add CHANGELOG.md

CHANGELOG.md

@@ -0,0 +1,232 @@
+# Changelog
+
+All notable changes to **pipecat** will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.0.15] - 2024-05-15
+
+### Fixed
+
+- Quick hot fix for receiving `DailyTransportMessage`.
+
+## [0.0.14] - 2024-05-15
+
+### Added
+
+- Added `DailyTransport` event `on_participant_left`.
+
+- Added support for receiving `DailyTransportMessage`.
+
+### Fixed
+
+- Images are now resized to the size of the output camera. This was causing
+  images not being displayed.
+
+- Fixed an issue in `DailyTransport` that would not allow the input processor to
+  shutdown if no participant ever joined the room.
+
+- Fixed base transports start and stop. In some situation processors would halt
+  or not shutdown properly.
+
+## [0.0.13] - 2024-05-14
+
+### Changed
+
+- `MoondreamService` argument `model_id` is now `model`.
+
+- `VADAnalyzer` arguments have been renamed for more clarity.
+
+### Fixed
+
+- Fixed an issue with `DailyInputTransport` and `DailyOutputTransport` that
+  could cause some threads to not start properly.
+
+- Fixed `STTService`. Add `max_silence_secs` and `max_buffer_secs` to handle
+  better what's being passed to the STT service. Also add exponential smoothing
+  to the RMS.
+
+- Fixed `WhisperSTTService`. Add `no_speech_prob` to avoid garbage output text.
+
+## [0.0.12] - 2024-05-14
+
+### Added
+
+- Added `DailyTranscriptionSettings` to be able to specify transcription
+  settings much easier (e.g. language).
+
+### Other
+
+- Updated `simple-chatbot` with Spanish.
+
+- Add missing dependencies in some of the examples.
+
+## [0.0.11] - 2024-05-13
+
+### Added
+
+- Allow stopping pipeline tasks with new `StopTaskFrame`.
+
+### Changed
+
+- TTS, STT and image generation service now use `AsyncGenerator`.
+
+### Fixed
+
+- `DailyTransport`: allow registering for participant transcriptions even if
+  input transport is not initialized yet.
+
+### Other
+
+- Updated `storytelling-chatbot`.
+
+## [0.0.10] - 2024-05-13
+
+### Added
+
+- Added Intel GPU support to `MoondreamService`.
+
+- Added support for sending transport messages (e.g. to communicate with an app
+  at the other end of the transport).
+
+- Added `FrameProcessor.push_error()` to easily send an `ErrorFrame` upstream.
+
+### Fixed
+
+- Fixed Azure services (TTS and image generation).
+
+### Other
+
+- Updated `simple-chatbot`, `moondream-chatbot` and `translation-chatbot`
+  examples.
+
+## [0.0.9] - 2024-05-12
+
+### Changed
+
+Many things have changed in this version. Many of the main ideas such as frames,
+processors, services and transports are still there but some things have changed
+a bit.
+
+- `Frame`s describe the basic units for processing. For example, text, image or
+  audio frames. Or control frames to indicate a user has started or stopped
+  speaking.
+
+- `FrameProcessor`s process frames (e.g. they convert a `TextFrame` to an
+  `ImageRawFrame`) and push new frames downstream or upstream to their linked
+  peers.
+
+- `FrameProcessor`s can be linked together. The easiest wait is to use the
+  `Pipeline` which is a container for processors. Linking processors allow
+  frames to travel upstream or downstream easily.
+
+- `Transport`s are a way to send or receive frames. There can be local
+  transports (e.g. local audio or native apps), network transports
+  (e.g. websocket) or service transports (e.g. https://daily.co).
+
+- `Pipeline`s are just a processor container for other processors.
+
+- A `PipelineTask` know how to run a pipeline.
+
+- A `PipelineRunner` can run one or more tasks and it is also used, for example,
+  to capture Ctrl-C from the user.
+
+## [0.0.8] - 2024-04-11
+
+### Added
+
+- Added `FireworksLLMService`.
+
+- Added `InterimTranscriptionFrame` and enable interim results in
+  `DailyTransport` transcriptions.
+
+### Changed
+
+- `FalImageGenService` now uses new `fal_client` package.
+
+### Fixed
+
+- `FalImageGenService`: use `asyncio.to_thread` to not block main loop when
+  generating images.
+
+- Allow `TranscriptionFrame` after an end frame (transcriptions can be delayed
+  and received after `UserStoppedSpeakingFrame`).
+
+## [0.0.7] - 2024-04-10
+
+### Added
+
+- Add `use_cpu` argument to `MoondreamService`.
+
+## [0.0.6] - 2024-04-10
+
+### Added
+
+- Added `FalImageGenService.InputParams`.
+
+- Added `URLImageFrame` and `UserImageFrame`.
+
+- Added `UserImageRequestFrame` and allow requesting an image from a participant.
+
+- Added base `VisionService` and `MoondreamService`
+
+### Changed
+
+- Don't pass `image_size` to `ImageGenService`, images should have their own size.
+
+- `ImageFrame` now receives a tuple`(width,height)` to specify the size.
+
+- `on_first_other_participant_joined` now gets a participant argument.
+
+### Fixed
+
+- Check if camera, speaker and microphone are enabled before writing to them.
+
+### Performance
+
+- `DailyTransport` only subscribe to desired participant video track.
+
+## [0.0.5] - 2024-04-06
+
+### Changed
+
+- Use `camera_bitrate` and `camera_framerate`.
+
+- Increase `camera_framerate` to 30 by default.
+
+### Fixed
+
+- Fixed `LocalTransport.read_audio_frames`.
+
+## [0.0.4] - 2024-04-04
+
+### Added
+
+- Added project optional dependencies `[silero,openai,...]`.
+
+### Changed
+
+- Moved thransports to its own directory.
+
+- Use `OPENAI_API_KEY` instead of `OPENAI_CHATGPT_API_KEY`.
+
+### Fixed
+
+- Don't write to microphone/speaker if not enabled.
+
+### Other
+
+- Added live translation example.
+
+- Fix foundational examples.
+
+## [0.0.3] - 2024-03-13
+
+### Other
+
+- Added `storybot` and `chatbot` examples.
+
+## [0.0.2] - 2024-03-12
+
+Initial public release.

CHANGELOG.md.template

@@ -0,0 +1,62 @@
+# Changelog
+
+All notable changes to the **<project name>** SDK will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+Please make sure to add your changes to the appropriate categories:
+
+## [Unreleased]
+
+### Added
+
+<!-- for new functionality -->
+
+- n/a
+
+### Changed
+
+<!-- for changed functionality -->
+
+- n/a
+
+### Deprecated
+
+<!-- for soon-to-be removed functionality -->
+
+- n/a
+
+### Removed
+
+<!-- for removed functionality -->
+
+- n/a
+
+### Fixed
+
+<!-- for fixed bugs -->
+
+- n/a
+
+### Performance
+
+<!-- for performance-relevant changes -->
+
+- n/a
+
+### Security
+
+<!-- for security-relevant changes -->
+
+- n/a
+
+### Other
+
+<!-- for everything else -->
+
+- n/a
+
+## [0.1.0] - YYYY-MM-DD
+
+Initial release.

LICENSE

@@ -1,6 +1,6 @@
 BSD 2-Clause License
 
-Copyright (c) 2024, Daily
+Copyright (c) 2024, Kwindla Hultman Kramer
 
 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions are met:

README.md

@@ -1,5 +1,5 @@
 <div align="center">
- <img alt="pipecat" width="300px" height="auto" src="pipecat.png">
+ <img alt="pipecat" width="300px" height="auto" src="https://raw.githubusercontent.com/pipecat-ai/pipecat/main/pipecat.png">
 </div>
 
 # Pipecat
@@ -7,15 +7,21 @@
 [![PyPI](https://img.shields.io/pypi/v/pipecat-ai)](https://pypi.org/project/pipecat-ai) [![Discord](https://img.shields.io/discord/1239284677165056021
 )](https://discord.gg/pipecat)
 
-`pipecat` is a framework for building voice (and multimodal) conversational agents. Things like personal coaches, meeting assistants, story-telling toys for kids, customer support bots, and snarky social companions.
+`pipecat` is a framework for building voice (and multimodal) conversational agents. Things like personal coaches, meeting assistants, [story-telling toys for kids](https://storytelling-chatbot.fly.dev/), customer support bots, [intake flows](https://www.youtube.com/watch?v=lDevgsp9vn0), and snarky social companions.
 
-Build things like this:
+Take a look at some example apps:
 
-[![AI-powered voice patient intake for healthcare](https://img.youtube.com/vi/lDevgsp9vn0/0.jpg)](https://www.youtube.com/watch?v=lDevgsp9vn0)
+<p float="left">
+    <a href="https://github.com/pipecat-ai/pipecat/tree/main/examples/simple-chatbot"><img src="https://raw.githubusercontent.com/pipecat-ai/pipecat/main/examples/simple-chatbot/image.png" width="280" /></a>&nbsp;
+    <a href="https://github.com/pipecat-ai/pipecat/tree/main/examples/storytelling-chatbot"><img src="https://raw.githubusercontent.com/pipecat-ai/pipecat/main/examples/storytelling-chatbot/image.png" width="280" /></a>
+    <br/>
+    <a href="https://github.com/pipecat-ai/pipecat/tree/main/examples/translation-chatbot"><img src="https://raw.githubusercontent.com/pipecat-ai/pipecat/main/examples/translation-chatbot/image.png" width="280" /></a>&nbsp;
+    <a href="https://github.com/pipecat-ai/pipecat/tree/main/examples/moondream-chatbot"><img src="https://raw.githubusercontent.com/pipecat-ai/pipecat/main/examples/moondream-chatbot/image.png" width="280" /></a>
+</p>
 
 ## Getting started with voice agents
 
-You can get started with Pipecat running on your local machine, then move your agent processes to the cloud when you’re ready. You can also add a telephone number, image output, video input, use different LLMs, and more.
+You can get started with Pipecat running on your local machine, then move your agent processes to the cloud when you’re ready. You can also add a 📞 telephone number, 🖼️ image output, 📺 video input, use different LLMs, and more.
 
 ```shell
 # install the module
@@ -34,135 +40,125 @@ pip install "pipecat-ai[option,...]"
 Your project may or may not need these, so they're made available as optional requirements. Here is a list:
 
 - **AI services**: `anthropic`, `azure`, `fal`, `moondream`, `openai`, `playht`, `silero`, `whisper`
-- **Transports**: `daily`, `local`, `websocket`
-
-## A simple voice agent running locally
-
-If you’re doing AI-related stuff, you probably have an OpenAI API key.
-
-To generate voice output, one service that’s easy to get started with is ElevenLabs. If you don’t already have an ElevenLabs developer account, you can sign up for one [here].
-
-So let’s run a really simple agent that’s just a GPT-4 prompt, wired up to voice input and speaker output.
-
-You can change the prompt, in the code. The current prompt is “Tell me something interesting about the Roman Empire.”
-
-`cd examples/getting-started` to run the following examples …
-
-```shell
-# Talk to a local pipecat process with your voice. Specify GPT-4 as the LLM.
-
-export OPENAI_API_KEY=...
-export ELEVENLABS_API_KEY=...
-python ./local-mic.py | ./pipecat-pipes-gpt-4.py | ./local-speaker.py
-```
-
-## WebSockets instead of pipes
-
-To run your agent in the cloud, you can switch the Pipecat transport layer to use a WebSocket instead of Unix pipes.
-
-```shell
-# Talk to a local pipecat process with your voice. Specify GPT-4 as the LLM.
-
-export OPENAI_API_KEY=...
-export ELEVENLABS_API_KEY=...
-python ./local-mic-and-speaker-wss.py wss://localhost:8088
-```
-
-## WebRTC for production use
-
-WebSockets are fine for server-to-server communication or for initial development. But for production use, you’ll need client-server audio to use a protocol designed for real-time media transport. (For an explanation of the difference between WebSockets and WebRTC, see [this post.])
-
-One way to get up and running quickly with WebRTC is to sign up for a Daily developer account. Daily gives you SDKs and global infrastructure for audio (and video) routing. Every account gets 10,000 audio/video/transcription minutes free each month.
-
-Sign up [here](https://dashboard.daily.co/u/signup) and [create a room](https://docs.daily.co/reference/rest-api/rooms) in the developer Dashboard. Then run the examples, this time connecting via WebRTC instead of a WebSocket.
-
-```shell
-# 1. Run the pipecat process. Provide your Daily API key and a Daily room
-export DAILY_API_KEY=...
-export OPENAI_API_KEY=...
-export ELEVENLABS_API_KEY=...
-python pipecat-daily-gpt-4.py --daily-room https://example.daily.co/pipecat
-
-# 2. Visit the Daily room link in any web browser to talk to the pipecat process.
-#    You'll want to use a Daily SDK to embed the client-side code into your own
-#    app. But visiting the room URL in a browser is a quick way to start building
-#    agents because you can focus on just the agent code at first.
-open -a "Google Chrome" https://example.daily.co/pipecat
-```
-
-## Deploy your agent to the cloud
-Now that you’ve decoupled client and server, and have a Pipecat process that can run anywhere you can run Python, you can deploy this example agent to the cloud.
-
-`TBC`
-
-## Taking it further
-
-### Add a telephone number
-Daily supports telephone connections in addition to WebRTC streams. You can add a telephone number to your Daily room with the following REST API call. Once you’ve done that, you can call your agent on the phone.
-
-You’ll need to add a credit card to your Daily account to enable telephone numbers.
-
-`TBC`
-
-
-### Add image output
-
-Daily supports telephone connections in addition to WebRTC streams. You can add a telephone number to your Daily room with the following REST API call. Once you’ve done that, you can call your agent on the phone.
-
-You’ll need to add a credit card to your Daily account to enable telephone numbers.
-
-`TBC`
-
-### Add video output
-
-
-`TBC`
-
+- **Transports**: `local`, `websocket`, `daily`
 
 ## Code examples
 
-There are two directories of examples:
+- [foundational](https://github.com/pipecat-ai/pipecat/tree/main/examples/foundational) — small snippets that build on each other, introducing one or two concepts at a time
+- [example apps](https://github.com/pipecat-ai/pipecat/tree/main/examples/) — complete applications that you can use as starting points for development
 
-- [foundational](https://github.com/daily-co/pipecat/tree/main/examples/foundational) — examples that build on each other, introducing one or two concepts at a time
-- [starter apps](https://github.com/daily-co/pipecat/tree/main/examples/starter-apps) — complete applications that you can use as starting points for development
+## A simple voice agent running locally
 
-Before running the examples you need to install the dependencies (which will install all the dependencies to run all of the examples):
+Here is a very basic Pipecat bot that greets a user when they join a real-time session. We'll use [Daily](https://daily.co) for real-time media transport, and [ElevenLabs](https://elevenlabs.io/) for text-to-speech.
 
-```
-pip install -r {env}-requirements.txt
+```python
+#app.py
+
+import asyncio
+import aiohttp
+
+from pipecat.frames.frames import EndFrame, TextFrame
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.task import PipelineTask
+from pipecat.pipeline.runner import PipelineRunner
+from pipecat.services.elevenlabs import ElevenLabsTTSService
+from pipecat.transports.services.daily import DailyParams, DailyTransport
+
+async def main():
+  async with aiohttp.ClientSession() as session:
+    # Use Daily as a real-time media transport (WebRTC)
+    transport = DailyTransport(
+      room_url=...,
+      token=...,
+      "Bot Name",
+      DailyParams(audio_out_enabled=True))
+
+    # Use Eleven Labs for Text-to-Speech
+    tts = ElevenLabsTTSService(
+      aiohttp_session=session,
+      api_key=...,
+      voice_id=...,
+      )
+
+    # Simple pipeline that will process text to speech and output the result
+    pipeline = Pipeline([tts, transport.output()])
+
+    # Create Pipecat processor that can run one or more pipelines tasks
+    runner = PipelineRunner()
+
+    # Assign the task callable to run the pipeline
+    task = PipelineTask(pipeline)
+
+    # Register an event handler to play audio when a
+    # participant joins the transport WebRTC session
+    @transport.event_handler("on_participant_joined")
+    async def on_new_participant_joined(transport, participant):
+      participant_name = participant["info"]["userName"] or ''
+      # Queue a TextFrame that will get spoken by the TTS service (Eleven Labs)
+      await task.queue_frames([TextFrame(f"Hello there, {participant_name}!"), EndFrame()])
+
+    # Run the pipeline task
+    await runner.run(task)
+
+if __name__ == "__main__":
+  asyncio.run(main())
 ```
 
-To run the example below you need to sign up for a [free Daily account](https://dashboard.daily.co/u/signup) and create a Daily room (so you can hear the LLM talking). After that, join the room's URL directly from a browser tab and run:
+Run it with:
 
+```shell
+python app.py
 ```
-python examples/foundational/02-llm-say-one-thing.py
+
+Daily provides a prebuilt WebRTC user interface. Whilst the app is running, you can visit at `https://<yourdomain>.daily.co/<room_url>` and listen to the bot say hello!
+
+
+## WebRTC for production use
+
+WebSockets are fine for server-to-server communication or for initial development. But for production use, you’ll need client-server audio to use a protocol designed for real-time media transport. (For an explanation of the difference between WebSockets and WebRTC, see [this post.](https://www.daily.co/blog/how-to-talk-to-an-llm-with-your-voice/#webrtc))
+
+One way to get up and running quickly with WebRTC is to sign up for a Daily developer account. Daily gives you SDKs and global infrastructure for audio (and video) routing. Every account gets 10,000 audio/video/transcription minutes free each month.
+
+Sign up [here](https://dashboard.daily.co/u/signup) and [create a room](https://docs.daily.co/reference/rest-api/rooms) in the developer Dashboard.
+
+## What is VAD?
+
+Voice Activity Detection &mdash; very important for knowing when a user has finished speaking to your bot. If you are not using press-to-talk, and want Pipecat to detect when the user has finished talking, VAD is an essential component for a natural feeling conversation.
+
+Pipecast makes use of WebRTC VAD by default when using a WebRTC transport layer. Optionally, you can use Silero VAD for improved accuracy at the cost of higher CPU usage.
+
+```shell
+pip install pipecat-ai[silero]
 ```
 
+The first time your run your bot with Silero, startup may take a while whilst it downloads and caches the model in the background. You can check the progress of this in the console.
+
+
 ## Hacking on the framework itself
 
 _Note that you may need to set up a virtual environment before following the instructions below. For instance, you might need to run the following from the root of the repo:_
 
-```
+```shell
 python3 -m venv venv
 source venv/bin/activate
 ```
 
 From the root of this repo, run the following:
 
-```
+```shell
 pip install -r dev-requirements.txt -r {env}-requirements.txt
 python -m build
 ```
 
 This builds the package. To use the package locally (eg to run sample files), run
 
-```
+```shell
 pip install --editable .
 ```
 
 If you want to use this package from another directory, you can run:
 
-```
+```shell
 pip install path_to_this_repo
 ```
 
@@ -170,7 +166,7 @@ pip install path_to_this_repo
 
 From the root directory, run:
 
-```
+```shell
 pytest --doctest-modules --ignore-glob="*to_be_updated*" src tests
 ```
 
@@ -217,3 +213,9 @@ Install the
     "--max-line-length=100"
 ],
 ```
+
+## Getting help
+
+➡️ [Join our Discord](https://discord.gg/pipecat)
+
+➡️ [Reach us on X](https://x.com/pipecat_ai)

docs/README.md

@@ -8,10 +8,3 @@ Learn about the thinking behind the framework's design.
 
 See how a Frame is processed through a Transport, a Pipeline, and a series of Frame Processors.
 
-## [Example Code](examples/)
-
-The repository includes several example apps in the `examples` directory. The docs explain how they work.
-
-## [API Reference](api/)
-
-Complete documentation of the available classes and methods in the SDK.

docs/examples/01-say-one-thing.md

@@ -1,119 +0,0 @@
-# 01: Say One Thing
-
-_video here - youtube?_
-
-This example uses a text-to-speech (TTS) service to say one predefined sentence. But first, a quick overview of the general structure of these examples.
-
-## Running the demos
-
-All of the demos have something like this at the bottom of the file:
-
-```python
-if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))
-```
-
-### `configure()`
-
-The `configure()` function comes from `examples/foundational/support/runner.py`, and it allows you to configure the examples from the command line directly, or using environment variables:
-
-```bash
-python 01-say-one-thing.py -u https://YOUR_DOMAIN.daily.co/YOUR_ROOM -k YOUR_API_KEY
-# or
-DAILY_ROOM_URL=https://YOUR_DOMAIN.daily.co/YOUR_ROOM DAILY_API_KEY=YOUR_API_KEY python 01-say-one-thing.py
-# or set DAILY_ROOM_URL and DAILY_API_KEY in a .env file
-python 01-say-one-thing.py
-```
-
-You'll need a Daily account to run these demos. You can sign up for free at [daily.co](https://daily.co). Once you've signed up you can create a room from the [Dashboard](https://dashboard.daily.co/rooms), and grab [your API key](https://dashboard.daily.co/developers) while you're there.
-
-Some functionality (such as transcription) requires the bot to have owner privileges in the room. `runner.py` uses the Daily REST API to create a meeting token with owner privileges. You can learn more about meeting tokens in the [Daily docs](https://docs.daily.co/reference/rest-api/meeting-tokens).
-
-### `asyncio.run()`
-
-The AI SDK makes heavy use of Python's `asyncio` module. [This is a reasonable intro to the topic](https://builtin.com/data-science/asyncio) if you haven't worked with `asyncio` and coroutines before.
-
-You can learn a bit more about the specifics of how the Daily AI SDK uses coroutines in the [Architecture Guide](../architecture.md).
-
-## The `main()` function
-
-All of the examples have a `main()` function with a similar structure:
-
-- Configure the transport
-- Configure the AI service(s) used in the demo
-- Configure any event listeners
-- Define a processing pipeline
-- Run the example's coroutine(s)
-
-### Configuring the transport
-
-The first section of the `main()` function configures the transport object:
-
-```python
-meeting_duration_minutes = 5
-transport = DailyTransportService(
-    room_url,
-    None,
-    "Say One Thing",
-    meeting_duration_minutes,
-)
-transport.mic_enabled = True
-```
-
-The [Architecture Guide](../architecture.md) explains the transport object in more detail. In this case, we're configuring a Daily transport object and enabling the virtual microphone, so our bot can play audio.
-
-### Configuring the services
-
-As described in the [Architecture Guide](../architecture.md), 'a 'Service' is a class that processes 'Frames' as part of a 'Pipeline'. In this demo app, we'll only need one service: a text-to-speech generator. We can create an instance of the `ElevenLabsTTSService` class with this line of code:
-
-```python
-tts = ElevenLabsTTSService(aiohttp_session=session, api_key=os.getenv("ELEVENLABS_API_KEY"), voice_id=os.getenv("ELEVENLABS_VOICE_ID"))
-```
-
-You'll need to make sure and set those environment variables somewhere. The easiest way to do that is to copy the `example.env` file in the repo and rename it to `.env`, and then add your credentials to that file. `runner.py` loads the `python-dotenv` module and initializes it, making the values in that file available in the environment.
-
-### Configuring event listeners
-
-This part isn't strictly necessary for an app like this. You could include the contents of the `on_participant_joined` function directly in the body of the `main()` function, and it would run as soon as you started the script from the command line.
-
-Instead, we can use an event handler to wait to run that code until someone else joins the meeting. We'll define a function called `greet_user()`, and use the `@transport.event_handler("on_participant_joined")` decorator to tell the SDK that we want to run that function whenever a user joins the room.
-
-```python
-@transport.event_handler("on_participant_joined")
-async def greet_user(transport, participant):
-    if participant["info"]["isLocal"]:
-        return
-
-    await tts.say(
-        "Hello there, " + participant["info"]["userName"] + "!",
-        transport.send_queue,
-    )
-
-    # wait for the output queue to be empty, then leave the meeting
-    await transport.stop_when_done()
-```
-
-### Defining a processing pipeline
-
-In this example, we don't actually have much of a processing pipeline! In fact, we're doing the whole thing inside the `greet_user()` function already.
-
-Pipelines usually look like a bunch of nested calls to the `run()` or `run_to_queue()` function from different Services. In this example, we're using the `say()` function from the TTS service. This is effectively a convenience wrapper around the `run_to_queue()` function, which we'll discuss more later. It's important to `await` this function to ensure that the speech frames are queued for playback before the next line of code, because of the `stop_when_done()` function being called immediately afterward.
-
-The output of the `say()` function goes to the transport's `send_queue`. This queue is the all-important connection between the world of the Services pipeline that's generating frames asynchronously and the ordered playback of audio and visual media in the WebRTC call.
-
-### Running the coroutines
-
-In this example, we don't actually have any separate processing pipelines—everything happens as a result of an event from the transport. So we only need to run the transport's coroutine, and await its completion:
-
-```python
-await transport.run()
-```
-
-In future examples, we'll run more processes in parallel. For now, this script can run until the transport exits—which will happen based on calling `stop_when_done()` in the `greet_user()` function.
-
-## Next Steps
-
-Next, we'll start connecting multiple AI services together by building a service pipeline.
-
-## [02 - LLM Say One Thing »](02-llm-say-one-thing.md)

docs/examples/README.md

@@ -1,5 +0,0 @@
-# Daily AI SDK Examples
-
-The docs in this folder pair with the example apps located in `examples/foundational`. They are designed to serve as a quick references for building different kinds of AI apps. But the examples also build on one another, so it can be really helpful to walk through them in order.
-
-To start, you can learn about the overall structure of the examples in [01 - Say One Thing](01-say-one-thing.md).

examples/README.md

@@ -0,0 +1,84 @@
+
+
+# Pipecat — Examples
+
+## Foundational snippets
+Small snippets that build on each other, introducing one or two concepts at a time.
+
+➡️ [Take a look](https://github.com/pipecat-ai/pipecat/tree/main/examples/foundational)
+
+## Chatbot examples
+Collection of self-contained real-time voice and video AI demo applications built with Pipecat.
+
+### Quickstart
+
+Each project has its own set of dependencies and configuration variables. They intentionally avoids shared code across projects — you can grab whichever demo folder you want to work with as a starting point.
+
+We recommend you start with a virtual environment:
+
+```shell
+cd pipecat-ai/examples/simple-chatbot
+
+python -m venv venv
+
+source venv/bin/activate
+
+pip install -r requirements.txt
+```
+
+Next, follow the steps in the README for each demo.
+
+ℹ️ Make sure you `pip install -r requirements.txt` for each demo project, so you can be sure to have the necessary service dependencies that extend the functionality of Pipecat. You can read more about the framework architecture [here](https://github.com/pipecat-ai/pipecat/tree/main/docs).
+
+## Projects:
+
+| Project                                      | Description                                                                                                                                | Services                                       |
+| -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------- |
+| [Simple Chatbot](simple-chatbot)             | Basic voice-driven conversational bot. A good starting point for learning the flow of the framework.                                       | Deepgram, OpenAI, Daily, Daily Prebuilt UI            |
+| [Storytelling Chatbot](storytelling-chatbot) | Stitches together multiple third-party services to create a collaborative storytime experience.                                            | Deepgram, ElevenLabs, Open AI, Fal, Daily, Custom UI  |
+| [Translation Chatbot](translation-chatbot)   | Listens for user speech, then translates that speech to Spanish and speaks the translation back. Demonstrates multi-participant use-cases. | Deepgram, Azure, OpenAI, Daily, Daily Prebuilt UI     |
+| [Moondream Chatbot](moondream-chatbot)       | Demonstrates how to add vision capabilities to GPT4. **Note: works best with a GPU**                                                       | Deepgram, OpenAI, Moondream, Daily, Daily Prebuilt UI |
+| Function-calling Chatbot (TBC)               | A chatbot that can call functions in response to user input                                                                                | Deepgram, OpenAI, Fireworks, Daily, Daily Prebuilt UI |
+
+> [!IMPORTANT]
+> These example projects use Daily as a WebRTC transport and can be joined using their hosted Prebuilt UI.
+> It provides a quick way to join a real-time session with your bot and test your ideas without building any frontend code. If you'd like to see an example of a custom UI, try Storybot.
+
+
+## FAQ
+
+### Deployment
+
+For each of these demos we've included a `Dockerfile`. Out of the box, this should provide everything needed to get the respective demo running on a VM:
+
+```shell
+docker build username/app:tag .
+
+docker run -p 7860:7860 --env-file ./.env username/app:tag
+
+docker push ...
+```
+
+### SSL
+
+If you're working with a custom UI (such as with the Storytelling Chatbot), it's important to ensure your deployment platform supports HTTPS, as accessing user devices such as mics and webcams requires SSL.
+
+If you try to run a custom UI without SSL, you may see an error in the console telling you that `navigator` is undefined, or no devices are available.
+
+### Are these examples production ready?
+
+Yes, kind of.
+
+These demos attempt to keep things simple and are unopinionated regarding environment or scalability.
+
+We're using FastAPI to spawn a subprocess for the bots / agents — useful for small tests, but not so great for production grade apps with many concurrent users. You can see how this works in each project's `start` endpoint in `server.py`.
+
+Creating virtualized worker pools and on-demand instances is out of scope for these examples, but we hope to add some examples to this repo soon!
+
+For projects that have CUDA as a requirement, such as Moondream Chatbot, be sure to deploy to a GPU-powered platform (such as [fly.io](https://fly.io) or [Runpod](https://runpod.io).)
+
+## Getting help
+
+➡️ [Join our Discord](https://discord.gg/pipecat)
+
+➡️ [Reach us on Twitter](https://x.com/pipecat_ai)

examples/foundational/09-mirror.py

@@ -7,12 +7,9 @@
 import asyncio
 import sys
 
-from pipecat.frames.frames import AudioRawFrame, ImageRawFrame
-from pipecat.processors.filter import Filter
 from pipecat.pipeline.pipeline import Pipeline
 from pipecat.pipeline.runner import PipelineRunner
 from pipecat.pipeline.task import PipelineTask
-from pipecat.pipeline.parallel_pipeline import ParallelPipeline
 from pipecat.transports.services.daily import DailyTransport, DailyParams
 
 from runner import configure
@@ -42,13 +39,7 @@ async def main(room_url, token):
     async def on_first_participant_joined(transport, participant):
         transport.capture_participant_video(participant["id"])
 
-    # The ParallelPipeline is not really necessary here but it shows how you
-    # would process audio and video concurrently in parallel pipelines.
-    pipeline = Pipeline([transport.input(),
-                         ParallelPipeline(
-                             [Filter([AudioRawFrame])],
-                             [Filter([ImageRawFrame])]),
-                         transport.output()])
+    pipeline = Pipeline([transport.input(), transport.output()])
 
     runner = PipelineRunner()
 

examples/image-gen.py

@@ -1,122 +0,0 @@
-import argparse
-import asyncio
-import requests
-import time
-import urllib.parse
-import random
-
-from pipecat.transports.daily_transport import DailyTransport
-from pipecat.services.azure_ai_services import AzureLLMService, AzureTTSService
-from pipecat.pipeline.frames import Frame
-from pipecat.services.fal_ai_services import FalImageGenService
-
-
-async def main(room_url: str, token):
-    global transport
-    global llm
-    global tts
-
-    transport = DailyTransport(
-        room_url,
-        token,
-        "Imagebot",
-        1,
-    )
-    transport._mic_enabled = True
-    transport._camera_enabled = True
-    transport._mic_sample_rate = 16000
-    transport._camera_width = 1024
-    transport._camera_height = 1024
-
-    llm = AzureLLMService()
-    tts = AzureTTSService()
-    img = FalImageGenService()
-
-    async def handle_transcriptions():
-        print("handle_transcriptions got called")
-
-        sentence = ""
-        async for message in transport.get_transcriptions():
-            print(f"transcription message: {message}")
-            if message["session_id"] == transport._my_participant_id:
-                continue
-            finder = message["text"].find("start over")
-            print(f"finder: {finder}")
-            if finder >= 0:
-                async for audio in tts.run_tts(f"Resetting."):
-                    transport.output_queue.put(
-                        Frame(FrameType.AUDIO_FRAME, audio))
-                sentence = ""
-                continue
-            # todo: we could differentiate between transcriptions from
-            # different participants
-            sentence += f" {message['text']}"
-            print(f"sentence is now: {sentence}")
-            # TODO: Cache this audio
-            phrase = random.choice(
-                ["OK.", "Got it.", "Sure.", "You bet.", "Sure thing."])
-            async for audio in tts.run_tts(phrase):
-                transport.output_queue.put(Frame(FrameType.AUDIO_FRAME, audio))
-            img_result = img.run_image_gen(sentence, "1024x1024")
-            awaited_img = await asyncio.gather(img_result)
-            transport.output_queue.put(
-                [
-                    Frame(FrameType.IMAGE_FRAME, awaited_img[0][1]),
-                ]
-            )
-
-    @transport.event_handler("on_participant_joined")
-    async def on_participant_joined(transport, participant):
-        print(f"participant joined: {participant['info']['userName']}")
-        if participant["info"]["isLocal"]:
-            return
-        async for audio in tts.run_tts("Describe an image, and I'll create it."):
-            audio_generator = tts.run_tts(
-                f"Hello, {participant['info']['userName']}! Describe an image and I'll create it. To start over, just say 'start over'.")
-            async for audio in audio_generator:
-                transport.output_queue.put(Frame(FrameType.AUDIO_FRAME, audio))
-
-    await asyncio.gather(transport.run(), handle_transcriptions())
-
-
-if __name__ == "__main__":
-    parser = argparse.ArgumentParser(description="Simple Daily Bot Sample")
-    parser.add_argument(
-        "-u",
-        "--url",
-        type=str,
-        required=True,
-        help="URL of the Daily room to join")
-    parser.add_argument(
-        "-k",
-        "--apikey",
-        type=str,
-        required=True,
-        help="Daily API Key (needed to create token)",
-    )
-
-    args, unknown = parser.parse_known_args()
-
-    # Create a meeting token for the given room with an expiration 1 hour in
-    # the future.
-    room_name: str = urllib.parse.urlparse(args.url).path[1:]
-    expiration: float = time.time() + 60 * 60
-
-    res: requests.Response = requests.post(
-        f"https://api.daily.co/v1/meeting-tokens",
-        headers={
-            "Authorization": f"Bearer {args.apikey}"},
-        json={
-            "properties": {
-                "room_name": room_name,
-                "is_owner": True,
-                "exp": expiration}},
-    )
-
-    if res.status_code != 200:
-        raise Exception(
-            f"Failed to create meeting token: {res.status_code} {res.text}")
-
-    token: str = res.json()["token"]
-
-    asyncio.run(main(args.url, token))

examples/internal/11a-dial-out.py

@@ -1,135 +0,0 @@
-import aiohttp
-import asyncio
-import os
-import wave
-
-from pipecat.transports.daily_transport import DailyTransport
-from pipecat.services.azure_ai_services import AzureLLMService, AzureTTSService
-from pipecat.pipeline.aggregators import LLMContextAggregator
-from pipecat.services.ai_services import AIService, FrameLogger
-from pipecat.pipeline.frames import Frame, AudioFrame, LLMResponseEndFrame, LLMMessagesFrame
-from typing import AsyncGenerator
-
-from runner import configure
-
-from dotenv import load_dotenv
-load_dotenv(override=True)
-
-sounds = {}
-sound_files = [
-    'ding1.wav',
-    'ding2.wav'
-]
-
-script_dir = os.path.dirname(__file__)
-
-for file in sound_files:
-    # Build the full path to the image file
-    full_path = os.path.join(script_dir, "assets", file)
-    # Get the filename without the extension to use as the dictionary key
-    filename = os.path.splitext(os.path.basename(full_path))[0]
-    # Open the image and convert it to bytes
-    with wave.open(full_path) as audio_file:
-        sounds[file] = audio_file.readframes(-1)
-
-
-class OutboundSoundEffectWrapper(AIService):
-    def __init__(self):
-        pass
-
-    async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
-        if isinstance(frame, LLMResponseEndFrame):
-            yield AudioFrame(sounds["ding1.wav"])
-            # In case anything else up the stack needs it
-            yield frame
-        else:
-            yield frame
-
-
-class InboundSoundEffectWrapper(AIService):
-    def __init__(self):
-        pass
-
-    async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
-        if isinstance(frame, LLMMessagesFrame):
-            yield AudioFrame(sounds["ding2.wav"])
-            # In case anything else up the stack needs it
-            yield frame
-        else:
-            yield frame
-
-
-async def main(room_url: str, token, phone):
-    async with aiohttp.ClientSession() as session:
-
-        global transport
-        global llm
-        global tts
-
-        transport = DailyTransport(
-            room_url,
-            token,
-            "Respond bot",
-            300,
-        )
-        transport._mic_enabled = True
-        transport._mic_sample_rate = 16000
-        transport._camera_enabled = False
-
-        llm = AzureLLMService()
-        tts = AzureTTSService()
-
-        @transport.event_handler("on_first_other_participant_joined")
-        async def on_first_other_participant_joined(transport, participant):
-            await tts.say("Hi, I'm listening!", transport.send_queue)
-            await transport.send_queue.put(AudioFrame(sounds["ding1.wav"]))
-
-        async def handle_transcriptions():
-            messages = [
-                {"role": "system", "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio. Respond to what the user said in a creative and helpful way."},
-            ]
-
-            tma_in = LLMContextAggregator(
-                messages, "user", transport._my_participant_id
-            )
-            tma_out = LLMContextAggregator(
-                messages, "assistant", transport._my_participant_id
-            )
-            out_sound = OutboundSoundEffectWrapper()
-            in_sound = InboundSoundEffectWrapper()
-            fl = FrameLogger("LLM Out")
-            fl2 = FrameLogger("Transcription In")
-            await out_sound.run_to_queue(
-                transport.send_queue,
-                tts.run(
-                    tma_out.run(
-                        llm.run(
-                            fl2.run(
-                                in_sound.run(
-                                    tma_in.run(
-                                        transport.get_receive_frames()
-                                    )
-                                )
-                            )
-                        )
-                    )
-                )
-            )
-
-        @transport.event_handler("on_participant_joined")
-        async def pax_joined(transport, pax):
-            print(f"PARTICIPANT JOINED: {pax}")
-
-        @transport.event_handler("on_call_state_updated")
-        async def on_call_state_updated(transport, state):
-            if (state == "joined"):
-                if (phone):
-                    transport.start_recording()
-                    transport.dialout(phone)
-
-        await asyncio.gather(transport.run(), handle_transcriptions())
-
-
-if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))

examples/moondream-chatbot/.dockerignore

@@ -0,0 +1,163 @@
+# flyctl launch added from .gitignore
+# Byte-compiled / optimized / DLL files
+**/__pycache__
+**/*.py[cod]
+**/*$py.class
+
+# C extensions
+**/*.so
+
+# Distribution / packaging
+**/.Python
+**/build
+**/develop-eggs
+**/dist
+**/downloads
+**/eggs
+**/.eggs
+**/lib
+**/lib64
+**/parts
+**/sdist
+**/var
+**/wheels
+**/share/python-wheels
+**/*.egg-info
+**/.installed.cfg
+**/*.egg
+**/MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+**/*.manifest
+**/*.spec
+
+# Installer logs
+**/pip-log.txt
+**/pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+**/htmlcov
+**/.tox
+**/.nox
+**/.coverage
+**/.coverage.*
+**/.cache
+**/nosetests.xml
+**/coverage.xml
+**/*.cover
+**/*.py,cover
+**/.hypothesis
+**/.pytest_cache
+**/cover
+
+# Translations
+**/*.mo
+**/*.pot
+
+# Django stuff:
+**/*.log
+**/local_settings.py
+**/db.sqlite3
+**/db.sqlite3-journal
+
+# Flask stuff:
+**/instance
+**/.webassets-cache
+
+# Scrapy stuff:
+**/.scrapy
+
+# Sphinx documentation
+**/docs/_build
+
+# PyBuilder
+**/.pybuilder
+**/target
+
+# Jupyter Notebook
+**/.ipynb_checkpoints
+
+# IPython
+**/profile_default
+**/ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+**/.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+**/__pypackages__
+
+# Celery stuff
+**/celerybeat-schedule
+**/celerybeat.pid
+
+# SageMath parsed files
+**/*.sage.py
+
+# Environments
+**/.env
+**/.venv
+**/env
+**/venv
+**/ENV
+**/env.bak
+**/venv.bak
+
+# Spyder project settings
+**/.spyderproject
+**/.spyproject
+
+# Rope project settings
+**/.ropeproject
+
+# mkdocs documentation
+site
+
+# mypy
+**/.mypy_cache
+**/.dmypy.json
+**/dmypy.json
+
+# Pyre type checker
+**/.pyre
+
+# pytype static type analyzer
+**/.pytype
+
+# Cython debug symbols
+**/cython_debug
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+**/runpod.toml
+fly.toml

examples/moondream-chatbot/.gitignore

@@ -0,0 +1,161 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+runpod.toml

examples/moondream-chatbot/Dockerfile

@@ -0,0 +1,25 @@
+FROM ubuntu:22.04
+
+RUN apt-get update && apt-get install -y wget
+RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
+RUN dpkg -i cuda-keyring_1.1-1_all.deb
+
+RUN echo "deb [signed-by=/usr/share/keyrings/cuda-archive-keyring.gpg] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/ /" > /etc/apt/sources.list.d/cuda-ubuntu2204-x86_64.list
+
+RUN apt-get update && apt-get install -y python3 python3-pip
+RUN apt-get install -y cuda-nvcc-12-4 libcublas-12-4 libcudnn8
+
+RUN mkdir /app
+RUN mkdir /app/assets
+RUN mkdir /app/utils
+COPY *.py /app/
+COPY requirements.txt /app/
+copy assets/* /app/assets/
+copy utils/* /app/utils/
+
+WORKDIR /app
+RUN pip3 install -r requirements.txt
+
+EXPOSE 7860
+
+CMD ["python3", "server.py"]

examples/moondream-chatbot/Dockerfile.intel

@@ -0,0 +1,76 @@
+FROM ubuntu:22.04
+
+# environment variables for Intel OneAPI components
+ENV DPCPPROOT=/opt/intel/oneapi/compiler/latest
+ENV MKLROOT=/opt/intel/oneapi/mkl/latest
+ENV CCLROOT=/opt/intel/oneapi/ccl/latest
+ENV MPIROOT=/opt/intel/oneapi/mpi/latest
+
+# Install necessary dependencies
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    build-essential \
+    wget \
+    lsb-release \
+    pciutils \
+    gnupg2 \
+    python3-pip
+
+# Add Intel OneAPI repository and GPG key
+# Intel GPU repository and GPG key
+# Install Intel OneAPI components and source the environment scripts
+RUN wget -O- https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB | gpg --dearmor | tee /usr/share/keyrings/oneapi-archive-keyring.gpg > /dev/null && \
+    echo "deb [signed-by=/usr/share/keyrings/oneapi-archive-keyring.gpg] https://apt.repos.intel.com/oneapi all main" | tee /etc/apt/sources.list.d/oneAPI.list && \
+    /bin/bash -c ' \
+    . /etc/os-release && \
+    if [[ " jammy " =~ " ${VERSION_CODENAME} " ]]; then \
+        wget -qO - https://repositories.intel.com/gpu/intel-graphics.key | gpg --dearmor --output /usr/share/keyrings/intel-graphics.gpg && \
+        echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/gpu/ubuntu ${VERSION_CODENAME}/lts/2350 unified" | \
+        tee /etc/apt/sources.list.d/intel-gpu-${VERSION_CODENAME}.list && \
+        apt-get update && \
+        apt-get install -y --no-install-recommends intel-opencl-icd \
+            intel-level-zero-gpu level-zero intel-media-va-driver-non-free \
+            libmfx1 libmfxgen1 libvpl2 libegl-mesa0 libegl1-mesa \
+            libegl1-mesa-dev libgbm1 libgl1-mesa-dev libgl1-mesa-dri \
+            libglapi-mesa libgles2-mesa-dev libglx-mesa0 libigdgmm12 \
+            libxatracker2 mesa-va-drivers mesa-vdpau-drivers \
+            mesa-vulkan-drivers va-driver-all; \
+    else \
+        echo "Ubuntu version ${VERSION_CODENAME} not supported. Exiting..."; \
+        exit 1; \
+    fi' && \
+    apt-get update && apt-get install -y --no-install-recommends \
+    intel-oneapi-dpcpp-cpp-2024.1=2024.1.0-963 intel-oneapi-mkl-devel=2024.1.0-691 \
+    intel-oneapi-ccl-devel=2021.12.0-309 && \
+    apt-get clean && rm -rf /var/lib/apt/lists/* && \
+    groupadd -r render && usermod -aG render root && \
+    echo "source ${DPCPPROOT}/env/vars.sh" >> ~/.bashrc && \
+    echo "source ${MKLROOT}/env/vars.sh" >> ~/.bashrc && \
+    echo "source ${CCLROOT}/env/vars.sh" >> ~/.bashrc && \
+    echo "source ${MPIROOT}/env/vars.sh" >> ~/.bashrc && \
+    echo "export LD_LIBRARY_PATH=${MKLROOT}/lib:${DPCPPROOT}/linux/compiler/lib/intel64_lin:$LD_LIBRARY_PATH" >> ~/.bashrc
+
+WORKDIR /app
+COPY . /app
+RUN mkdir -p /app /app/assets /app/utils
+COPY *.py requirements.txt assets/* utils/* /app/
+
+# Install the Intel-specific versions of torch
+RUN python3 -m pip install --no-cache-dir -r requirements.txt && \
+    pip uninstall -y torch && \
+    pip freeze | grep 'nvidia-' | xargs pip uninstall -y && \
+    pip install --no-cache-dir --force-reinstall torch==2.1.0.post2 torchvision==0.16.0.post2 torchaudio==2.1.0.post2 \
+    intel-extension-for-pytorch==2.1.30+xpu oneccl_bind_pt==2.1.300+xpu \
+    --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/  
+
+RUN echo '#!/bin/bash\n\
+source ${DPCPPROOT}/env/vars.sh\n\
+source ${MKLROOT}/env/vars.sh\n\
+source ${CCLROOT}/env/vars.sh\n\
+source ${MPIROOT}/env/vars.sh\n\
+export LD_LIBRARY_PATH=${MKLROOT}/lib:${DPCPPROOT}/linux/compiler/lib/intel64_lin:$LD_LIBRARY_PATH\n\
+python3 server.py' > /usr/local/bin/run_app.sh && \
+    chmod +x /usr/local/bin/run_app.sh && \
+    find / -type d -name "__pycache__" -exec rm -rf {} + 
+
+EXPOSE 7860
+ENTRYPOINT ["/usr/local/bin/run_app.sh"]

examples/moondream-chatbot/README.md

@@ -0,0 +1,44 @@
+# Moondream Chatbot
+
+<img src="image.png" width="420px">
+
+
+This app connects you to a chatbot powered by GPT-4, complete with animations generated by Stable Video Diffusion. The chatbot also has vision powers thanks to [Moondream](https://moondream.ai) so you can ask it, for example, "what do you see?".
+
+ℹ️ The first time, things might take some time to get started since VAD (Voice Activity Detection) and vision models need to be downloaded.
+
+## Get started
+
+```python
+python3 -m venv venv
+source venv/bin/activate
+pip install -r requirements.txt
+
+cp env.example .env # and add your credentials
+
+```
+
+## Run the server
+
+```bash
+python server.py
+```
+
+Then, visit `http://localhost:7860/start` in your browser to start a chatbot
+session.
+
+## Build and test the Docker image
+
+```
+docker build -t moonbot .
+docker run --env-file .env -p 7860:7860 moonbot
+```
+
+### For Intel GPUs (Arc, Max and Flex series)
+
+```
+docker build -t moonbot -f Dockerfile.intel .
+docker run --env-file .env -p 7860:7860 --device /dev/dri moonbot
+```
+
+You can try to visit `http://localhost:7860/start` again.

examples/moondream-chatbot/bot.py

@@ -0,0 +1,195 @@
+import asyncio
+import aiohttp
+import os
+import sys
+
+from PIL import Image
+
+from pipecat.frames.frames import (
+    ImageRawFrame,
+    SpriteFrame,
+    Frame,
+    LLMMessagesFrame,
+    AudioRawFrame,
+    TTSStoppedFrame,
+    TextFrame,
+    UserImageRawFrame,
+    UserImageRequestFrame,
+)
+
+from pipecat.pipeline.parallel_pipeline import ParallelPipeline
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.runner import PipelineRunner
+from pipecat.pipeline.task import PipelineTask
+from pipecat.processors.aggregators.llm_response import LLMUserResponseAggregator
+from pipecat.processors.aggregators.sentence import SentenceAggregator
+from pipecat.processors.aggregators.vision_image_frame import VisionImageFrameAggregator
+from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
+from pipecat.services.elevenlabs import ElevenLabsTTSService
+from pipecat.services.moondream import MoondreamService
+from pipecat.services.openai import OpenAILLMService
+from pipecat.transports.services.daily import DailyParams, DailyTransport
+from pipecat.vad.silero import SileroVAD
+
+from runner import configure
+
+from loguru import logger
+
+from dotenv import load_dotenv
+load_dotenv(override=True)
+
+logger.remove(0)
+logger.add(sys.stderr, level="DEBUG")
+
+user_request_answer = "Let me take a look."
+
+sprites = []
+
+script_dir = os.path.dirname(__file__)
+
+for i in range(1, 26):
+    # Build the full path to the image file
+    full_path = os.path.join(script_dir, f"assets/robot0{i}.png")
+    # Get the filename without the extension to use as the dictionary key
+    # Open the image and convert it to bytes
+    with Image.open(full_path) as img:
+        sprites.append(ImageRawFrame(image=img.tobytes(), size=img.size, format=img.format))
+
+flipped = sprites[::-1]
+sprites.extend(flipped)
+
+# When the bot isn't talking, show a static image of the cat listening
+quiet_frame = sprites[0]
+talking_frame = SpriteFrame(images=sprites)
+
+
+class TalkingAnimation(FrameProcessor):
+    """
+    This class starts a talking animation when it receives an first AudioFrame,
+    and then returns to a "quiet" sprite when it sees a LLMResponseEndFrame.
+    """
+
+    def __init__(self):
+        super().__init__()
+        self._is_talking = False
+
+    async def process_frame(self, frame: Frame, direction: FrameDirection):
+        if isinstance(frame, AudioRawFrame):
+            if not self._is_talking:
+                await self.push_frame(talking_frame)
+                self._is_talking = True
+        elif isinstance(frame, TTSStoppedFrame):
+            await self.push_frame(quiet_frame)
+            self._is_talking = False
+        await self.push_frame(frame)
+
+
+class UserImageRequester(FrameProcessor):
+    def __init__(self):
+        super().__init__()
+        self.participant_id = None
+
+    def set_participant_id(self, participant_id: str):
+        self.participant_id = participant_id
+
+    async def process_frame(self, frame: Frame, direction: FrameDirection):
+        if self.participant_id and isinstance(frame, TextFrame):
+            if frame.text == user_request_answer:
+                await self.push_frame(UserImageRequestFrame(self.participant_id), FrameDirection.UPSTREAM)
+                await self.push_frame(TextFrame("Describe the image in a short sentence."))
+        elif isinstance(frame, UserImageRawFrame):
+            await self.push_frame(frame)
+
+
+class TextFilterProcessor(FrameProcessor):
+    def __init__(self, text: str):
+        super().__init__()
+        self.text = text
+
+    async def process_frame(self, frame: Frame, direction: FrameDirection):
+        if isinstance(frame, TextFrame):
+            if frame.text != self.text:
+                await self.push_frame(frame)
+        else:
+            await self.push_frame(frame)
+
+
+class ImageFilterProcessor(FrameProcessor):
+    async def process_frame(self, frame: Frame, direction: FrameDirection):
+        if not isinstance(frame, ImageRawFrame):
+            await self.push_frame(frame)
+
+
+async def main(room_url: str, token):
+    async with aiohttp.ClientSession() as session:
+        transport = DailyTransport(
+            room_url,
+            token,
+            "Chatbot",
+            DailyParams(
+                audio_in_enabled=True,
+                audio_out_enabled=True,
+                camera_out_enabled=True,
+                camera_out_width=1024,
+                camera_out_height=576,
+                transcription_enabled=True
+            )
+        )
+
+        vad = SileroVAD()
+
+        tts = ElevenLabsTTSService(
+            aiohttp_session=session,
+            api_key=os.getenv("ELEVENLABS_API_KEY"),
+            voice_id="pNInz6obpgDQGcFmaJgB",
+        )
+
+        llm = OpenAILLMService(
+            api_key=os.getenv("OPENAI_API_KEY"),
+            model="gpt-4-turbo-preview")
+
+        ta = TalkingAnimation()
+
+        sa = SentenceAggregator()
+        ir = UserImageRequester()
+        va = VisionImageFrameAggregator()
+
+        # If you run into weird description, try with use_cpu=True
+        moondream = MoondreamService()
+
+        tf = TextFilterProcessor(user_request_answer)
+        imgf = ImageFilterProcessor()
+
+        messages = [
+            {
+                "role": "system",
+                "content": f"You are Chatbot, a friendly, helpful robot. Let the user know that you are capable of chatting or describing what you see. Your goal is to demonstrate your capabilities in a succinct way. Reply with only '{user_request_answer}' if the user asks you to describe what you see. Your output will be converted to audio so never include special characters in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by introducing yourself.",
+            },
+        ]
+
+        ura = LLMUserResponseAggregator(messages)
+
+        pipeline = Pipeline([transport.input(), vad, ura, llm,
+                             ParallelPipeline(
+                                 [sa, ir, va, moondream],
+                                 [tf, imgf]),
+                             tts, ta, transport.output()])
+
+        task = PipelineTask(pipeline)
+        await task.queue_frame(quiet_frame)
+
+        @transport.event_handler("on_first_participant_joined")
+        async def on_first_participant_joined(transport, participant):
+            transport.capture_participant_transcription(participant["id"])
+            transport.capture_participant_video(participant["id"], framerate=0)
+            ir.set_participant_id(participant["id"])
+            await task.queue_frames([LLMMessagesFrame(messages)])
+
+        runner = PipelineRunner()
+
+        await runner.run(task)
+
+
+if __name__ == "__main__":
+    (url, token) = configure()
+    asyncio.run(main(url, token))

examples/moondream-chatbot/env.example

@@ -0,0 +1,4 @@
+DAILY_SAMPLE_ROOM_URL=https://yourdomain.daily.co/yourroom # (for joining the bot to the same room repeatedly for local dev)
+DAILY_API_KEY=7df...
+OPENAI_API_KEY=sk-PL...
+ELEVENLABS_API_KEY=aeb...

examples/moondream-chatbot/requirements.txt

@@ -0,0 +1,5 @@
+python-dotenv
+requests
+fastapi[all]
+uvicorn
+pipecat-ai[daily,moondream,openai,silero]

examples/moondream-chatbot/server.py

@@ -0,0 +1,124 @@
+import os
+import argparse
+import subprocess
+import atexit
+
+from fastapi import FastAPI, Request, HTTPException
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import JSONResponse, RedirectResponse
+
+from utils.daily_helpers import create_room as _create_room, get_token
+
+MAX_BOTS_PER_ROOM = 1
+
+# Bot sub-process dict for status reporting and concurrency control
+bot_procs = {}
+
+
+def cleanup():
+    # Clean up function, just to be extra safe
+    for proc in bot_procs.values():
+        proc.terminate()
+        proc.wait()
+
+
+atexit.register(cleanup)
+
+
+app = FastAPI()
+
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_credentials=True,
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+
+@app.get("/start")
+async def start_agent(request: Request):
+    print(f"!!! Creating room")
+    room_url, room_name = _create_room()
+    print(f"!!! Room URL: {room_url}")
+    # Ensure the room property is present
+    if not room_url:
+        raise HTTPException(
+            status_code=500,
+            detail="Missing 'room' property in request data. Cannot start agent without a target room!")
+
+    # Check if there is already an existing process running in this room
+    num_bots_in_room = sum(
+        1 for proc in bot_procs.values() if proc[1] == room_url and proc[0].poll() is None)
+    if num_bots_in_room >= MAX_BOTS_PER_ROOM:
+        raise HTTPException(
+            status_code=500, detail=f"Max bot limited reach for room: {room_url}")
+
+    # Get the token for the room
+    token = get_token(room_url)
+
+    if not token:
+        raise HTTPException(
+            status_code=500, detail=f"Failed to get token for room: {room_url}")
+
+    # Spawn a new agent, and join the user session
+    # Note: this is mostly for demonstration purposes (refer to 'deployment' in README)
+    try:
+        proc = subprocess.Popen(
+            [
+                f"python3 -m bot -u {room_url} -t {token}"
+            ],
+            shell=True,
+            bufsize=1,
+            cwd=os.path.dirname(os.path.abspath(__file__))
+        )
+        bot_procs[proc.pid] = (proc, room_url)
+    except Exception as e:
+        raise HTTPException(
+            status_code=500, detail=f"Failed to start subprocess: {e}")
+
+    return RedirectResponse(room_url)
+
+
+@app.get("/status/{pid}")
+def get_status(pid: int):
+    # Look up the subprocess
+    proc = bot_procs.get(pid)
+
+    # If the subprocess doesn't exist, return an error
+    if not proc:
+        raise HTTPException(
+            status_code=404, detail=f"Bot with process id: {pid} not found")
+
+    # Check the status of the subprocess
+    if proc[0].poll() is None:
+        status = "running"
+    else:
+        status = "finished"
+
+    return JSONResponse({"bot_id": pid, "status": status})
+
+
+if __name__ == "__main__":
+    import uvicorn
+
+    default_host = os.getenv("HOST", "0.0.0.0")
+    default_port = int(os.getenv("FAST_API_PORT", "7860"))
+
+    parser = argparse.ArgumentParser(
+        description="Daily Moondream FastAPI server")
+    parser.add_argument("--host", type=str,
+                        default=default_host, help="Host address")
+    parser.add_argument("--port", type=int,
+                        default=default_port, help="Port number")
+    parser.add_argument("--reload", action="store_true",
+                        help="Reload code on change")
+
+    config = parser.parse_args()
+
+    uvicorn.run(
+        "server:app",
+        host=config.host,
+        port=config.port,
+        reload=config.reload,
+    )

examples/moondream-chatbot/utils/daily_helpers.py

@@ -0,0 +1,109 @@
+
+import urllib.parse
+import os
+import time
+import urllib
+import requests
+
+from dotenv import load_dotenv
+load_dotenv()
+
+
+daily_api_path = os.getenv("DAILY_API_URL") or "api.daily.co/v1"
+daily_api_key = os.getenv("DAILY_API_KEY")
+
+
+def create_room() -> tuple[str, str]:
+    """
+    Helper function to create a Daily room.
+    # See: https://docs.daily.co/reference/rest-api/rooms
+
+    Returns:
+        tuple: A tuple containing the room URL and room name.
+
+    Raises:
+        Exception: If the request to create the room fails or if the response does not contain the room URL or room name.
+    """
+    room_props = {
+        "exp": time.time() + 60 * 60,  # 1 hour
+        "enable_chat": True,
+        "enable_emoji_reactions": True,
+        "eject_at_room_exp": True,
+        "enable_prejoin_ui": False,  # Important for the bot to be able to join headlessly
+    }
+    res = requests.post(
+        f"https://{daily_api_path}/rooms",
+        headers={"Authorization": f"Bearer {daily_api_key}"},
+        json={
+            "properties": room_props
+        },
+    )
+    if res.status_code != 200:
+        raise Exception(f"Unable to create room: {res.text}")
+
+    data = res.json()
+    room_url: str = data.get("url")
+    room_name: str = data.get("name")
+    if room_url is None or room_name is None:
+        raise Exception("Missing room URL or room name in response")
+
+    return room_url, room_name
+
+
+def get_name_from_url(room_url: str) -> str:
+    """
+    Extracts the name from a given room URL.
+
+    Args:
+        room_url (str): The URL of the room.
+
+    Returns:
+        str: The extracted name from the room URL.
+    """
+    return urllib.parse.urlparse(room_url).path[1:]
+
+
+def get_token(room_url: str) -> str:
+    """
+    Retrieves a meeting token for the specified Daily room URL.
+    # See: https://docs.daily.co/reference/rest-api/meeting-tokens
+
+    Args:
+        room_url (str): The URL of the Daily room.
+
+    Returns:
+        str: The meeting token.
+
+    Raises:
+        Exception: If no room URL is specified or if no Daily API key is specified.
+        Exception: If there is an error creating the meeting token.
+    """
+    if not room_url:
+        raise Exception(
+            "No Daily room specified. You must specify a Daily room in order a token to be generated.")
+
+    if not daily_api_key:
+        raise Exception(
+            "No Daily API key specified. set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.")
+
+    expiration: float = time.time() + 60 * 60
+    room_name = get_name_from_url(room_url)
+
+    res: requests.Response = requests.post(
+        f"https://{daily_api_path}/meeting-tokens",
+        headers={
+            "Authorization": f"Bearer {daily_api_key}"},
+        json={
+            "properties": {
+                "room_name": room_name,
+                "is_owner": True,  # Owner tokens required for transcription
+                "exp": expiration}},
+    )
+
+    if res.status_code != 200:
+        raise Exception(
+            f"Failed to create meeting token: {res.status_code} {res.text}")
+
+    token: str = res.json()["token"]
+
+    return token

examples/server/README.md

@@ -1,34 +0,0 @@
-# Server Example
-
-Use this server app to quickly host a bot on the web:
-
-```
-flask --app daily-bot-manager.py --debug run
-```
-
-It's currently configured to serve example apps defined in the APPS constant in the server file:
-
-```
-chatbot
-patient-intake
-storybot
-translator
-```
-
-Once the server is started, you can create a bot instance by opening `http://127.0.0.1:5000/start/chatbot` in a browser, and the server will do the following:
-
-- Create a new, randomly-named Daily room with `DAILY_API_KEY` from your .env file or environment
-- Start an instance of `chatbot.py` and connect it to that room
-- 301 redirect your browser to the room
-
-### Options
-
-The server supports several options, which can be set in the body of a POST request, or as params in the URL of a GET request.
-
-- `room_url` (default: none): A room URL to join. If empty, the server will create a Daily room and return the URL in the response.
-  room_properties (none): A JSON object (URL encoded if included as a GET parameter) for overriding default room creation properties, as described here: https://docs.daily.co/reference/rest-api/rooms/create-room This will be ignored if a room_url is provided.
-- `token_properties` (none): A JSON object (URL encoded if included as a GET parameter) for overriding default token properties. By default, the server creates an owner token with an expiration time of one hour.
-- `duration` (7200 seconds, or two hours): Use this property to set a time limit for the bot, as well as an expiration time for the room (if the server is creating one). This will not add an expiration time to an existing room. Expiration times in `token_properties` or `room_properties` will also take precedence over this value. You can set this property to `0` to disable timeouts, but this isn't recommended.
-- `bot_args` (none): A string containing any additional command-line args to pass to the bot.
-- `wait_for_bot` (true): Whether to wait for the bot to successfully join the room before returning a response from the server. If true, the server will start the bot script, then poll the room for up to 5 seconds to confirm the bot has joined the room. If it doesn't, the server will stop the bot and return a 500 response. If set to `false`, the server will start the bot, but immediately return a 200 response. This can be useful if the server is creating rooms for you, and you need the room URL to join the user to the room.
-- `redirect` (true): Instead of returning a 200 for GET requests, the server will return a 301 redirect to the ROOM_URL. This is handy for testing by creating a bot with a GET request directly in the browser. POST requests will never return redirects. Set to `false` to get 200 responses with info in a JSON object even for GET requests.

examples/server/daily-bot-manager.py

@@ -1,165 +0,0 @@
-import os
-import requests
-import urllib
-import subprocess
-import time
-
-from flask import Flask, jsonify, redirect, request
-from flask_cors import CORS
-
-from dotenv import load_dotenv
-load_dotenv(override=True)
-
-app = Flask(__name__)
-CORS(app)
-
-APPS = {
-    "chatbot": "../starter-apps/chatbot.py",
-    "patient-intake": "../starter-apps/patient-intake.py",
-    "storybot": "../starter-apps/storybot.py",
-    "translator": "../starter-apps/translator.py"
-}
-
-daily_api_key = os.getenv("DAILY_API_KEY")
-api_path = os.getenv("DAILY_API_PATH") or "https://api.daily.co/v1"
-
-
-def get_room_name(room_url):
-    return urllib.parse.urlparse(room_url).path[1:]
-
-
-def create_room(room_properties, exp):
-    room_props = {
-        "exp": exp,
-        "enable_chat": True,
-        "enable_emoji_reactions": True,
-        "eject_at_room_exp": True,
-        "enable_prejoin_ui": False,
-        "enable_recording": "cloud"
-    }
-    if room_properties:
-        room_props |= room_properties
-
-    res = requests.post(
-        f"{api_path}/rooms",
-        headers={"Authorization": f"Bearer {daily_api_key}"},
-        json={
-            "properties": room_props
-        },
-    )
-    if res.status_code != 200:
-        raise Exception(f"Unable to create room: {res.text}")
-
-    room_url = res.json()["url"]
-    room_name = res.json()["name"]
-    return (room_url, room_name)
-
-
-def create_token(room_name, token_properties, exp):
-    token_props = {"exp": exp, "is_owner": True}
-    if token_properties:
-        token_props |= token_properties
-    # Force the token to be limited to the room
-    token_props |= {"room_name": room_name}
-    res = requests.post(
-        f'{api_path}/meeting-tokens',
-        headers={
-            'Authorization': f'Bearer {daily_api_key}'},
-        json={
-            'properties': token_props})
-    if res.status_code != 200:
-        if res.status_code != 200:
-            raise Exception(f"Unable to create meeting token: {res.text}")
-
-    meeting_token = res.json()['token']
-    return meeting_token
-
-
-def start_bot(*, bot_path, room_url, token, bot_args, wait_for_bot):
-
-    room_name = get_room_name(room_url)
-    proc = subprocess.Popen(
-        [f"python {bot_path} -u {room_url} -t {token} -k {daily_api_key} {bot_args}"],
-        shell=True,
-        bufsize=1,
-    )
-
-    if wait_for_bot:
-        # Don't return until the bot has joined the room, but wait for at most 5
-        # seconds.
-        attempts = 0
-        while attempts < 50:
-            time.sleep(0.1)
-            attempts += 1
-            res = requests.get(
-                f"{api_path}/rooms/{room_name}/get-session-data",
-                headers={"Authorization": f"Bearer {daily_api_key}"},
-            )
-            if res.status_code == 200:
-                print(f"Took {attempts} attempts to join room {room_name}")
-                return True
-
-        # If we don't break from the loop, that means we never found the bot in the room
-        raise Exception("The bot was unable to join the room. Please try again.")
-
-    return True
-
-
-@app.route("/start/<string:botname>", methods=["GET", "POST"])
-def start(botname):
-    try:
-        if botname not in APPS:
-            raise Exception(f"Bot '{botname}' is not in the allowlist.")
-
-        bot_path = APPS[botname]
-        props = {
-            "room_url": None,
-            "room_properties": None,
-            "token_properties": None,
-            "bot_args": None,
-            "wait_for_bot": True,
-            "duration": None,
-            "redirect": True
-        }
-        props |= request.values.to_dict()  # gets URL params as well as plaintext POST body
-        try:
-            props |= request.json
-        except BaseException:
-            pass
-        if props['redirect'] == "false":
-            props['redirect'] = False
-        if props['wait_for_bot'] == "false":
-            props['wait_for_bot'] = False
-
-        duration = int(os.getenv("DAILY_BOT_DURATION") or 7200)
-        if props['duration']:
-            duration = props['duration']
-        exp = time.time() + duration
-        if (props['room_url']):
-            room_url = props['room_url']
-            try:
-                room_name = get_room_name(room_url)
-            except ValueError:
-                raise Exception(
-                    "There was a problem detecting the room name. Please double-check the value of room_url.")
-        else:
-            room_url, room_name = create_room(props['room_properties'], exp)
-        token = create_token(room_name, props['token_properties'], exp)
-        bot = start_bot(
-            room_url=room_url,
-            bot_path=bot_path,
-            token=token,
-            bot_args=props['bot_args'],
-            wait_for_bot=props['wait_for_bot'])
-
-        if props['redirect'] and request.method == "GET":
-            return redirect(room_url, 302)
-        else:
-            return jsonify({"room_url": room_url, "token": token})
-    except BaseException as e:
-        return f"There was a problem starting the bot: {e}", 500
-
-
-@app.route("/healthz")
-def health_check():
-    return "ok", 200

examples/simple-chatbot/.gitignore

@@ -0,0 +1,161 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+runpod.toml

examples/simple-chatbot/Dockerfile

@@ -0,0 +1,16 @@
+FROM python:3.10-bullseye
+
+RUN mkdir /app
+RUN mkdir /app/assets
+RUN mkdir /app/utils
+COPY *.py /app/
+COPY requirements.txt /app/
+copy assets/* /app/assets/
+copy utils/* /app/utils/
+
+WORKDIR /app
+RUN pip3 install -r requirements.txt
+
+EXPOSE 7860
+
+CMD ["python3", "server.py"]

examples/simple-chatbot/README.md

@@ -0,0 +1,37 @@
+# Simple Chatbot
+
+<img src="image.png" width="420px">
+
+This app connects you to a chatbot powered by GPT-4, complete with animations generated by Stable Video Diffusion.
+
+See a video of it in action: https://x.com/kwindla/status/1778628911817183509
+
+And a quick video walkthrough of the code: https://www.loom.com/share/13df1967161f4d24ade054e7f8753416
+
+ℹ️ The first time, things might take extra time to get started since VAD (Voice Activity Detection) model needs to be downloaded.
+
+## Get started
+
+```python
+python3 -m venv venv
+source venv/bin/activate
+pip install -r requirements.txt
+
+cp env.example .env # and add your credentials
+
+```
+
+## Run the server
+
+```bash
+python server.py
+```
+
+Then, visit `http://localhost:7860/start` in your browser to start a chatbot session.
+
+## Build and test the Docker image
+
+```
+docker build -t chatbot .
+docker run --env-file .env -p 7860:7860 chatbot
+```

examples/simple-chatbot/bot.py

@@ -0,0 +1,164 @@
+import asyncio
+import aiohttp
+import os
+import sys
+import argparse
+
+from PIL import Image
+
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.runner import PipelineRunner
+from pipecat.pipeline.task import PipelineTask
+from pipecat.processors.aggregators.llm_response import LLMUserResponseAggregator
+from pipecat.frames.frames import (
+    AudioRawFrame,
+    ImageRawFrame,
+    SpriteFrame,
+    Frame,
+    LLMMessagesFrame,
+    TTSStoppedFrame
+)
+from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
+from pipecat.services.elevenlabs import ElevenLabsTTSService
+from pipecat.services.openai import OpenAILLMService
+from pipecat.transports.services.daily import DailyParams, DailyTranscriptionSettings, DailyTransport
+from pipecat.vad.silero import SileroVAD
+
+from loguru import logger
+
+from dotenv import load_dotenv
+load_dotenv(override=True)
+
+logger.remove(0)
+logger.add(sys.stderr, level="DEBUG")
+
+sprites = []
+
+script_dir = os.path.dirname(__file__)
+
+for i in range(1, 26):
+    # Build the full path to the image file
+    full_path = os.path.join(script_dir, f"assets/robot0{i}.png")
+    # Get the filename without the extension to use as the dictionary key
+    # Open the image and convert it to bytes
+    with Image.open(full_path) as img:
+        sprites.append(ImageRawFrame(image=img.tobytes(),
+                       size=img.size, format=img.format))
+
+flipped = sprites[::-1]
+sprites.extend(flipped)
+
+# When the bot isn't talking, show a static image of the cat listening
+quiet_frame = sprites[0]
+talking_frame = SpriteFrame(images=sprites)
+
+
+class TalkingAnimation(FrameProcessor):
+    """
+    This class starts a talking animation when it receives an first AudioFrame,
+    and then returns to a "quiet" sprite when it sees a LLMResponseEndFrame.
+    """
+
+    def __init__(self):
+        super().__init__()
+        self._is_talking = False
+
+    async def process_frame(self, frame: Frame, direction: FrameDirection):
+        if isinstance(frame, AudioRawFrame):
+            if not self._is_talking:
+                await self.push_frame(talking_frame)
+                self._is_talking = True
+        elif isinstance(frame, TTSStoppedFrame):
+            await self.push_frame(quiet_frame)
+            self._is_talking = False
+
+        await self.push_frame(frame)
+
+
+async def main(room_url: str, token):
+    async with aiohttp.ClientSession() as session:
+        transport = DailyTransport(
+            room_url,
+            token,
+            "Chatbot",
+            DailyParams(
+                audio_in_enabled=True,
+                audio_out_enabled=True,
+                camera_out_enabled=True,
+                camera_out_width=1024,
+                camera_out_height=576,
+                transcription_enabled=True,
+                #
+                # Spanish
+                #
+                # transcription_settings=DailyTranscriptionSettings(
+                #     language="es",
+                #     tier="nova",
+                #     model="2-general"
+                # )
+            )
+        )
+
+        vad = SileroVAD()
+
+        tts = ElevenLabsTTSService(
+            aiohttp_session=session,
+            api_key=os.getenv("ELEVENLABS_API_KEY"),
+            #
+            # English
+            #
+            voice_id="pNInz6obpgDQGcFmaJgB",
+
+            #
+            # Spanish
+            #
+            # model="eleven_multilingual_v2",
+            # voice_id="gD1IexrzCvsXPHUuT0s3",
+        )
+
+        llm = OpenAILLMService(
+            api_key=os.getenv("OPENAI_API_KEY"),
+            model="gpt-4-turbo-preview")
+
+        messages = [
+            {
+                "role": "system",
+                #
+                # English
+                #
+                "content": "You are Chatbot, a friendly, helpful robot. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by introducing yourself.",
+
+                #
+                # Spanish
+                #
+                # "content": "Eres Chatbot, un amigable y útil robot. Tu objetivo es demostrar tus capacidades de una manera breve. Tus respuestas se convertiran a audio así que nunca no debes incluir caracteres especiales. Contesta a lo que el usuario pregunte de una manera creativa, útil y breve. Empieza por presentarte a ti mismo.",
+            },
+        ]
+
+        user_response = LLMUserResponseAggregator()
+
+        ta = TalkingAnimation()
+
+        pipeline = Pipeline([transport.input(), vad, user_response,
+                            llm, tts, ta, transport.output()])
+
+        task = PipelineTask(pipeline)
+        await task.queue_frame(quiet_frame)
+
+        @transport.event_handler("on_first_participant_joined")
+        async def on_first_participant_joined(transport, participant):
+            transport.capture_participant_transcription(participant["id"])
+            await task.queue_frames([LLMMessagesFrame(messages)])
+
+        runner = PipelineRunner()
+
+        await runner.run(task)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="Daily Storyteller Bot")
+    parser.add_argument("-u", type=str, help="Room URL")
+    parser.add_argument("-t", type=str, help="Token")
+    config = parser.parse_args()
+
+    asyncio.run(main(config.u, config.t))

examples/simple-chatbot/bot_runner.py

@@ -0,0 +1,169 @@
+from daily_helpers import create_room, get_token, check_room_url
+import os
+import argparse
+import subprocess
+import atexit
+from pathlib import Path
+from typing import Optional
+
+from fastapi import FastAPI, Request, HTTPException
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.staticfiles import StaticFiles
+from fastapi.responses import FileResponse, JSONResponse
+
+from dotenv import load_dotenv
+load_dotenv(override=True)
+
+# Bot sub-process dict for status reporting and concurrency control
+bot_procs = {}
+
+
+def cleanup():
+    # Clean up function, just to be extra safe
+    for proc in bot_procs.values():
+        proc[0].terminate()
+        proc[0].wait()
+
+
+atexit.register(cleanup)
+
+# ------------ Configuration ------------ #
+
+MAX_SESSION_TIME = + 5 * 60  # 5 minutes
+BOT_CAN_IDLE = True
+SERVE_STATIC = True
+STATIC_DIR = "../web-ui/dist"
+STATIC_ROUTE = "/static"
+STATIC_INDEX = "index.html"
+USE_OPEN_MIC = True  # Can the user freely talk, or do they need to wait their turn?
+
+# ----------------- API ----------------- #
+
+app = FastAPI()
+
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_credentials=True,
+    allow_methods=["*"],
+    allow_headers=["*"]
+)
+
+# Optionally serve client static files
+if SERVE_STATIC:
+    app.mount(STATIC_ROUTE, StaticFiles(
+        directory=STATIC_DIR, html=True), name="static")
+
+    @app.get("/{path_name:path}", response_class=FileResponse)
+    async def catch_all(path_name: Optional[str] = ""):
+        if path_name == "":
+            return FileResponse(f"{STATIC_DIR}/{STATIC_INDEX}")
+
+        file_path = Path(STATIC_DIR) / (path_name or "")
+
+        if file_path.is_file():
+            return file_path
+
+        html_file_path = file_path.with_suffix(".html")
+        if html_file_path.is_file():
+            return FileResponse(html_file_path)
+
+        raise HTTPException(
+            status_code=404, detail="Page not found")
+
+
+@app.post("/start_bot")
+async def start_bot(request: Request) -> JSONResponse:
+    try:
+        data = await request.json()
+        # Is this a webhook creation request?
+        if "test" in data:
+            return JSONResponse({"test": True})
+    except Exception:
+        pass
+
+    # Use specified room URL, or create a new one if not specified
+    room_url = os.getenv("DAILY_SAMPLE_ROOM_URL", None)
+
+    if not room_url:
+        try:
+            room_url = create_room(MAX_SESSION_TIME)
+        except Exception:
+            raise HTTPException(
+                status_code=500,
+                detail="Unable to provision room")
+    else:
+        # Check passed room URL exists
+        try:
+            check_room_url(room_url)
+        except Exception:
+            raise HTTPException(
+                status_code=500, detail=f"Room not found: {room_url}")
+
+    # Give the agent a token to join the session
+    token = get_token(room_url)
+
+    if not room_url or not token:
+        raise HTTPException(
+            status_code=500, detail=f"Failed to get token for room: {room_url}")
+
+    # Spawn a new agent, and join the user session
+    # Note: this is mostly for demonstration purposes (refer to 'deployment' in README)
+
+    # @TODO: Spawn a new fly machine here...
+    try:
+        proc = subprocess.Popen(
+            [
+                f"python3 -m bot -u {room_url} -t {token}"
+            ],
+            shell=True,
+            bufsize=1,
+            cwd=os.path.dirname(os.path.abspath(__file__))
+        )
+        bot_procs[proc.pid] = (proc, room_url)
+    except Exception as e:
+        raise HTTPException(
+            status_code=500, detail=f"Failed to start subprocess: {e}")
+
+     # Grab a token for the user to join with
+    user_token = get_token(room_url)
+
+    return JSONResponse({
+        "bot_id": proc.pid,
+        "room_url": room_url,
+        "token": user_token,
+        "config": {"open_mic": USE_OPEN_MIC}})
+
+
+# ----------------- Main ----------------- #
+
+if __name__ == "__main__":
+    # Check environment variables
+    required_env_vars = ['OPENAI_API_KEY', 'DAILY_API_KEY',
+                         'ELEVENLABS_VOICE_ID', 'ELEVENLABS_API_KEY']
+    for env_var in required_env_vars:
+        if env_var not in os.environ:
+            raise Exception(f"Missing environment variable: {env_var}.")
+
+    parser = argparse.ArgumentParser(description="Pipecat Bot Runner")
+    parser.add_argument("--host", type=str,
+                        default=os.getenv("HOST", "localhost"), help="Host address")
+    parser.add_argument("--port", type=int,
+                        default=os.getenv("PORT", 7860), help="Port number")
+    parser.add_argument("--reload", action="store_true",
+                        default=True, help="Reload code on change")
+
+    config = parser.parse_args()
+
+    try:
+        import uvicorn
+
+        uvicorn.run(
+            "bot_runner:app",
+            host=config.host,
+            port=config.port,
+            reload=config.reload
+        )
+
+    except KeyboardInterrupt:
+        print("Pipecat runner shutting down...")

examples/simple-chatbot/daily_helpers.py

@@ -0,0 +1,135 @@
+
+from re import X
+import urllib.parse
+import os
+import time
+import urllib
+import requests
+
+from dotenv import load_dotenv
+load_dotenv()
+
+
+daily_api_path = os.getenv("DAILY_API_URL", "api.daily.co/v1")
+daily_api_key = os.getenv("DAILY_API_KEY")
+
+
+def create_room(expiry_time=5 * 60) -> tuple[str, str]:
+    """
+    Helper function to create a Daily room.
+    # See: https://docs.daily.co/reference/rest-api/rooms
+
+    Returns:
+        tuple: A tuple containing the room URL and room name.
+
+    Raises:
+        Exception: If the request to create the room fails or if the response does not contain the room URL or room name.
+    """
+    room_props = {
+        "exp": time.time() * expiry_time,
+        "enable_chat": True,
+        "enable_emoji_reactions": True,
+        "eject_at_room_exp": True,
+        "enable_prejoin_ui": False,  # Important for the bot to be able to join headlessly
+    }
+    res = requests.post(
+        f"https://{daily_api_path}/rooms",
+        headers={"Authorization": f"Bearer {daily_api_key}"},
+        json={
+            "properties": room_props
+        },
+    )
+    if res.status_code != 200:
+        raise Exception(f"Unable to create room: {res.text}")
+
+    data = res.json()
+    room_url: str = data.get("url")
+    room_name: str = data.get("name")
+    if room_url is None or room_name is None:
+        raise Exception("Missing room URL or room name in response")
+
+    return room_url, room_name
+
+
+def check_room_url(room_url: str) -> bool:
+    """
+    Checks if a room exists in Daily.
+    # See: https://docs.daily.co/reference/rest-api/rooms/get-room-config
+
+    Args:
+        room_name (str): The url of the room to check for
+
+    Returns:
+        bool: True if 200 OK, Exception otherwise.
+    """
+
+    room_name = get_name_from_url(room_url)
+
+    res: requests.Response = requests.get(
+        f"https://{daily_api_path}/rooms/{room_name}",
+        headers={"Authorization": f"Bearer {daily_api_key}"}
+    )
+
+    if res.status_code != 200:
+        raise Exception(f"Room not found: {room_name}")
+
+    return True
+
+
+def get_name_from_url(room_url: str) -> str:
+    """
+    Extracts the name from a given room URL.
+
+    Args:
+        room_url (str): The URL of the room.
+
+    Returns:
+        str: The extracted name from the room URL.
+    """
+    return urllib.parse.urlparse(room_url).path[1:]
+
+
+def get_token(room_url: str) -> str:
+    """
+    Retrieves a meeting token for the specified Daily room URL.
+    # See: https://docs.daily.co/reference/rest-api/meeting-tokens
+
+    Args:
+        room_url (str): The URL of the Daily room.
+
+    Returns:
+        str: The meeting token.
+
+    Raises:
+        Exception: If no room URL is specified or if no Daily API key is specified.
+        Exception: If there is an error creating the meeting token.
+    """
+    if not room_url:
+        raise Exception(
+            "No Daily room specified. You must specify a Daily room in order a token to be generated.")
+
+    if not daily_api_key:
+        raise Exception(
+            "No Daily API key specified. set DAILY_API_KEY in your environment to specify a Daily API key, available from https://dashboard.daily.co/developers.")
+
+    expiration: float = time.time() + 60 * 60
+    room_name = get_name_from_url(room_url)
+
+    res: requests.Response = requests.post(
+        f"https://{daily_api_path}/meeting-tokens",
+        headers={
+            "Authorization": f"Bearer {daily_api_key}"},
+        json={
+            "properties": {
+                "room_name": room_name,
+                "is_owner": True,  # Owner tokens required for transcription
+                "exp": expiration}},
+    )
+
+    if res.status_code != 200:
+        raise Exception(
+            f"Failed to create meeting token: {res.status_code} {res.text}")
+
+    token: str = res.json()["token"]
+
+    return token

examples/simple-chatbot/env.example

@@ -0,0 +1,4 @@
+DAILY_SAMPLE_ROOM_URL=https://yourdomain.daily.co/yourroom # (for joining the bot to the same room repeatedly for local dev)
+DAILY_API_KEY=7df...
+OPENAI_API_KEY=sk-PL...
+ELEVENLABS_API_KEY=aeb...

examples/simple-chatbot/requirements.txt

@@ -0,0 +1,7 @@
+pipecat-ai[daily,openai,fal]
+fastapi
+uvicorn
+requests
+python-dotenv
+loguru
+requests

examples/starter-apps/chatbot.py

@@ -1,145 +0,0 @@
-import asyncio
-import aiohttp
-import logging
-import os
-from PIL import Image
-from typing import AsyncGenerator
-
-from pipecat.pipeline.aggregators import (
-    LLMAssistantResponseAggregator,
-    LLMUserResponseAggregator,
-)
-from pipecat.pipeline.frames import (
-    ImageFrame,
-    SpriteFrame,
-    Frame,
-    LLMResponseEndFrame,
-    LLMMessagesFrame,
-    AudioFrame,
-    PipelineStartedFrame,
-)
-from pipecat.services.ai_services import AIService
-from pipecat.pipeline.pipeline import Pipeline
-from pipecat.transports.daily_transport import DailyTransport
-from pipecat.services.open_ai_services import OpenAILLMService
-from pipecat.services.elevenlabs_ai_services import ElevenLabsTTSService
-
-from runner import configure
-
-from dotenv import load_dotenv
-load_dotenv(override=True)
-
-logging.basicConfig(format=f"%(levelno)s %(asctime)s %(message)s")
-logger = logging.getLogger("pipecat")
-logger.setLevel(logging.DEBUG)
-
-sprites = []
-
-script_dir = os.path.dirname(__file__)
-
-for i in range(1, 26):
-    # Build the full path to the image file
-    full_path = os.path.join(script_dir, f"assets/robot0{i}.png")
-    # Get the filename without the extension to use as the dictionary key
-    # Open the image and convert it to bytes
-    with Image.open(full_path) as img:
-        sprites.append(img.tobytes())
-
-flipped = sprites[::-1]
-sprites.extend(flipped)
-# When the bot isn't talking, show a static image of the cat listening
-quiet_frame = ImageFrame(sprites[0], (1024, 576))
-talking_frame = SpriteFrame(images=sprites)
-
-
-class TalkingAnimation(AIService):
-    """
-    This class starts a talking animation when it receives an first AudioFrame,
-    and then returns to a "quiet" sprite when it sees a LLMResponseEndFrame.
-    """
-
-    def __init__(self):
-        super().__init__()
-        self._is_talking = False
-
-    async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
-        if isinstance(frame, AudioFrame):
-            if not self._is_talking:
-                yield talking_frame
-                yield frame
-                self._is_talking = True
-            else:
-                yield frame
-        elif isinstance(frame, LLMResponseEndFrame):
-            yield quiet_frame
-            yield frame
-            self._is_talking = False
-        else:
-            yield frame
-
-
-class AnimationInitializer(AIService):
-
-    async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
-        if isinstance(frame, PipelineStartedFrame):
-            yield quiet_frame
-            yield frame
-        else:
-            yield frame
-
-
-async def main(room_url: str, token):
-    async with aiohttp.ClientSession() as session:
-        transport = DailyTransport(
-            room_url,
-            token,
-            "Chatbot",
-            duration_minutes=5,
-            start_transcription=True,
-            mic_enabled=True,
-            mic_sample_rate=16000,
-            camera_enabled=True,
-            camera_width=1024,
-            camera_height=576,
-            vad_enabled=True,
-        )
-
-        tts = ElevenLabsTTSService(
-            aiohttp_session=session,
-            api_key=os.getenv("ELEVENLABS_API_KEY"),
-            voice_id="pNInz6obpgDQGcFmaJgB",
-        )
-
-        llm = OpenAILLMService(
-            api_key=os.getenv("OPENAI_API_KEY"),
-            model="gpt-4-turbo-preview")
-
-        ta = TalkingAnimation()
-        ai = AnimationInitializer()
-        pipeline = Pipeline([ai, llm, tts, ta])
-        messages = [
-            {
-                "role": "system",
-                "content": "You are Chatbot, a friendly, helpful robot. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by introducing yourself.",
-            },
-        ]
-
-        @transport.event_handler("on_first_other_participant_joined")
-        async def on_first_other_participant_joined(transport, participant):
-            print(f"!!! in here, pipeline.source is {pipeline.source}")
-            await pipeline.queue_frames([LLMMessagesFrame(messages)])
-
-        async def run_conversation():
-
-            await transport.run_interruptible_pipeline(
-                pipeline,
-                post_processor=LLMAssistantResponseAggregator(messages),
-                pre_processor=LLMUserResponseAggregator(messages),
-            )
-
-        await asyncio.gather(transport.run(), run_conversation())
-
-
-if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))

examples/starter-apps/patient-intake.py

@@ -1,352 +0,0 @@
-import copy
-import aiohttp
-import asyncio
-import json
-import logging
-import os
-import re
-import wave
-from typing import AsyncGenerator, List
-from pipecat.pipeline.opeanai_llm_aggregator import (
-    OpenAIAssistantContextAggregator,
-    OpenAIUserContextAggregator,
-)
-
-from pipecat.pipeline.pipeline import Pipeline
-from pipecat.transports.daily_transport import DailyTransport
-from pipecat.services.openai_llm_context import OpenAILLMContext
-from pipecat.services.open_ai_services import OpenAILLMService
-# from pipecat.services.deepgram_ai_services import DeepgramTTSService
-from pipecat.services.elevenlabs_ai_services import ElevenLabsTTSService
-from pipecat.services.fireworks_ai_services import FireworksLLMService
-from pipecat.pipeline.frames import (
-    Frame,
-    LLMFunctionCallFrame,
-    LLMFunctionStartFrame,
-    AudioFrame,
-)
-from pipecat.pipeline.openai_frames import OpenAILLMContextFrame
-from pipecat.services.ai_services import FrameLogger, AIService
-from openai._types import NotGiven, NOT_GIVEN
-
-from openai.types.chat import (
-    ChatCompletionToolParam,
-)
-
-from runner import configure
-
-from dotenv import load_dotenv
-load_dotenv(override=True)
-
-logging.basicConfig(format="%(levelno)s %(asctime)s %(message)s")
-logger = logging.getLogger("pipecat")
-logger.setLevel(logging.DEBUG)
-
-sounds = {}
-sound_files = [
-    "clack-short.wav",
-    "clack.wav",
-    "clack-short-quiet.wav",
-    "ding.wav",
-    "ding2.wav",
-]
-
-script_dir = os.path.dirname(__file__)
-
-for file in sound_files:
-    # Build the full path to the sound file
-    full_path = os.path.join(script_dir, "assets", file)
-    # Get the filename without the extension to use as the dictionary key
-    filename = os.path.splitext(os.path.basename(full_path))[0]
-    # Open the sound and convert it to bytes
-    with wave.open(full_path) as audio_file:
-        sounds[file] = audio_file.readframes(-1)
-
-
-steps = [{"prompt": "Start by introducing yourself. Then, ask the user to confirm their identity by telling you their birthday, including the year. When they answer with their birthday, call the verify_birthday function.",
-          "run_async": False,
-          "failed": "The user provided an incorrect birthday. Ask them for their birthday again. When they answer, call the verify_birthday function.",
-          "tools": [{"type": "function",
-                     "function": {"name": "verify_birthday",
-                                  "description": "Use this function to verify the user has provided their correct birthday.",
-                                  "parameters": {"type": "object",
-                                                 "properties": {"birthday": {"type": "string",
-                                                                             "description": "The user's birthdate, including the year. The user can provide it in any format, but convert it to YYYY-MM-DD format to call this function.",
-                                                                             }},
-                                                 },
-                                  },
-                     }],
-          },
-         {"prompt": "Next, thank the user for confirming their identity, then ask the user to list their current prescriptions. Each prescription needs to have a medication name and a dosage. Do not call the list_prescriptions function with any unknown dosages.",
-          "run_async": True,
-          "tools": [{"type": "function",
-                     "function": {"name": "list_prescriptions",
-                                  "description": "Once the user has provided a list of their prescription medications, call this function.",
-                                  "parameters": {"type": "object",
-                                                 "properties": {"prescriptions": {"type": "array",
-                                                                                  "items": {"type": "object",
-                                                                                            "properties": {"medication": {"type": "string",
-                                                                                                                          "description": "The medication's name",
-                                                                                                                          },
-                                                                                                           "dosage": {"type": "string",
-                                                                                                                      "description": "The prescription's dosage",
-                                                                                                                      },
-                                                                                                           },
-                                                                                            },
-                                                                                  }},
-                                                 },
-                                  },
-                     }],
-          },
-         {"prompt": "Next, ask the user if they have any allergies. Once they have listed their allergies or confirmed they don't have any, call the list_allergies function.",
-          "run_async": True,
-          "tools": [{"type": "function",
-                     "function": {"name": "list_allergies",
-                                  "description": "Once the user has provided a list of their allergies, call this function.",
-                                  "parameters": {"type": "object",
-                                                 "properties": {"allergies": {"type": "array",
-                                                                              "items": {"type": "object",
-                                                                                        "properties": {"name": {"type": "string",
-                                                                                                                "description": "What the user is allergic to",
-                                                                                                                }},
-                                                                                        },
-                                                                              }},
-                                                 },
-                                  },
-                     }],
-          },
-         {"prompt": "Now ask the user if they have any medical conditions the doctor should know about. Once they've answered the question, call the list_conditions function.",
-          "run_async": True,
-          "tools": [{"type": "function",
-                     "function": {"name": "list_conditions",
-                                  "description": "Once the user has provided a list of their medical conditions, call this function.",
-                                  "parameters": {"type": "object",
-                                                 "properties": {"conditions": {"type": "array",
-                                                                               "items": {"type": "object",
-                                                                                         "properties": {"name": {"type": "string",
-                                                                                                                 "description": "The user's medical condition",
-                                                                                                                 }},
-                                                                                         },
-                                                                               }},
-                                                 },
-                                  },
-                     },
-                    ],
-          },
-         {"prompt": "Finally, ask the user the reason for their doctor visit today. Once they answer, call the list_visit_reasons function.",
-          "run_async": True,
-          "tools": [{"type": "function",
-                     "function": {"name": "list_visit_reasons",
-                                  "description": "Once the user has provided a list of the reasons they are visiting a doctor today, call this function.",
-                                  "parameters": {"type": "object",
-                                                 "properties": {"visit_reasons": {"type": "array",
-                                                                                  "items": {"type": "object",
-                                                                                            "properties": {"name": {"type": "string",
-                                                                                                                    "description": "The user's reason for visiting the doctor",
-                                                                                                                    }},
-                                                                                            },
-                                                                                  }},
-                                                 },
-                                  },
-                     }],
-          },
-         {"prompt": "Now, thank the user and end the conversation.",
-          "run_async": True,
-          "tools": [],
-          },
-         {"prompt": "",
-          "run_async": True,
-          "tools": []},
-         ]
-current_step = 0
-
-
-class ChecklistProcessor(AIService):
-
-    def __init__(
-        self,
-        context: OpenAILLMContext,
-        llm: AIService,
-        tools: List[ChatCompletionToolParam] | NotGiven = NOT_GIVEN,
-        *args,
-        **kwargs,
-    ):
-        super().__init__(*args, **kwargs)
-        self._context: OpenAILLMContext = context
-        self._llm = llm
-        self._id = "You are Jessica, an agent for a company called Tri-County Health Services. Your job is to collect important information from the user before their doctor visit. You're talking to Chad Bailey. You should address the user by their first name and be polite and professional. You're not a medical professional, so you shouldn't provide any advice. Keep your responses short. Your job is to collect information to give to a doctor. Don't make assumptions about what values to plug into functions. Ask for clarification if a user response is ambiguous."
-        self._acks = ["One sec.", "Let me confirm that.", "Thanks.", "OK."]
-
-        # Create an allowlist of functions that the LLM can call
-        self._functions = [
-            "verify_birthday",
-            "list_prescriptions",
-            "list_allergies",
-            "list_conditions",
-            "list_visit_reasons",
-        ]
-
-        self._context.add_message(
-            {"role": "system", "content": f"{self._id} {steps[0]['prompt']}"}
-        )
-
-        if tools:
-            self._context.set_tools(tools)
-
-    def verify_birthday(self, args):
-        return args["birthday"] == "1983-01-01"
-
-    def list_prescriptions(self, args):
-        # print(f"--- Prescriptions: {args['prescriptions']}\n")
-        pass
-
-    def list_allergies(self, args):
-        # print(f"--- Allergies: {args['allergies']}\n")
-        pass
-
-    def list_conditions(self, args):
-        # print(f"--- Medical Conditions: {args['conditions']}")
-        pass
-
-    def list_visit_reasons(self, args):
-        # print(f"Visit Reasons: {args['visit_reasons']}")
-        pass
-
-    async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
-        global current_step
-        this_step = steps[current_step]
-        self._context.set_tools(this_step["tools"])
-        if isinstance(frame, LLMFunctionStartFrame):
-            print(f"... Preparing function call: {frame.function_name}")
-            self._function_name = frame.function_name
-            if this_step["run_async"]:
-                # Get the LLM talking about the next step before getting the rest
-                # of the function call completion
-                current_step += 1
-                self._context.add_message(
-                    {"role": "system", "content": steps[current_step]["prompt"]}
-                )
-                yield OpenAILLMContextFrame(self._context)
-
-                local_context = copy.deepcopy(self._context)
-                local_context.set_tool_choice("none")
-                async for frame in llm.process_frame(
-                    OpenAILLMContextFrame(local_context)
-                ):
-                    yield frame
-            else:
-                # Insert a quick response while we run the function
-                yield AudioFrame(sounds["ding2.wav"])
-                pass
-        elif isinstance(frame, LLMFunctionCallFrame):
-
-            if frame.function_name and frame.arguments:
-                print(
-                    f"--> Calling function: {frame.function_name} with arguments:")
-                pretty_json = re.sub(
-                    "\n", "\n    ", json.dumps(
-                        json.loads(
-                            frame.arguments), indent=2))
-                print(f"--> {pretty_json}\n")
-                if frame.function_name not in self._functions:
-                    raise Exception(
-                        f"Unknown function.")
-                fn = getattr(self, frame.function_name)
-                result = fn(json.loads(frame.arguments))
-
-                if not this_step["run_async"]:
-                    if result:
-                        current_step += 1
-                        self._context.add_message(
-                            {"role": "system", "content": steps[current_step]["prompt"]}
-                        )
-                        yield OpenAILLMContextFrame(self._context)
-
-                        local_context = copy.deepcopy(self._context)
-                        local_context.set_tool_choice("none")
-                        async for frame in llm.process_frame(
-                            OpenAILLMContextFrame(local_context)
-                        ):
-                            yield frame
-                    else:
-                        self._context.add_message(
-                            {"role": "system", "content": this_step["failed"]}
-                        )
-                        yield OpenAILLMContextFrame(self._context)
-
-                        local_context = copy.deepcopy(self._context)
-                        local_context.set_tool_choice("none")
-                        async for frame in llm.process_frame(
-                            OpenAILLMContextFrame(local_context)
-                        ):
-                            yield frame
-                    print(f"<-- Verify result: {result}\n")
-
-        else:
-            yield frame
-
-
-async def main(room_url: str, token):
-    async with aiohttp.ClientSession() as session:
-        global transport
-        global llm
-        global tts
-
-        transport = DailyTransport(
-            room_url,
-            token,
-            "Intake Bot",
-            5,
-            mic_enabled=True,
-            mic_sample_rate=16000,
-            camera_enabled=False,
-            start_transcription=True,
-            vad_enabled=True,
-        )
-
-        messages = []
-
-        llm = FireworksLLMService(
-            api_key=os.getenv("FIREWORKS_API_KEY"),
-            model="accounts/fireworks/models/firefunction-v1"
-        )
-        # tts = DeepgramTTSService(
-        #     aiohttp_session=session,
-        #     api_key=os.getenv("DEEPGRAM_API_KEY"),
-        #     voice="aura-asteria-en",
-        # )
-        tts = ElevenLabsTTSService(
-            aiohttp_session=session,
-            api_key=os.getenv("ELEVENLABS_API_KEY"),
-            voice_id="XrExE9yKIg1WjnnlVkGX",
-        )
-        context = OpenAILLMContext(
-            messages=messages,
-        )
-
-        checklist = ChecklistProcessor(context, llm)
-        fl = FrameLogger("FRAME LOGGER 1:")
-        fl2 = FrameLogger("FRAME LOGGER 2:")
-        pipeline = Pipeline(processors=[fl, llm, fl2, checklist, tts])
-
-        @transport.event_handler("on_first_other_participant_joined")
-        async def on_first_other_participant_joined(transport, participant):
-            await pipeline.queue_frames([OpenAILLMContextFrame(context)])
-
-        async def handle_intake():
-            await transport.run_interruptible_pipeline(
-                pipeline,
-                post_processor=OpenAIAssistantContextAggregator(context),
-                pre_processor=OpenAIUserContextAggregator(context),
-            )
-
-        try:
-            await asyncio.gather(transport.run(), handle_intake())
-        except (asyncio.CancelledError, KeyboardInterrupt):
-            print("whoops")
-            transport.stop()
-
-
-if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))

examples/starter-apps/storybot.py

@@ -1,294 +0,0 @@
-import aiohttp
-import asyncio
-import json
-import random
-import logging
-import os
-import re
-import wave
-from typing import AsyncGenerator
-from PIL import Image
-
-from pipecat.pipeline.pipeline import Pipeline
-from pipecat.pipeline.frame_processor import FrameProcessor
-from pipecat.services.live_stream import LiveStream
-from pipecat.transports.daily_transport import DailyTransport
-from pipecat.services.azure_ai_services import AzureLLMService, AzureTTSService
-from pipecat.services.fal_ai_services import FalImageGenService
-from pipecat.services.open_ai_services import OpenAILLMService
-from pipecat.services.deepgram_ai_services import DeepgramTTSService
-from pipecat.services.elevenlabs_ai_services import ElevenLabsTTSService
-from pipecat.pipeline.aggregators import (
-    LLMAssistantContextAggregator,
-    LLMAssistantResponseAggregator,
-    LLMUserResponseAggregator,
-)
-from pipecat.pipeline.frames import (
-    EndPipeFrame,
-    LLMMessagesFrame,
-    Frame,
-    TextFrame,
-    LLMResponseEndFrame,
-    AudioFrame,
-    ImageFrame,
-    UserStoppedSpeakingFrame,
-)
-from pipecat.services.ai_services import FrameLogger, AIService
-
-from runner import configure
-
-from dotenv import load_dotenv
-load_dotenv(override=True)
-
-logging.basicConfig(format=f"%(levelno)s %(asctime)s %(message)s")
-logger = logging.getLogger("pipecat")
-logger.setLevel(logging.DEBUG)
-
-sounds = {}
-images = {}
-sound_files = ["talking.wav", "listening.wav", "ding3.wav"]
-image_files = ["grandma-writing.png", "grandma-listening.png"]
-script_dir = os.path.dirname(__file__)
-
-for file in sound_files:
-    # Build the full path to the sound file
-    full_path = os.path.join(script_dir, "assets", file)
-    # Get the filename without the extension to use as the dictionary key
-    filename = os.path.splitext(os.path.basename(full_path))[0]
-    # Open the sound and convert it to bytes
-    with wave.open(full_path) as audio_file:
-        sounds[file] = audio_file.readframes(-1)
-
-for file in image_files:
-    # Build the full path to the image file
-    full_path = os.path.join(script_dir, "assets", file)
-    # Get the filename without the extension to use as the dictionary key
-    filename = os.path.splitext(os.path.basename(full_path))[0]
-    # Open the image and convert it to bytes
-    with Image.open(full_path) as img:
-        images[file] = img.tobytes()
-
-
-class StoryStartFrame(TextFrame):
-    pass
-
-
-class StoryPageFrame(TextFrame):
-    pass
-
-
-class StoryPromptFrame(TextFrame):
-    pass
-
-
-class StoryProcessor(FrameProcessor):
-    def __init__(self, messages, story):
-        self._messages = messages
-        self._text = ""
-        self._story = story
-
-    async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
-        """
-        The response from the LLM service looks like:
-        A comment about the user's choice
-        [start] (when the cat starts telling parts of the story)
-        A sentence of the story
-        [break] (between each sentence/'page' of the story)
-        [prompt] (when the cat asks the user to make a decision)
-        Question about the next part of the story
-
-        1. Catch the frames that are generated by the LLM service
-        """
-        if isinstance(frame, UserStoppedSpeakingFrame):
-            yield ImageFrame(images["grandma-writing.png"], (1024, 1024))
-            yield AudioFrame(sounds["talking.wav"])
-
-        elif isinstance(frame, TextFrame):
-            self._text += frame.text
-
-            if re.findall(r".*\[[sS]tart\].*", self._text):
-                # Then we have the intro. Send it to speech ASAP
-                self._text = self._text.replace("[Start]", "")
-                self._text = self._text.replace("[start]", "")
-
-                self._text = self._text.replace("\n", " ")
-                if len(self._text) > 2:
-                    yield ImageFrame(images["grandma-writing.png"], (1024, 1024))
-                    yield StoryStartFrame(self._text)
-                    yield AudioFrame(sounds["ding3.wav"])
-                self._text = ""
-
-            elif re.findall(r".*\[[bB]reak\].*", self._text):
-                # Then it's a page of the story. Get an image too
-                self._text = self._text.replace("[Break]", "")
-                self._text = self._text.replace("[break]", "")
-                self._text = self._text.replace("\n", " ")
-                if len(self._text) > 2:
-                    self._story.append(self._text)
-                    yield StoryPageFrame(self._text)
-                    yield AudioFrame(sounds["ding3.wav"])
-
-                self._text = ""
-            elif re.findall(r".*\[[pP]rompt\].*", self._text):
-                # Then it's question time. Flush any
-                # text here as a story page, then set
-                # the var to get to prompt mode
-                # cb: trying scene now
-                # self.handle_chunk(self._text)
-                self._text = self._text.replace("[Prompt]", "")
-                self._text = self._text.replace("[prompt]", "")
-
-                self._text = self._text.replace("\n", " ")
-                if len(self._text) > 2:
-                    self._story.append(self._text)
-                    yield StoryPageFrame(self._text)
-            else:
-                # After the prompt thing, we'll catch an LLM end to get the
-                # last bit
-                pass
-        elif isinstance(frame, LLMResponseEndFrame):
-            yield ImageFrame(images["grandma-writing.png"], (1024, 1024))
-            yield StoryPromptFrame(self._text)
-            self._text = ""
-            yield frame
-            yield ImageFrame(images["grandma-listening.png"], (1024, 1024))
-            yield AudioFrame(sounds["listening.wav"])
-
-        else:
-            # pass through everything that's not a TextFrame
-            yield frame
-
-
-class StoryImageGenerator(FrameProcessor):
-    def __init__(self, story, llm, img):
-        self._story = story
-        self._llm = llm
-        self._img = img
-
-    async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
-        if isinstance(frame, StoryPageFrame):
-            if len(self._story) == 1:
-                prompt = f'You are an illustrator for a children\'s story book. Generate a prompt for DALL-E to create an illustration for the first page of the book, which reads: "{self._story[0]}"\n\n Your response should start with the phrase "Children\'s book illustration of".'
-            else:
-                prompt = f"You are an illustrator for a children's story book. Here is the story so far:\n\n\"{' '.join(self._story[:-1])}\"\n\nGenerate a prompt for DALL-E to create an illustration for the next page. Here's the sentence for the next page:\n\n\"{self._story[-1:][0]}\"\n\n Your response should start with the phrase \"Children's book illustration of\"."
-            msgs = [{"role": "system", "content": prompt}]
-            image_prompt = ""
-            async for f in self._llm.process_frame(LLMMessagesFrame(msgs)):
-                if isinstance(f, TextFrame):
-                    image_prompt += f.text
-            async for f in self._img.process_frame(TextFrame(image_prompt)):
-                yield f
-            # Yield the original StoryPageFrame for basic image/audio sync
-            yield frame
-        else:
-            yield frame
-
-
-async def main(room_url: str, token):
-    async with aiohttp.ClientSession() as session:
-        messages = [
-            {
-                "role": "system",
-                "content": "You are a storytelling grandma who loves to make up fantastic, fun, and educational stories for children between the ages of 5 and 10 years old. Your stories are full of friendly, magical creatures. Your stories are never scary. Each sentence of your story will become a page in a storybook. Stop after 3-4 sentences and give the child a choice to make that will influence the next part of the story. Once the child responds, start by saying something nice about the choice they made, then include [start] in your response. Include [break] after each sentence of the story. Include [prompt] between the story and the prompt.",
-            }
-        ]
-
-        story = []
-
-        llm = OpenAILLMService(
-            api_key=os.getenv("OPENAI_API_KEY"),
-            model="gpt-4-1106-preview",
-        )  # gpt-4-1106-preview
-        tts = ElevenLabsTTSService(
-            aiohttp_session=session,
-            api_key=os.getenv("ELEVENLABS_API_KEY"),
-            voice_id="Xb7hH8MSUJpSbSDYk0k2",
-        )  # matilda
-        img = FalImageGenService(
-            params={
-                image_size = "1024x1024",
-            },
-            aiohttp_session=session,
-            key=os.getenv("FAL_KEY"),
-        )
-        lra = LLMAssistantResponseAggregator(messages)
-        ura = LLMUserResponseAggregator(messages)
-        sp = StoryProcessor(messages, story)
-        sig = StoryImageGenerator(story, llm, img)
-
-        transport = DailyTransport(
-            room_url,
-            token,
-            "Storybot",
-            5,
-            mic_enabled=True,
-            mic_sample_rate=16000,
-            camera_enabled=True,
-            camera_width=1024,
-            camera_height=1024,
-            start_transcription=True,
-            vad_enabled=True,
-            vad_stop_s=1.5,
-        )
-
-        start_story_event = asyncio.Event()
-
-        @transport.event_handler("on_first_other_participant_joined")
-        async def on_first_other_participant_joined(transport, participant):
-            start_story_event.set()
-
-        async def storytime():
-            await start_story_event.wait()
-
-            # We're being a bit tricky here by using a special system prompt to
-            # ask the user for a story topic. After their intial response, we'll
-            # use a different system prompt to create story pages.
-            intro_messages = [
-                {
-                    "role": "system",
-                    "content": "You are a storytelling grandma who loves to make up fantastic, fun, and educational stories for children between the ages of 5 and 10 years old. Your stories are full of friendly, magical creatures. Your stories are never scary. Begin by asking what a child wants you to tell a story about. Keep your reponse to only a few sentences.",
-                }
-            ]
-            lca = LLMAssistantContextAggregator(messages)
-            local_pipeline = Pipeline(
-                [llm, lca, tts], sink=transport.send_queue)
-            await local_pipeline.queue_frames(
-                [
-                    ImageFrame(images["grandma-listening.png"], (1024, 1024)),
-                    LLMMessagesFrame(intro_messages),
-                    AudioFrame(sounds["listening.wav"]),
-                    EndPipeFrame(),
-                ]
-            )
-            await local_pipeline.run_pipeline()
-
-            pipeline = Pipeline([llm, lca, tts, ls_sink])
-            pipeline.queue_frames([...])
-            pipeline.run()
-
-            fl = FrameLogger("### After Image Generation")
-            pipeline = Pipeline(
-                processors=[
-                    ura,
-                    llm,
-                    sp,
-                    sig,
-                    fl,
-                    tts,
-                    lra,
-                ]
-            )
-            await transport.run_pipeline(
-                pipeline,
-            )
-
-        try:
-            await asyncio.gather(transport.run(), storytime())
-        except (asyncio.CancelledError, KeyboardInterrupt):
-            print("whoops")
-            transport.stop()
-
-
-if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))

examples/starter-apps/translator.py

@@ -1,107 +0,0 @@
-import asyncio
-import aiohttp
-import logging
-import os
-from typing import AsyncGenerator
-
-from pipecat.pipeline.aggregators import (
-    SentenceAggregator,
-)
-from pipecat.pipeline.frames import (
-    Frame,
-    LLMMessagesFrame,
-    TextFrame,
-    SendAppMessageFrame,
-)
-from pipecat.pipeline.frame_processor import FrameProcessor
-from pipecat.pipeline.pipeline import Pipeline
-from pipecat.transports.daily_transport import DailyTransport
-from pipecat.services.azure_ai_services import AzureTTSService
-from pipecat.services.open_ai_services import OpenAILLMService
-from pipecat.pipeline.aggregators import LLMFullResponseAggregator
-
-from runner import configure
-
-from dotenv import load_dotenv
-
-
-load_dotenv(override=True)
-
-logging.basicConfig(format=f"%(levelno)s %(asctime)s %(message)s")
-logger = logging.getLogger("pipecat")
-logger.setLevel(logging.DEBUG)
-
-"""
-This example looks a bit different than the chatbot example, because it isn't waiting on the user to stop talking to start translating.
-It also isn't saving what the user or bot says into the context object for use in subsequent interactions.
-"""
-
-
-# We need to use a custom service here to yield LLM frames without saving
-# any context
-class TranslationProcessor(FrameProcessor):
-    def __init__(self, language):
-        self._language = language
-
-    async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
-        if isinstance(frame, TextFrame):
-            context = [
-                {
-                    "role": "system",
-                    "content": f"You will be provided with a sentence in English, and your task is to translate it into {self._language}.",
-                },
-                {"role": "user", "content": frame.text},
-            ]
-            yield LLMMessagesFrame(context)
-        else:
-            yield frame
-
-
-class TranslationSubtitles(FrameProcessor):
-    def __init__(self, language):
-        self._language = language
-
-    async def process_frame(self, frame: Frame) -> AsyncGenerator[Frame, None]:
-        if isinstance(frame, TextFrame):
-            app_message = {
-                "language": self._language,
-                "text": frame.text
-            }
-            yield SendAppMessageFrame(app_message, None)
-            yield frame
-        else:
-            yield frame
-
-
-async def main(room_url: str, token):
-    async with aiohttp.ClientSession() as session:
-        transport = DailyTransport(
-            room_url,
-            token,
-            "Translator",
-            duration_minutes=5,
-            start_transcription=True,
-            mic_enabled=True,
-            mic_sample_rate=16000,
-            camera_enabled=False,
-        )
-        tts = AzureTTSService(
-            api_key=os.getenv("AZURE_SPEECH_API_KEY"),
-            region=os.getenv("AZURE_SPEECH_REGION"),
-            voice="es-ES-AlvaroNeural",
-        )
-        llm = OpenAILLMService(
-            api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4-turbo-preview"
-        )
-        sa = SentenceAggregator()
-        tp = TranslationProcessor("Spanish")
-        lfra = LLMFullResponseAggregator()
-        ts = TranslationSubtitles("spanish")
-        pipeline = Pipeline([sa, tp, llm, lfra, ts, tts])
-
-        await transport.run(pipeline)
-
-
-if __name__ == "__main__":
-    (url, token) = configure()
-    asyncio.run(main(url, token))

examples/storytelling-chatbot/.dockerignore

@@ -0,0 +1,2 @@
+frontend/node_modules
+frontend/out

examples/storytelling-chatbot/.gitignore

@@ -0,0 +1,158 @@
+node_modules/
+.idea/
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+.DS_Store
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintainted in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+read.html

examples/storytelling-chatbot/Dockerfile

@@ -0,0 +1,54 @@
+FROM python:3.11-bullseye
+
+ARG DEBIAN_FRONTEND=noninteractive
+ARG USE_PERSISTENT_DATA
+ENV PYTHONUNBUFFERED=1
+ENV NODE_MAJOR=20
+
+# Expose FastAPI port
+ENV FAST_API_PORT=7860
+EXPOSE 7860
+
+# Install system dependencies
+RUN apt-get update && apt-get install --no-install-recommends -y \
+    build-essential \
+    git \
+    ffmpeg \
+    google-perftools \
+    ca-certificates curl gnupg \
+    && apt-get clean && rm -rf /var/lib/apt/lists/*
+
+# Install Node.js
+RUN mkdir -p /etc/apt/keyrings 
+RUN curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg
+RUN echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_${NODE_MAJOR}.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list > /dev/null
+RUN apt-get update && apt-get install nodejs -y
+
+# Set up a new user named "user" with user ID 1000
+RUN useradd -m -u 1000 user
+
+# Set home to the user's home directory
+ENV HOME=/home/user \
+    PATH=/home/user/.local/bin:$PATH \
+    PYTHONPATH=$HOME/app \
+    PYTHONUNBUFFERED=1
+
+# Switch to the "user" user
+USER user
+
+# Set the working directory to the user's home directory
+WORKDIR $HOME/app
+
+# Install Python dependencies
+COPY ./requirements.txt requirements.txt
+RUN pip3 install --no-cache-dir --upgrade -r requirements.txt
+
+# Copy everything else
+COPY --chown=user ./src/ src/
+
+# Copy frontend app and build
+COPY --chown=user ./frontend/ frontend/
+RUN cd frontend && npm install && npm run build
+
+# Start the FastAPI server
+CMD python3 src/server.py --port ${FAST_API_PORT}

examples/storytelling-chatbot/README.md

@@ -0,0 +1,83 @@
+[![Try](https://img.shields.io/badge/try_it-here-blue)](https://storytelling-chatbot.fly.dev)
+
+# Storytelling Chatbot
+
+<img src="image.png" width="420px">
+
+This example shows how to build a voice-driven interactive storytelling experience.
+It periodically prompts the user for input for a 'choose your own adventure' style experience.
+
+We add visual elements to the story by generating images at lightning speed using Fal.
+
+
+---
+
+### It uses the following AI services:
+
+**Deepgram - Speech-to-Text**
+
+Transcribes inbound participant voice media to text.
+
+**OpenAI (GPT4) - LLM**
+
+Our creative writer LLM. You can see the context used to prompt it [here](src/prompts.py)
+
+**ElevenLabs - Text-to-Speech**
+
+Converts and streams the LLM response from text to audio
+
+**Fal.ai - Image Generation**
+
+Adds pictures to our story (really fast!) Prompting is quite key for style consistency, so we task the LLM to turn each story page into a short image prompt.
+
+---
+
+## Setup
+
+**Install requirements**
+
+```shell
+pip install -r requirements.txt
+```
+
+**Create environment file and set variables:**
+
+```shell
+mv env.example .env
+```
+
+**Build the frontend:**
+
+This project uses a custom frontend, which needs to built. Note: this is done automatically as part of the Docker deployment.
+
+```shell
+cd frontend/
+npm install / yarn
+npm run build
+```
+
+The build UI files can be found in `frontend/out`
+
+## Running it locally
+
+Start the API / bot manager:
+
+`python src/server.py`
+
+If you'd like to run a custom domain or port:
+
+`python src/server.py --host somehost --p 7777`
+
+➡️ Open the host URL in your browser
+
+> [!IMPORTANT]
+> Whilst working on the frontend code, please `yarn run dev`
+> and open the NextJS hosted service vs. the Python server.
+> (Usually localhost:3000.)
+
+---
+
+## Improvements to make
+
+- Wait for track_started event to avoid rushed intro
+- Show 5 minute timer on the UI

examples/storytelling-chatbot/env.example

@@ -0,0 +1,5 @@
+DAILY_API_KEY=7df...
+ELEVENLABS_API_KEY=aeb...
+ELEVENLABS_VOICE_ID=7S...
+FAL_KEY=8c...
+OPENAI_API_KEY=sk-PL...