From 5c402eee819aeb77f3b8b717d8587255f923b982 Mon Sep 17 00:00:00 2001 From: Chad Bailey Date: Thu, 8 Feb 2024 16:31:17 +0000 Subject: [PATCH] started adding docs --- README.md | 62 ++++++++++++++-- docs/README.md | 13 ++++ docs/architecture.md | 2 + docs/examples/01-say-one-thing.md | 119 ++++++++++++++++++++++++++++++ docs/examples/README.md | 5 ++ 5 files changed, 196 insertions(+), 5 deletions(-) create mode 100644 docs/README.md create mode 100644 docs/architecture.md create mode 100644 docs/examples/01-say-one-thing.md create mode 100644 docs/examples/README.md diff --git a/README.md b/README.md index 9e782214e..12219ad89 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,33 @@ -# dailyai SDK +# Daily AI SDK + +Build conversational, multi-modal AI apps with real-time voice and video, like this: + +_Demo Video_ + +With built-in support for many of the best AI platforms (or [add your own](/docs)): + +- Azure - DALL-E, ChatGPT, and Azure AI Text-to-Speech +- Deepgram - Speech-to-text, and Aura text-to-speech +- Eleven Labs text-to-speech +- Fal.ai image generation +- OpenAI DALL-E and ChatGPT +- Whisper local speech-to-text + +## Step 1: Get Started + +Installation here. Also sign up for a Daily account, I guess? also we need an ENV + +Requires python 3.11 or later. Don't forget virtualenv + +pip install vs download and build? + +## Step 2: Build Things + +Once you've got the SDK working, head over to the [docs folder](/docs) to start building! + +--- + +# Old Readme This SDK can help you build applications that participate in WebRTC meetings and use various AI services to interact with other participants. @@ -55,20 +84,31 @@ export $(grep -v '^#' .env | xargs) ``` ## Overview + The Daily AI SDK allows you to build applications that can participate in WebRTC sessions and interact with AI Services. Some examples of what you can build with this: -* conversational bots that interact 1:1 with a user, using voice recognition and text-to-speech -* assistant bots that aggregate transcriptions from multiple participants in a meeting and provide realtime summaries or other AI-generated output. -* image-recognition bots -* etc + +- conversational bots that interact 1:1 with a user, using voice recognition and text-to-speech +- assistant bots that aggregate transcriptions from multiple participants in a meeting and provide realtime summaries or other AI-generated output. +- image-recognition bots +- etc + ## Concepts + ### Transport Service + The SDK provides one “transport service”, which is a wrapper around Daily’s `daily-python` client (tk add link). You can use this service to listen for events related to a WebRTC session, such as “a participant joined the meeting”. The transport service also exposes a send queue, and a receive queue. You can use the send queue to send audio and video to the WebRTC session, and you can listen to the receive queue to see audio, video and transcription data from the WebRTC session. + ### AI Services + The AI Service classes provide wrappers around various AI providers, and allow you to query LLMs, convert text to speech and make images from text. The audio and images can then be placed on the transport service’s send queue, where they’ll be sent to the WebRTC session. + ### Queue Frames + Communication between the transport service and AI services, and between various AI services, takes place in Queue Frames. These frames contain an indication of the type of data as well as the data itself. + ## Using Transports, AI Services and Frames + AI Services all define a `.run` method. This method consumes and generates `QueueFrame` frames. The kind of frames that can be consumed and generated depend on the kind of service. For instance, an LLM AI Service consumes `LLM_MESSAGE` frames (which define a history of interaction with an LLM) and emit `TEXT` frames (the response from the LLM). The `.run` method is an `AsyncIterable`, and it takes an `iterable`, `AsyncIterable` or `asyncio.Queue` that produces QueueFrames as a parameter. This makes it easy to chain AI Services, and consume input from the Transport’s `receive_queue` . @@ -76,18 +116,25 @@ The `.run` method is an `AsyncIterable`, and it takes an `iterable`, `AsyncItera AI Services also have a `.run_to_queue` method. This method is not an AsyncIterable, but instead sends processed QueueFrames to a queue. This makes it easy to send the output of an AI Service to the Transport’s `send_queue`. AI Services also define convenience functions that let you bypass creating QueueFrames for some simple cases (eg. using the TTS service to convert a string to audio output and send that audio to the transport’s `send_queue`). See below for examples. + ## Examples + ### Say Something + The base TTS AI service exposes a `.say` method. After creating a transport and TTS service, you can use this method like so: + ``` transport = DailyTransportService(...) tts = AzureTTSService() await tts.say("hello world", transport.send_queue) ``` + This will call the TTS service to render the text to audio frames, then put the audio frames on the transport’s send queue. The transport will then send those frames along to the WebRTC session. ### Speak an LLM response + Given a system prompt contained in a `messages` array, you can emit the LLM’s response as audio with a chain like this: + ``` transport = DailyTransportService(...) # setup parameters omitted tts = AzureTTSService() @@ -99,14 +146,17 @@ await tts.run_to_queue( llm.run([QueueFrame.LLM_MESSAGES, messages]) ) ``` + In this code, the LLM service object sends the messages to Azure’s OpenAI implementation, which streams chunks back asynchronously. Those chunks are aggregated by the TTS Service to ensure the best audio response (TTS works best when it gets complete sentence, so it can inflect correctly), then sent to Azure’s TTS service, converted to audio frames, and sent to the WebRTC session via the Daily transport. ### Pre-cache an LLM response + Sometimes LLMs can be slower than we’d like for natural-feeling communication. Here’s an example where we take advantage of the time it takes to speak some pre-defined text to get a head start on the LLM response: (TK link to 04- sample) In this sample, we set up a buffer queue to receive the audio frames from the LLM response before while we are joining the call and start an asynchronous task to start filling this buffer: + ``` buffer_queue = asyncio.Queue() llm_response_task = asyncio.create_task( @@ -119,11 +169,13 @@ In this sample, we set up a buffer queue to receive the audio frames from the LL ``` Then, when we’ve joined the call, we speak the static text: + ``` await azure_tts.say("My friend...", transport.send_queue) ``` As that text is being spoken, the asynchronous LLM task continues in the background. When the text is done, we pull the frames off the buffer queue and put them in the transport’s `send_queue`: + ``` async def buffer_to_send_queue(): while True: diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 000000000..c3be534f0 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,13 @@ +# Daily AI SDK Docs + +## [Architecture Overview](architecture.md) + +Learn about the thinking behind the SDK's design. + +## [Example Code](examples/) + +The repo includes several example apps in the `src/examples` directory. The docs explain how they work. + +## [API Reference](api/) + +Complete documentation of the available classes and methods in the SDK. diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 000000000..5566a29ba --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,2 @@ +# Daily AI SDK Architecture Guide + diff --git a/docs/examples/01-say-one-thing.md b/docs/examples/01-say-one-thing.md new file mode 100644 index 000000000..319c89960 --- /dev/null +++ b/docs/examples/01-say-one-thing.md @@ -0,0 +1,119 @@ +# 01: Say One Thing + +_video here - youtube?_ + +This example uses a text-to-speech (TTS) service to say one predefined sentence. But first, a quick overview of the general structure of these examples. + +## Running the demos + +All of the demos have something like this at the bottom of the file: + +```python +if __name__ == "__main__": + (url, token) = configure() + asyncio.run(main(url, token)) +``` + +### `configure()` + +The `configure()` function comes from `src/examples/foundational/support/runner.py`, and it allows you to configure the examples from the command line directly, or using environment variables: + +```bash +python 01-say-one-thing.py -u https://YOUR_DOMAIN.daily.co/YOUR_ROOM -k YOUR_API_KEY +# or +DAILY_ROOM_URL=https://YOUR_DOMAIN.daily.co/YOUR_ROOM DAILY_API_KEY=YOUR_API_KEY python 01-say-one-thing.py +# or set DAILY_ROOM_URL and DAILY_API_KEY in a .env file +python 01-say-one-thing.py +``` + +You'll need a Daily account to run these demos. You can sign up for free at [daily.co](https://daily.co). Once you've signed up you can create a room from the [Dashboard](https://dashboard.daily.co/rooms), and grab [your API key](https://dashboard.daily.co/developers) while you're there. + +Some functionality (such as transcription) requires the bot to have owner privileges in the room. `runner.py` uses the Daily REST API to create a meeting token with owner privileges. You can learn more about meeting tokens in the [Daily docs](https://docs.daily.co/reference/rest-api/meeting-tokens). + +### `asyncio.run()` + +The AI SDK makes heavy use of Python's `asyncio` module. [This is a reasonable intro to the topic](https://builtin.com/data-science/asyncio) if you haven't worked with `asyncio` and coroutines before. + +You can learn a bit more about the specifics of how the Daily AI SDK uses coroutines in the [Architecture Guide](../architecture.md). + +## The `main()` function + +All of the examples have a `main()` function with a similar structure: + +- Configure the transport +- Configure the AI service(s) used in the demo +- Configure any event listeners +- Define a processing pipeline +- Run the example's coroutine(s) + +### Configuring the transport + +The first section of the `main()` function configures the transport object: + +```python +meeting_duration_minutes = 5 +transport = DailyTransportService( + room_url, + None, + "Say One Thing", + meeting_duration_minutes, +) +transport.mic_enabled = True +``` + +The [Architecture Guide](../architecture.md) explains the transport object in more detail. In this case, we're configuring a Daily transport object and enabling the virtual microphone, so our bot can play audio. + +### Configuring the services + +As described in the [Architecture Guide](../architecture.md), 'a 'Service' is a class that processes 'Frames' as part of a 'Pipeline'. In this demo app, we'll only need one service: a text-to-speech generator. We can create an instance of the `ElevenLabsTTSService` class with this line of code: + +```python +tts = ElevenLabsTTSService(aiohttp_session=session, api_key=os.getenv("ELEVENLABS_API_KEY"), voice_id=os.getenv("ELEVENLABS_VOICE_ID")) +``` + +You'll need to make sure and set those environment variables somewhere. The easiest way to do that is to copy the `example.env` file in the repo and rename it to `.env`, and then add your credentials to that file. `runner.py` loads the `python-dotenv` module and initializes it, making the values in that file available in the environment. + +### Configuring event listeners + +This part isn't strictly necessary for an app like this. You could include the contents of the `on_participant_joined` function directly in the body of the `main()` function, and it would run as soon as you started the script from the command line. + +Instead, we can use an event handler to wait to run that code until someone else joins the meeting. We'll define a function called `greet_user()`, and use the `@transport.event_handler("on_participant_joined")` decorator to tell the SDK that we want to run that function whenever a user joins the room. + +```python +@transport.event_handler("on_participant_joined") +async def greet_user(transport, participant): + if participant["info"]["isLocal"]: + return + + await tts.say( + "Hello there, " + participant["info"]["userName"] + "!", + transport.send_queue, + ) + + # wait for the output queue to be empty, then leave the meeting + await transport.stop_when_done() +``` + +### Defining a processing pipeline + +In this example, we don't actually have much of a processing pipeline! In fact, we're doing the whole thing inside the `greet_user()` function already. + +Pipelines usually look like a bunch of nested calls to the `run()` or `run_to_queue()` function from different Services. In this example, we're using the `say()` function from the TTS service. This is effectively a convenience wrapper around the `run_to_queue()` function, which we'll discuss more later. It's important to `await` this function to ensure that the speech frames are queued for playback before the next line of code, because of the `stop_when_done()` function being called immediately afterward. + +The output of the `say()` function goes to the transport's `send_queue`. This queue is the all-important connection between the world of the Services pipeline that's generating frames asynchronously and the ordered playback of audio and visual media in the WebRTC call. + +### Running the coroutines + +In this example, we don't actually have any separate processing pipelines—everything happens as a result of an event from the transport. So we only need to run the transport's coroutine, and await its completion: + +```python +await transport.run() +``` + +In future examples, we'll run more processes in parallel. For now, this script can run until the transport exits—which will happen based on calling `stop_when_done()` in the `greet_user()` function. + +## Next Steps + +Next, we'll start connecting multiple AI services together by building a service pipeline. + +## [02 - LLM Say One Thing »](02-llm-say-one-thing.md) diff --git a/docs/examples/README.md b/docs/examples/README.md new file mode 100644 index 000000000..ba3f6f45d --- /dev/null +++ b/docs/examples/README.md @@ -0,0 +1,5 @@ +# Daily AI SDK Examples + +The docs in this folder pair with the example apps located in `src/examples/foundational`. They are designed to serve as a quick references for building different kinds of AI apps. But the examples also build on one another, so it can be really helpful to walk through them in order. + +To start, you can learn about the overall structure of the examples in [01 - Say One Thing](01-say-one-thing.md).