Merge branch 'main' into cb/golden-kitty-aws

Dockerfile

@@ -0,0 +1,39 @@
+# setup
+FROM python:3.11.5
+
+WORKDIR /app
+COPY requirements.txt /app
+COPY *.py /app
+COPY pyproject.toml /app
+
+COPY src/ /app/src/
+
+WORKDIR /app
+RUN ls --recursive /app/
+RUN pip3 install --upgrade -r requirements.txt
+RUN python -m build .
+RUN pip3 install .
+
+# If running on Ubuntu, Azure TTS requires some extra config
+# https://learn.microsoft.com/en-us/azure/ai-services/speech-service/quickstarts/setup-platform?pivots=programming-language-python&tabs=linux%2Cubuntu%2Cdotnetcli%2Cdotnet%2Cjre%2Cmaven%2Cnodejs%2Cmac%2Cpypi
+
+RUN wget -O - https://www.openssl.org/source/openssl-1.1.1w.tar.gz | tar zxf -
+WORKDIR openssl-1.1.1w
+RUN ./config --prefix=/usr/local
+RUN make -j $(nproc)
+RUN make install_sw install_ssldirs
+RUN ldconfig -v
+ENV SSL_CERT_DIR=/etc/ssl/certs
+
+#ENV LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
+RUN apt clean
+RUN apt-get update
+RUN apt-get -y install build-essential libssl-dev ca-certificates libasound2 wget
+
+ENV PYTHONUNBUFFERED=1
+
+WORKDIR /app
+
+EXPOSE 8000
+# run
+CMD ["gunicorn", "--workers=2", "--log-level", "debug", "--capture-output", "daily-bot-manager:app", "--bind=0.0.0.0:8000"]

README.md

@@ -55,20 +55,31 @@ export $(grep -v '^#' .env | xargs)
 ```
 
 ## Overview
+
 The Daily AI SDK allows you to build applications that can participate in WebRTC sessions and interact with AI Services. Some examples of what you can build with this:
-* conversational bots that interact 1:1 with a user, using voice recognition and text-to-speech
-* assistant bots that aggregate transcriptions from multiple participants in a meeting and provide realtime summaries or other AI-generated output.
-* image-recognition bots
-* etc
+
+- conversational bots that interact 1:1 with a user, using voice recognition and text-to-speech
+- assistant bots that aggregate transcriptions from multiple participants in a meeting and provide realtime summaries or other AI-generated output.
+- image-recognition bots
+- etc
+
 ## Concepts
+
 ### Transport Service
+
 The SDK provides one “transport service”, which is a wrapper around Daily’s `daily-python` client (tk add link). You can use this service to listen for events related to a WebRTC session, such as “a participant joined the meeting”.
 The transport service also exposes a send queue, and a receive queue. You can use the send queue to send audio and video to the WebRTC session, and you can listen to the receive queue to see audio, video and transcription data from the WebRTC session.
+
 ### AI Services
+
 The AI Service classes provide wrappers around various AI providers, and allow you to query LLMs, convert text to speech and make images from text. The audio and images can then be placed on the transport service’s send queue, where they’ll be sent to the WebRTC session.
+
 ### Queue Frames
+
 Communication between the transport service and AI services, and between various AI services, takes place in Queue Frames. These frames contain an indication of the type of data as well as the data itself.
+
 ## Using Transports, AI Services and Frames
+
 AI Services all define a `.run` method. This method consumes and generates `QueueFrame` frames. The kind of frames that can be consumed and generated depend on the kind of service. For instance, an LLM AI Service consumes `LLM_MESSAGE` frames (which define a history of interaction with an LLM) and emit `TEXT` frames (the response from the LLM).
 
 The `.run` method is an `AsyncIterable`, and it takes an `iterable`, `AsyncIterable` or `asyncio.Queue` that produces QueueFrames as a parameter. This makes it easy to chain AI Services, and consume input from the Transport’s `receive_queue` .
@@ -76,18 +87,25 @@ The `.run` method is an `AsyncIterable`, and it takes an `iterable`, `AsyncItera
 AI Services also have a `.run_to_queue` method. This method is not an AsyncIterable, but instead sends processed QueueFrames to a queue. This makes it easy to send the output of an AI Service to the Transport’s `send_queue`.
 
 AI Services also define convenience functions that let you bypass creating QueueFrames for some simple cases (eg. using the TTS service to convert a string to audio output and send that audio to the transport’s `send_queue`). See below for examples.
+
 ## Examples
+
 ### Say Something
+
 The base TTS AI service exposes a `.say` method. After creating a transport and TTS service, you can use this method like so:
+
 ```
 transport = DailyTransportService(...)
 tts = AzureTTSService()
 await tts.say("hello world", transport.send_queue)
 ```
+
 This will call the TTS service to render the text to audio frames, then put the audio frames on the transport’s send queue. The transport will then send those frames along to the WebRTC session.
 
 ### Speak an LLM response
+
 Given a system prompt contained in a `messages` array, you can emit the LLM’s response as audio with a chain like this:
+
 ```
 transport = DailyTransportService(...) # setup parameters omitted
 tts = AzureTTSService()
@@ -99,14 +117,17 @@ await tts.run_to_queue(
   llm.run([QueueFrame.LLM_MESSAGES, messages])
 )
 ```
+
 In this code, the LLM service object sends the messages to Azure’s OpenAI implementation, which streams chunks back asynchronously. Those chunks are aggregated by the TTS Service to ensure the best audio response (TTS works best when it gets complete sentence, so it can inflect correctly), then sent to Azure’s TTS service, converted to audio frames, and sent to the WebRTC session via the Daily transport.
 
 ### Pre-cache an LLM response
+
 Sometimes LLMs can be slower than we’d like for natural-feeling communication. Here’s an example where we take advantage of the time it takes to speak some pre-defined text to get a head start on the LLM response:
 
 (TK link to 04- sample)
 
 In this sample, we set up a buffer queue to receive the audio frames from the LLM response before while we are joining the call and start an asynchronous task to start filling this buffer:
+
 ```
     buffer_queue = asyncio.Queue()
     llm_response_task = asyncio.create_task(
@@ -119,11 +140,13 @@ In this sample, we set up a buffer queue to receive the audio frames from the LL
 ```
 
 Then, when we’ve joined the call, we speak the static text:
+
 ```
         await azure_tts.say("My friend...", transport.send_queue)
 ```
 
 As that text is being spoken, the asynchronous LLM task continues in the background. When the text is done, we pull the frames off the buffer queue and put them in the transport’s `send_queue`:
+
 ```
         async def buffer_to_send_queue():
             while True:
@@ -138,3 +161,11 @@ As that text is being spoken, the asynchronous LLM task continues in the backgro
 ```
 
 One thing to note here is the last parameter to `run_to_queue` in the first code clause above: this causes the `run_to_queue` method to send an `END_STREAM` frame when it’s done rendering. This lets us know when to stop our `buffer_to_send_queue` task above.
+
+## Test Server
+
+To start the test server:
+
+```python
+flask --app daily-bot-manager.py --debug run
+```

auth.py

@@ -0,0 +1,26 @@
+import time
+import urllib
+
+from dotenv import load_dotenv
+import requests
+from flask import jsonify
+import os
+
+load_dotenv()
+
+def get_meeting_token(room_name, daily_api_key, token_expiry):
+    api_path = os.getenv('DAILY_API_PATH') or 'https://api.daily.co/v1'
+
+    if not token_expiry:
+        token_expiry = time.time() + 600
+    res = requests.post(f'{api_path}/meeting-tokens',
+                        headers={'Authorization': f'Bearer {daily_api_key}'},
+                        json={'properties': {'room_name': room_name, 'is_owner': True, 'exp': token_expiry}})
+    if res.status_code != 200:
+        return jsonify({'error': 'Unable to create meeting token', 'detail': res.text}), 500
+    meeting_token = res.json()['token']
+    return meeting_token
+
+
+def get_room_name(room_url):
+    return urllib.parse.urlparse(room_url).path[1:]

daily-bot-manager.py

@@ -0,0 +1,103 @@
+import os
+import requests
+import subprocess
+import time
+
+from flask import Flask, jsonify, request, redirect
+from flask_cors import CORS
+from auth import get_meeting_token
+
+from dotenv import load_dotenv
+
+load_dotenv()
+
+app = Flask(__name__)
+CORS(app)
+
+print(f"I loaded an environment, and my FAL_KEY_ID is {os.getenv('FAL_KEY_ID')}")
+
+def start_bot(bot_path, args=None):
+    daily_api_key = os.getenv("DAILY_API_KEY")
+    api_path = os.getenv("DAILY_API_PATH") or "https://api.daily.co/v1"
+
+    timeout = int(os.getenv("ROOM_TIMEOUT") or os.getenv("BOT_MAX_DURATION") or 300)
+    exp = time.time() + timeout
+    res = requests.post(
+        f"{api_path}/rooms",
+        headers={"Authorization": f"Bearer {daily_api_key}"},
+        json={
+            "properties": {
+                "exp": exp,
+                "enable_chat": True,
+                "enable_emoji_reactions": True,
+                "eject_at_room_exp": True,
+                "enable_prejoin_ui": False,
+                "enable_recording": "cloud"
+            }
+        },
+    )
+    if res.status_code != 200:
+        return (
+            jsonify(
+                {
+                    "error": "Unable to create room",
+                    "status_code": res.status_code,
+                    "text": res.text,
+                }
+            ),
+            500,
+        )
+    room_url = res.json()["url"]
+    room_name = res.json()["name"]
+
+    meeting_token = get_meeting_token(room_name, daily_api_key, exp)
+
+    if args:
+        extra_args = " ".join([f'-{x[0]} "{x[1]}"' for x in args])
+    else:
+        extra_args = ""
+
+    proc = subprocess.Popen(
+        [
+            f"python {bot_path} -u {room_url} -t {meeting_token} -k {daily_api_key} {extra_args}"
+        ],
+        shell=True,
+        bufsize=1,
+    )
+
+    # Don't return until the bot has joined the room, but wait for at most 2 seconds.
+    attempts = 0
+    while attempts < 20:
+        time.sleep(0.1)
+        attempts += 1
+        res = requests.get(
+            f"{api_path}/rooms/{room_name}/get-session-data",
+            headers={"Authorization": f"Bearer {daily_api_key}"},
+        )
+        if res.status_code == 200:
+            break
+    print(f"Took {attempts} attempts to join room {room_name}")
+    
+    # Additional client config
+    config = {}
+    if os.getenv("CLIENT_VAD_TIMEOUT_SEC"):
+        config['vad_timeout_sec'] = float(os.getenv("CLIENT_VAD_TIMEOUT_SEC"))
+    else:
+        config['vad_timeout_sec'] = 1.5
+
+    #return jsonify({"room_url": room_url, "token": meeting_token, "config": config}), 200
+    return redirect(room_url, code=302)
+
+
+@app.route("/spin-up-kitty", methods=["POST"])
+def spin_up_kitty():
+    return start_bot("./src/samples/foundational/06a-golden-kitty.py")
+
+@app.route("/spin-up-kitty", methods=["GET"])
+def quick_start_kitty():
+    return start_bot("./src/samples/foundational/06a-golden-kitty.py")
+
+
+@app.route("/healthz")
+def health_check():
+    return "ok", 200

requirements.txt

@@ -1,4 +1,4 @@
 autopep8==2.0.4
 build==1.0.3
 packaging==23.2
-pyproject_hooks==1.0.0
+pyproject_hooks==1.0.0

src/dailyai/queue_frame.py

@@ -30,6 +30,10 @@ class ImageQueueFrame(QueueFrame):
     image: bytes
 
 
+@dataclass()
+class ImageListQueueFrame(QueueFrame):
+    images: list[bytes] | None
+
 @dataclass()
 class TextQueueFrame(QueueFrame):
     text: str

src/dailyai/services/daily_transport_service.py

@@ -1,6 +1,7 @@
 import asyncio
 import inspect
 import logging
+import sys
 import threading
 import time
 import types
@@ -13,6 +14,7 @@ from dailyai.queue_frame import (
     AudioQueueFrame,
     EndStreamQueueFrame,
     ImageQueueFrame,
+    ImageListQueueFrame,
     QueueFrame,
     StartStreamQueueFrame,
     TextQueueFrame,
@@ -69,8 +71,8 @@ class DailyTransportService(EventHandler):
         self.story_started = False
         self.mic_enabled = False
         self.mic_sample_rate = 16000
-        self.camera_width = 1024
-        self.camera_height = 768
+        self.camera_width = 960
+        self.camera_height = 960
         self.camera_enabled = False
         self.speaker_enabled = speaker_enabled
         self.speaker_sample_rate = speaker_sample_rate
@@ -169,6 +171,7 @@ class DailyTransportService(EventHandler):
             Daily.select_speaker_device("speaker")
 
         self.image: bytes | None = None
+        self.images: list[bytes] | None = None
         self.camera_thread = Thread(target=self.run_camera, daemon=True)
         self.camera_thread.start()
 
@@ -343,12 +346,23 @@ class DailyTransportService(EventHandler):
 
     def set_image(self, image: bytes):
         self.image: bytes | None = image
-
+        self.images: list[bytes] | None = None
+    
+    def set_images(self, images: list[bytes], start_frame=0):
+        self.images: list[bytes] | None = images
+        self.image = None
+        self.current_frame = start_frame
+    
     def run_camera(self):
         try:
             while not self.stop_threads.is_set():
                 if self.image:
                     self.camera.write_frame(self.image)
+                if self.images:
+                    frame_index = self.current_frame % len(self.images)
+                    this_frame = self.images[frame_index]
+                    self.camera.write_frame(this_frame)
+                    self.current_frame = frame_index + 1
 
                 time.sleep(1.0 / 8)  # 8 fps
         except Exception as e:
@@ -391,6 +405,8 @@ class DailyTransportService(EventHandler):
                                     b = b[l:]
                             elif isinstance(frame, ImageQueueFrame):
                                 self.set_image(frame.image)
+                            elif isinstance(frame, ImageListQueueFrame):
+                                self.set_images(frame.images)
                         elif len(b):
                             self.mic.write_frames(bytes(b))
                             b = bytearray()

src/samples/foundational/06a-golden-kitty.py

@@ -0,0 +1,174 @@
+import argparse
+import asyncio
+import os
+import random
+import requests
+import time
+import urllib.parse
+
+from dotenv import load_dotenv
+from PIL import Image
+
+load_dotenv()
+
+from dailyai.services.daily_transport_service import DailyTransportService
+from dailyai.services.azure_ai_services import AzureLLMService, AzureTTSService
+from dailyai.services.elevenlabs_ai_service import ElevenLabsTTSService
+from dailyai.services.fal_ai_services import FalImageGenService
+from dailyai.services.open_ai_services import OpenAIImageGenService
+from dailyai.queue_aggregators import LLMContextAggregator
+from dailyai.queue_frame import LLMMessagesQueueFrame, QueueFrame, TextQueueFrame, ImageQueueFrame, ImageListQueueFrame
+from dailyai.services.ai_services import AIService
+
+from typing import AsyncGenerator, List
+
+sprites = {}
+image_files = [
+    'cat1.png',
+    'cat2.png',
+    'cat3.png'
+]
+
+script_dir = os.path.dirname(__file__)
+
+for file in image_files:
+    # Build the full path to the image file
+    full_path = os.path.join(script_dir, "images", file)
+    # Get the filename without the extension to use as the dictionary key
+    filename = os.path.splitext(os.path.basename(full_path))[0]
+    # Open the image and convert it to bytes
+    with Image.open(full_path) as img:
+        sprites[file] = img.tobytes()
+
+quiet_frame = ImageQueueFrame("", sprites["cat1.png"])
+sprite_list = list(sprites.values())
+talking = [random.choice(sprite_list) for x in range(30)]
+talking_frame = ImageListQueueFrame(images=talking)
+class TranscriptFilter(AIService):
+    def __init__(self, bot_participant_id=None):
+        self.bot_participant_id = bot_participant_id
+
+    async def process_frame(self, frame:QueueFrame) -> AsyncGenerator[QueueFrame, None]:
+        if frame.participantId != self.bot_participant_id:
+            yield frame
+
+class NameCheckFilter(AIService):
+    def __init__(self, names=None):
+        self.names = names
+        self.sentence = ""
+
+    async def process_frame(self, frame:QueueFrame) -> AsyncGenerator[QueueFrame, None]:
+        content: str = ""
+
+        # TODO: split up transcription by participant
+        if isinstance(frame, TextQueueFrame):
+            content = frame.text
+
+        self.sentence += content
+        if self.sentence.endswith((".", "?", "!")):
+            if any(name in self.sentence for name in self.names):
+                print(f"I got one: {frame.text}")
+                out = self.sentence
+                self.sentence = ""
+                yield TextQueueFrame(out)
+            else:
+                out = self.sentence
+                self.sentence = ""
+                print(f"ignoring: {out}")
+
+async def main(room_url:str, token):
+    global transport
+    global llm
+    global tts
+
+    transport = DailyTransportService(
+        room_url,
+        token,
+        "Derrick",
+        180,
+    )
+    transport.mic_enabled = True
+    transport.mic_sample_rate = 16000
+    transport.camera_enabled = True
+    transport.camera_width = 960
+    transport.camera_height = 960
+
+    llm = AzureLLMService()
+    tts = ElevenLabsTTSService()
+
+    @transport.event_handler("on_first_other_participant_joined")
+    async def on_first_other_participant_joined(transport):
+        await tts.say("Hi, I'm listening!", transport.send_queue)
+
+    async def handle_transcriptions():
+        messages = [
+            {"role": "system", "content": "You are Derek, the Golden Kitty, the mascot for Product Hunt's annual awards. You are a cat who knows everything about all the cool new tech startups. You should be clever, and a bit sarcastic. You should also tell jokes every once in a while.  Your responses should only be a few sentences long."},
+        ]
+
+        tma_in = LLMContextAggregator(
+            messages, "user", transport.my_participant_id
+        )
+        tma_out = LLMContextAggregator(
+            messages, "assistant", transport.my_participant_id
+        )
+        tf = TranscriptFilter(transport.my_participant_id)
+        ncf = NameCheckFilter(["Derek", "Derrick"])
+        await tts.run_to_queue(
+            transport.send_queue,
+            tma_out.run(
+                llm.run(
+                    tma_in.run(
+                        ncf.run(
+                            tf.run(
+                                transport.get_receive_frames()
+                            )
+                        )
+
+                    )
+                )
+            )
+        )
+
+    async def make_cats():
+        await transport.send_queue.put(quiet_frame)
+        
+    transport.transcription_settings["extra"]["punctuate"] = True
+    await asyncio.gather(transport.run(), handle_transcriptions(), make_cats())
+
+
+
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="Simple Daily Bot Sample")
+    parser.add_argument(
+        "-u", "--url", type=str, required=True, help="URL of the Daily room to join"
+    )
+    parser.add_argument(
+        "-k",
+        "--apikey",
+        type=str,
+        required=True,
+        help="Daily API Key (needed to create token)",
+    )
+
+    args, unknown = parser.parse_known_args()
+
+    # Create a meeting token for the given room with an expiration 24 hours in the future.
+    room_name: str = urllib.parse.urlparse(args.url).path[1:]
+    expiration: float = time.time() + 60 * 60 * 24
+
+    res: requests.Response = requests.post(
+        f"https://api.daily.co/v1/meeting-tokens",
+        headers={"Authorization": f"Bearer {args.apikey}"},
+        json={
+            "properties": {"room_name": room_name, "is_owner": True, "exp": expiration}
+        },
+    )
+
+    if res.status_code != 200:
+        raise Exception(f"Failed to create meeting token: {res.status_code} {res.text}")
+
+    token: str = res.json()["token"]
+
+    asyncio.run(main(args.url, token))