Merge pull request #831 from pipecat-ai/mb/gemini-simple-chatbot
Gemini updates to the simple-chatbot demo
This commit is contained in:
@@ -2,7 +2,18 @@
|
|||||||
|
|
||||||
<img src="image.png" width="420px">
|
<img src="image.png" width="420px">
|
||||||
|
|
||||||
This repository demonstrates a simple AI chatbot with real-time audio/video interaction, implemented in three different ways. The bot server remains the same, but you can connect to it using three different client approaches.
|
This repository demonstrates a simple AI chatbot with real-time audio/video interaction, implemented in three different ways. The bot server supports multiple AI backends, and you can connect to it using three different client approaches.
|
||||||
|
|
||||||
|
## Two Bot Options
|
||||||
|
|
||||||
|
1. **OpenAI Bot** (Default)
|
||||||
|
|
||||||
|
- Uses gpt-4o for conversation
|
||||||
|
- Requires OpenAI API key
|
||||||
|
|
||||||
|
2. **Gemini Bot**
|
||||||
|
- Uses Google's Gemini Multimodal Live model
|
||||||
|
- Requires Gemini API key
|
||||||
|
|
||||||
## Three Ways to Connect
|
## Three Ways to Connect
|
||||||
|
|
||||||
@@ -13,13 +24,13 @@ This repository demonstrates a simple AI chatbot with real-time audio/video inte
|
|||||||
|
|
||||||
2. **JavaScript**
|
2. **JavaScript**
|
||||||
|
|
||||||
- Basic implementation using RTVI JavaScript SDK
|
- Basic implementation using [Pipecat JavaScript SDK](https://docs.pipecat.ai/client/reference/js/introduction)
|
||||||
- No framework dependencies
|
- No framework dependencies
|
||||||
- Good for learning the fundamentals
|
- Good for learning the fundamentals
|
||||||
|
|
||||||
3. **React**
|
3. **React**
|
||||||
- Basic impelmentation using RTVI React SDK
|
- Basic impelmentation using [Pipecat React SDK](https://docs.pipecat.ai/client/reference/react/introduction)
|
||||||
- Demonstrates the basic client principles with RTVI React
|
- Demonstrates the basic client principles with Pipecat React
|
||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
|
||||||
@@ -38,8 +49,12 @@ This repository demonstrates a simple AI chatbot with real-time audio/video inte
|
|||||||
```bash
|
```bash
|
||||||
pip install -r requirements.txt
|
pip install -r requirements.txt
|
||||||
```
|
```
|
||||||
4. Copy env.example to .env and add your credentials
|
4. Copy env.example to .env and configure:
|
||||||
|
- Add your API keys
|
||||||
|
- Choose your bot implementation:
|
||||||
|
```ini
|
||||||
|
BOT_IMPLEMENTATION= # Options: 'openai' (default) or 'gemini'
|
||||||
|
```
|
||||||
5. Start the server:
|
5. Start the server:
|
||||||
```bash
|
```bash
|
||||||
python server.py
|
python server.py
|
||||||
@@ -48,7 +63,7 @@ This repository demonstrates a simple AI chatbot with real-time audio/video inte
|
|||||||
### Next, connect using your preferred client app:
|
### Next, connect using your preferred client app:
|
||||||
|
|
||||||
- [Daily Prebuilt](examples/prebuilt/README.md)
|
- [Daily Prebuilt](examples/prebuilt/README.md)
|
||||||
- [Vanilla JavaScript Guide](examples/javascript/README.md)
|
- [JavaScript Guide](examples/javascript/README.md)
|
||||||
- [React Guide](examples/react/README.md)
|
- [React Guide](examples/react/README.md)
|
||||||
|
|
||||||
## Important Note
|
## Important Note
|
||||||
@@ -60,21 +75,23 @@ The bot server must be running for any of the client implementations to work. St
|
|||||||
- Python 3.10+
|
- Python 3.10+
|
||||||
- Node.js 16+ (for JavaScript and React implementations)
|
- Node.js 16+ (for JavaScript and React implementations)
|
||||||
- Daily API key
|
- Daily API key
|
||||||
- OpenAI API key
|
- OpenAI API key (for OpenAI bot)
|
||||||
- Cartesia API key
|
- Gemini API key (for Gemini bot)
|
||||||
|
- ElevenLabs API key
|
||||||
- Modern web browser with WebRTC support
|
- Modern web browser with WebRTC support
|
||||||
|
|
||||||
## Project Structure
|
## Project Structure
|
||||||
|
|
||||||
```
|
```
|
||||||
simple-chatbot-full-stack/
|
simple-chatbot/
|
||||||
├── server/ # Bot server implementation
|
├── server/ # Bot server implementation
|
||||||
│ ├── bot.py # Bot logic and media handling
|
│ ├── bot-openai.py # OpenAI bot implementation
|
||||||
│ ├── runner.py # Server runner utilities
|
│ ├── bot-gemini.py # Gemini bot implementation
|
||||||
│ ├── server.py # FastAPI server
|
│ ├── runner.py # Server runner utilities
|
||||||
|
│ ├── server.py # FastAPI server
|
||||||
│ └── requirements.txt
|
│ └── requirements.txt
|
||||||
└── examples/ # Client implementations
|
└── examples/ # Client implementations
|
||||||
├── prebuilt/ # Daily Prebuilt connection
|
├── prebuilt/ # Daily Prebuilt connection
|
||||||
├── javascript/ # JavaScript RTVI client
|
├── javascript/ # Pipecat JavaScript client
|
||||||
└── react/ # React RTVI client
|
└── react/ # Pipecat React client
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -1,10 +1,10 @@
|
|||||||
# JavaScript Implementation
|
# JavaScript Implementation
|
||||||
|
|
||||||
Basic implementation using the RTVI JavaScript SDK.
|
Basic implementation using the [Pipecat JavaScript SDK](https://docs.pipecat.ai/client/reference/js/introduction).
|
||||||
|
|
||||||
## Setup
|
## Setup
|
||||||
|
|
||||||
1. Run the bot server; see [README](../../README).
|
1. Run the bot server. See the [server README](../../README).
|
||||||
|
|
||||||
2. Navigate to the `examples/javascript` directory:
|
2. Navigate to the `examples/javascript` directory:
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# React Implementation
|
# React Implementation
|
||||||
|
|
||||||
Basic implementation using the RTVI React SDK.
|
Basic implementation using the [Pipecat React SDK](https://docs.pipecat.ai/client/reference/react/introduction).
|
||||||
|
|
||||||
## Setup
|
## Setup
|
||||||
|
|
||||||
|
|||||||
@@ -4,7 +4,7 @@
|
|||||||
<head>
|
<head>
|
||||||
<meta charset="UTF-8" />
|
<meta charset="UTF-8" />
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||||
<title>RTVI React Client</title>
|
<title>Pipecat React Client</title>
|
||||||
</head>
|
</head>
|
||||||
|
|
||||||
<body>
|
<body>
|
||||||
|
|||||||
@@ -1,11 +1,11 @@
|
|||||||
# Simple Chatbot Server
|
# Simple Chatbot Server
|
||||||
|
|
||||||
A FastAPI server that manages bot instances and provides endpoints for both Daily Prebuilt and RTVI client connections.
|
A FastAPI server that manages bot instances and provides endpoints for both Daily Prebuilt and Pipecat client connections.
|
||||||
|
|
||||||
## Endpoints
|
## Endpoints
|
||||||
|
|
||||||
- `GET /` - Direct browser access, redirects to a Daily Prebuilt room
|
- `GET /` - Direct browser access, redirects to a Daily Prebuilt room
|
||||||
- `POST /connect` - RTVI client connection endpoint
|
- `POST /connect` - Pipecat client connection endpoint
|
||||||
- `GET /status/{pid}` - Get status of a specific bot process
|
- `GET /status/{pid}` - Get status of a specific bot process
|
||||||
|
|
||||||
## Environment Variables
|
## Environment Variables
|
||||||
@@ -13,14 +13,37 @@ A FastAPI server that manages bot instances and provides endpoints for both Dail
|
|||||||
Copy `env.example` to `.env` and configure:
|
Copy `env.example` to `.env` and configure:
|
||||||
|
|
||||||
```ini
|
```ini
|
||||||
|
# Required API Keys
|
||||||
DAILY_API_KEY= # Your Daily API key
|
DAILY_API_KEY= # Your Daily API key
|
||||||
|
OPENAI_API_KEY= # Your OpenAI API key (required for OpenAI bot)
|
||||||
|
GEMINI_API_KEY= # Your Gemini API key (required for Gemini bot)
|
||||||
|
ELEVENLABS_API_KEY= # Your ElevenLabs API key
|
||||||
|
|
||||||
|
# Bot Selection
|
||||||
|
BOT_IMPLEMENTATION= # Options: 'openai' or 'gemini'
|
||||||
|
|
||||||
|
# Optional Configuration
|
||||||
DAILY_API_URL= # Optional: Daily API URL (defaults to https://api.daily.co/v1)
|
DAILY_API_URL= # Optional: Daily API URL (defaults to https://api.daily.co/v1)
|
||||||
OPENAI_API_KEY= # Your OpenAI API key
|
DAILY_SAMPLE_ROOM_URL= # Optional: Fixed room URL for development
|
||||||
CARTESIA_API_KEY= # Your Cartesia API key
|
|
||||||
HOST= # Optional: Host address (defaults to 0.0.0.0)
|
HOST= # Optional: Host address (defaults to 0.0.0.0)
|
||||||
FAST_API_PORT= # Optional: Port number (defaults to 7860)
|
FAST_API_PORT= # Optional: Port number (defaults to 7860)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Available Bots
|
||||||
|
|
||||||
|
The server supports two bot implementations:
|
||||||
|
|
||||||
|
1. **OpenAI Bot** (Default)
|
||||||
|
|
||||||
|
- Uses GPT-4 for conversation
|
||||||
|
- Requires OPENAI_API_KEY
|
||||||
|
|
||||||
|
2. **Gemini Bot**
|
||||||
|
- Uses Google's Gemini model
|
||||||
|
- Requires GEMINI_API_KEY
|
||||||
|
|
||||||
|
Select your preferred bot by setting `BOT_IMPLEMENTATION` in your `.env` file.
|
||||||
|
|
||||||
## Running the Server
|
## Running the Server
|
||||||
|
|
||||||
Set up and activate your virtual environment:
|
Set up and activate your virtual environment:
|
||||||
|
|||||||
223
examples/simple-chatbot/server/bot-gemini.py
Normal file
223
examples/simple-chatbot/server/bot-gemini.py
Normal file
@@ -0,0 +1,223 @@
|
|||||||
|
#
|
||||||
|
# Copyright (c) 2024, Daily
|
||||||
|
#
|
||||||
|
# SPDX-License-Identifier: BSD 2-Clause License
|
||||||
|
#
|
||||||
|
|
||||||
|
"""Gemini Bot Implementation.
|
||||||
|
|
||||||
|
This module implements a chatbot using Google's Gemini Multimodal Live model.
|
||||||
|
It includes:
|
||||||
|
- Real-time audio/video interaction through Daily
|
||||||
|
- Animated robot avatar
|
||||||
|
- Speech-to-speech model
|
||||||
|
|
||||||
|
The bot runs as part of a pipeline that processes audio/video frames and manages
|
||||||
|
the conversation flow using Gemini's streaming capabilities.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import aiohttp
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
from loguru import logger
|
||||||
|
from PIL import Image
|
||||||
|
from runner import configure
|
||||||
|
|
||||||
|
from pipecat.audio.vad.silero import SileroVADAnalyzer
|
||||||
|
from pipecat.audio.vad.vad_analyzer import VADParams
|
||||||
|
from pipecat.frames.frames import (
|
||||||
|
BotStartedSpeakingFrame,
|
||||||
|
BotStoppedSpeakingFrame,
|
||||||
|
EndFrame,
|
||||||
|
Frame,
|
||||||
|
OutputImageRawFrame,
|
||||||
|
SpriteFrame,
|
||||||
|
)
|
||||||
|
from pipecat.pipeline.pipeline import Pipeline
|
||||||
|
from pipecat.pipeline.runner import PipelineRunner
|
||||||
|
from pipecat.pipeline.task import PipelineParams, PipelineTask
|
||||||
|
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
|
||||||
|
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
|
||||||
|
from pipecat.processors.frameworks.rtvi import (
|
||||||
|
RTVIBotTranscriptionProcessor,
|
||||||
|
RTVIMetricsProcessor,
|
||||||
|
RTVISpeakingProcessor,
|
||||||
|
RTVIUserTranscriptionProcessor,
|
||||||
|
)
|
||||||
|
from pipecat.services.elevenlabs import ElevenLabsTTSService
|
||||||
|
from pipecat.services.gemini_multimodal_live.gemini import GeminiMultimodalLiveLLMService
|
||||||
|
from pipecat.services.openai import OpenAILLMService
|
||||||
|
from pipecat.transports.services.daily import DailyParams, DailyTransport
|
||||||
|
|
||||||
|
load_dotenv(override=True)
|
||||||
|
|
||||||
|
logger.remove(0)
|
||||||
|
logger.add(sys.stderr, level="DEBUG")
|
||||||
|
|
||||||
|
sprites = []
|
||||||
|
script_dir = os.path.dirname(__file__)
|
||||||
|
|
||||||
|
for i in range(1, 26):
|
||||||
|
# Build the full path to the image file
|
||||||
|
full_path = os.path.join(script_dir, f"assets/robot0{i}.png")
|
||||||
|
# Get the filename without the extension to use as the dictionary key
|
||||||
|
# Open the image and convert it to bytes
|
||||||
|
with Image.open(full_path) as img:
|
||||||
|
sprites.append(OutputImageRawFrame(image=img.tobytes(), size=img.size, format=img.format))
|
||||||
|
|
||||||
|
# Create a smooth animation by adding reversed frames
|
||||||
|
flipped = sprites[::-1]
|
||||||
|
sprites.extend(flipped)
|
||||||
|
|
||||||
|
# Define static and animated states
|
||||||
|
quiet_frame = sprites[0] # Static frame for when bot is listening
|
||||||
|
talking_frame = SpriteFrame(images=sprites) # Animation sequence for when bot is talking
|
||||||
|
|
||||||
|
|
||||||
|
class TalkingAnimation(FrameProcessor):
|
||||||
|
"""Manages the bot's visual animation states.
|
||||||
|
|
||||||
|
Switches between static (listening) and animated (talking) states based on
|
||||||
|
the bot's current speaking status.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
super().__init__()
|
||||||
|
self._is_talking = False
|
||||||
|
|
||||||
|
async def process_frame(self, frame: Frame, direction: FrameDirection):
|
||||||
|
"""Process incoming frames and update animation state.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
frame: The incoming frame to process
|
||||||
|
direction: The direction of frame flow in the pipeline
|
||||||
|
"""
|
||||||
|
await super().process_frame(frame, direction)
|
||||||
|
|
||||||
|
# Switch to talking animation when bot starts speaking
|
||||||
|
if isinstance(frame, BotStartedSpeakingFrame):
|
||||||
|
if not self._is_talking:
|
||||||
|
await self.push_frame(talking_frame)
|
||||||
|
self._is_talking = True
|
||||||
|
# Return to static frame when bot stops speaking
|
||||||
|
elif isinstance(frame, BotStoppedSpeakingFrame):
|
||||||
|
await self.push_frame(quiet_frame)
|
||||||
|
self._is_talking = False
|
||||||
|
|
||||||
|
await self.push_frame(frame, direction)
|
||||||
|
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
"""Main bot execution function.
|
||||||
|
|
||||||
|
Sets up and runs the bot pipeline including:
|
||||||
|
- Daily video transport with specific audio parameters
|
||||||
|
- Gemini Live multimodal model integration
|
||||||
|
- Voice activity detection
|
||||||
|
- Animation processing
|
||||||
|
- RTVI event handling
|
||||||
|
"""
|
||||||
|
async with aiohttp.ClientSession() as session:
|
||||||
|
(room_url, token) = await configure(session)
|
||||||
|
|
||||||
|
# Set up Daily transport with specific audio/video parameters for Gemini
|
||||||
|
transport = DailyTransport(
|
||||||
|
room_url,
|
||||||
|
token,
|
||||||
|
"Chatbot",
|
||||||
|
DailyParams(
|
||||||
|
audio_in_sample_rate=16000,
|
||||||
|
audio_out_sample_rate=24000,
|
||||||
|
audio_out_enabled=True,
|
||||||
|
camera_out_enabled=True,
|
||||||
|
camera_out_width=1024,
|
||||||
|
camera_out_height=576,
|
||||||
|
vad_enabled=True,
|
||||||
|
vad_audio_passthrough=True,
|
||||||
|
vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.5)),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Initialize the Gemini Multimodal Live model
|
||||||
|
llm = GeminiMultimodalLiveLLMService(
|
||||||
|
api_key=os.getenv("GEMINI_API_KEY"),
|
||||||
|
voice_id="Puck", # Aoede, Charon, Fenrir, Kore, Puck
|
||||||
|
transcribe_user_audio=True,
|
||||||
|
transcribe_model_audio=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
messages = [
|
||||||
|
{
|
||||||
|
"role": "user",
|
||||||
|
"content": "You are Chatbot, a friendly, helpful robot. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by introducing yourself.",
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
# Set up conversation context and management
|
||||||
|
# The context_aggregator will automatically collect conversation context
|
||||||
|
context = OpenAILLMContext(messages)
|
||||||
|
context_aggregator = llm.create_context_aggregator(context)
|
||||||
|
|
||||||
|
ta = TalkingAnimation()
|
||||||
|
|
||||||
|
#
|
||||||
|
# RTVI events for Pipecat client UI
|
||||||
|
#
|
||||||
|
|
||||||
|
# This will send `user-*-speaking` and `bot-*-speaking` messages.
|
||||||
|
rtvi_speaking = RTVISpeakingProcessor()
|
||||||
|
|
||||||
|
# This will emit UserTranscript events.
|
||||||
|
rtvi_user_transcription = RTVIUserTranscriptionProcessor()
|
||||||
|
|
||||||
|
# This will emit BotTranscript events.
|
||||||
|
rtvi_bot_transcription = RTVIBotTranscriptionProcessor()
|
||||||
|
|
||||||
|
# This will send `metrics` messages.
|
||||||
|
rtvi_metrics = RTVIMetricsProcessor()
|
||||||
|
|
||||||
|
pipeline = Pipeline(
|
||||||
|
[
|
||||||
|
transport.input(),
|
||||||
|
context_aggregator.user(),
|
||||||
|
llm,
|
||||||
|
rtvi_speaking,
|
||||||
|
rtvi_user_transcription,
|
||||||
|
rtvi_bot_transcription,
|
||||||
|
ta,
|
||||||
|
rtvi_metrics,
|
||||||
|
transport.output(),
|
||||||
|
context_aggregator.assistant(),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
task = PipelineTask(
|
||||||
|
pipeline,
|
||||||
|
PipelineParams(
|
||||||
|
allow_interruptions=True,
|
||||||
|
enable_metrics=True,
|
||||||
|
enable_usage_metrics=True,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
await task.queue_frame(quiet_frame)
|
||||||
|
|
||||||
|
@transport.event_handler("on_first_participant_joined")
|
||||||
|
async def on_first_participant_joined(transport, participant):
|
||||||
|
await transport.capture_participant_transcription(participant["id"])
|
||||||
|
await task.queue_frames([context_aggregator.user().get_context_frame()])
|
||||||
|
|
||||||
|
@transport.event_handler("on_participant_left")
|
||||||
|
async def on_participant_left(transport, participant, reason):
|
||||||
|
print(f"Participant left: {participant}")
|
||||||
|
await task.queue_frame(EndFrame())
|
||||||
|
|
||||||
|
runner = PipelineRunner()
|
||||||
|
|
||||||
|
await runner.run(task)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
asyncio.run(main())
|
||||||
@@ -4,6 +4,19 @@
|
|||||||
# SPDX-License-Identifier: BSD 2-Clause License
|
# SPDX-License-Identifier: BSD 2-Clause License
|
||||||
#
|
#
|
||||||
|
|
||||||
|
"""OpenAI Bot Implementation.
|
||||||
|
|
||||||
|
This module implements a chatbot using OpenAI's GPT-4 model for natural language
|
||||||
|
processing. It includes:
|
||||||
|
- Real-time audio/video interaction through Daily
|
||||||
|
- Animated robot avatar
|
||||||
|
- Text-to-speech using ElevenLabs
|
||||||
|
- Support for both English and Spanish
|
||||||
|
|
||||||
|
The bot runs as part of a pipeline that processes audio/video frames and manages
|
||||||
|
the conversation flow.
|
||||||
|
"""
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import os
|
import os
|
||||||
import sys
|
import sys
|
||||||
@@ -31,6 +44,8 @@ from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
|
|||||||
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
|
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
|
||||||
from pipecat.processors.frameworks.rtvi import (
|
from pipecat.processors.frameworks.rtvi import (
|
||||||
RTVIBotTranscriptionProcessor,
|
RTVIBotTranscriptionProcessor,
|
||||||
|
RTVIMetricsProcessor,
|
||||||
|
RTVISpeakingProcessor,
|
||||||
RTVIUserTranscriptionProcessor,
|
RTVIUserTranscriptionProcessor,
|
||||||
)
|
)
|
||||||
from pipecat.services.elevenlabs import ElevenLabsTTSService
|
from pipecat.services.elevenlabs import ElevenLabsTTSService
|
||||||
@@ -38,14 +53,13 @@ from pipecat.services.openai import OpenAILLMService
|
|||||||
from pipecat.transports.services.daily import DailyParams, DailyTransport
|
from pipecat.transports.services.daily import DailyParams, DailyTransport
|
||||||
|
|
||||||
load_dotenv(override=True)
|
load_dotenv(override=True)
|
||||||
|
|
||||||
logger.remove(0)
|
logger.remove(0)
|
||||||
logger.add(sys.stderr, level="DEBUG")
|
logger.add(sys.stderr, level="DEBUG")
|
||||||
|
|
||||||
sprites = []
|
sprites = []
|
||||||
|
|
||||||
script_dir = os.path.dirname(__file__)
|
script_dir = os.path.dirname(__file__)
|
||||||
|
|
||||||
|
# Load sequential animation frames
|
||||||
for i in range(1, 26):
|
for i in range(1, 26):
|
||||||
# Build the full path to the image file
|
# Build the full path to the image file
|
||||||
full_path = os.path.join(script_dir, f"assets/robot0{i}.png")
|
full_path = os.path.join(script_dir, f"assets/robot0{i}.png")
|
||||||
@@ -54,18 +68,20 @@ for i in range(1, 26):
|
|||||||
with Image.open(full_path) as img:
|
with Image.open(full_path) as img:
|
||||||
sprites.append(OutputImageRawFrame(image=img.tobytes(), size=img.size, format=img.format))
|
sprites.append(OutputImageRawFrame(image=img.tobytes(), size=img.size, format=img.format))
|
||||||
|
|
||||||
|
# Create a smooth animation by adding reversed frames
|
||||||
flipped = sprites[::-1]
|
flipped = sprites[::-1]
|
||||||
sprites.extend(flipped)
|
sprites.extend(flipped)
|
||||||
|
|
||||||
# When the bot isn't talking, show a static image of the cat listening
|
# Define static and animated states
|
||||||
quiet_frame = sprites[0]
|
quiet_frame = sprites[0] # Static frame for when bot is listening
|
||||||
talking_frame = SpriteFrame(images=sprites)
|
talking_frame = SpriteFrame(images=sprites) # Animation sequence for when bot is talking
|
||||||
|
|
||||||
|
|
||||||
class TalkingAnimation(FrameProcessor):
|
class TalkingAnimation(FrameProcessor):
|
||||||
"""
|
"""Manages the bot's visual animation states.
|
||||||
This class starts a talking animation when it receives an first AudioFrame,
|
|
||||||
and then returns to a "quiet" sprite when it sees a TTSStoppedFrame.
|
Switches between static (listening) and animated (talking) states based on
|
||||||
|
the bot's current speaking status.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self):
|
def __init__(self):
|
||||||
@@ -73,12 +89,20 @@ class TalkingAnimation(FrameProcessor):
|
|||||||
self._is_talking = False
|
self._is_talking = False
|
||||||
|
|
||||||
async def process_frame(self, frame: Frame, direction: FrameDirection):
|
async def process_frame(self, frame: Frame, direction: FrameDirection):
|
||||||
|
"""Process incoming frames and update animation state.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
frame: The incoming frame to process
|
||||||
|
direction: The direction of frame flow in the pipeline
|
||||||
|
"""
|
||||||
await super().process_frame(frame, direction)
|
await super().process_frame(frame, direction)
|
||||||
|
|
||||||
|
# Switch to talking animation when bot starts speaking
|
||||||
if isinstance(frame, BotStartedSpeakingFrame):
|
if isinstance(frame, BotStartedSpeakingFrame):
|
||||||
if not self._is_talking:
|
if not self._is_talking:
|
||||||
await self.push_frame(talking_frame)
|
await self.push_frame(talking_frame)
|
||||||
self._is_talking = True
|
self._is_talking = True
|
||||||
|
# Return to static frame when bot stops speaking
|
||||||
elif isinstance(frame, BotStoppedSpeakingFrame):
|
elif isinstance(frame, BotStoppedSpeakingFrame):
|
||||||
await self.push_frame(quiet_frame)
|
await self.push_frame(quiet_frame)
|
||||||
self._is_talking = False
|
self._is_talking = False
|
||||||
@@ -87,9 +111,19 @@ class TalkingAnimation(FrameProcessor):
|
|||||||
|
|
||||||
|
|
||||||
async def main():
|
async def main():
|
||||||
|
"""Main bot execution function.
|
||||||
|
|
||||||
|
Sets up and runs the bot pipeline including:
|
||||||
|
- Daily video transport
|
||||||
|
- Speech-to-text and text-to-speech services
|
||||||
|
- Language model integration
|
||||||
|
- Animation processing
|
||||||
|
- RTVI event handling
|
||||||
|
"""
|
||||||
async with aiohttp.ClientSession() as session:
|
async with aiohttp.ClientSession() as session:
|
||||||
(room_url, token) = await configure(session)
|
(room_url, token) = await configure(session)
|
||||||
|
|
||||||
|
# Set up Daily transport with video/audio parameters
|
||||||
transport = DailyTransport(
|
transport = DailyTransport(
|
||||||
room_url,
|
room_url,
|
||||||
token,
|
token,
|
||||||
@@ -113,6 +147,7 @@ async def main():
|
|||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Initialize text-to-speech service
|
||||||
tts = ElevenLabsTTSService(
|
tts = ElevenLabsTTSService(
|
||||||
api_key=os.getenv("ELEVENLABS_API_KEY"),
|
api_key=os.getenv("ELEVENLABS_API_KEY"),
|
||||||
#
|
#
|
||||||
@@ -126,6 +161,7 @@ async def main():
|
|||||||
# voice_id="gD1IexrzCvsXPHUuT0s3",
|
# voice_id="gD1IexrzCvsXPHUuT0s3",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Initialize LLM service
|
||||||
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
|
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
|
||||||
|
|
||||||
messages = [
|
messages = [
|
||||||
@@ -142,12 +178,19 @@ async def main():
|
|||||||
},
|
},
|
||||||
]
|
]
|
||||||
|
|
||||||
|
# Set up conversation context and management
|
||||||
|
# The context_aggregator will automatically collect conversation context
|
||||||
context = OpenAILLMContext(messages)
|
context = OpenAILLMContext(messages)
|
||||||
context_aggregator = llm.create_context_aggregator(context)
|
context_aggregator = llm.create_context_aggregator(context)
|
||||||
|
|
||||||
ta = TalkingAnimation()
|
ta = TalkingAnimation()
|
||||||
|
|
||||||
# RTVI
|
#
|
||||||
|
# RTVI events for Pipecat client UI
|
||||||
|
#
|
||||||
|
|
||||||
|
# This will send `user-*-speaking` and `bot-*-speaking` messages.
|
||||||
|
rtvi_speaking = RTVISpeakingProcessor()
|
||||||
|
|
||||||
# This will emit UserTranscript events.
|
# This will emit UserTranscript events.
|
||||||
rtvi_user_transcription = RTVIUserTranscriptionProcessor()
|
rtvi_user_transcription = RTVIUserTranscriptionProcessor()
|
||||||
@@ -155,21 +198,33 @@ async def main():
|
|||||||
# This will emit BotTranscript events.
|
# This will emit BotTranscript events.
|
||||||
rtvi_bot_transcription = RTVIBotTranscriptionProcessor()
|
rtvi_bot_transcription = RTVIBotTranscriptionProcessor()
|
||||||
|
|
||||||
|
# This will send `metrics` messages.
|
||||||
|
rtvi_metrics = RTVIMetricsProcessor()
|
||||||
|
|
||||||
pipeline = Pipeline(
|
pipeline = Pipeline(
|
||||||
[
|
[
|
||||||
transport.input(),
|
transport.input(),
|
||||||
|
rtvi_speaking,
|
||||||
rtvi_user_transcription,
|
rtvi_user_transcription,
|
||||||
context_aggregator.user(),
|
context_aggregator.user(),
|
||||||
llm,
|
llm,
|
||||||
rtvi_bot_transcription,
|
rtvi_bot_transcription,
|
||||||
tts,
|
tts,
|
||||||
ta,
|
ta,
|
||||||
|
rtvi_metrics,
|
||||||
transport.output(),
|
transport.output(),
|
||||||
context_aggregator.assistant(),
|
context_aggregator.assistant(),
|
||||||
]
|
]
|
||||||
)
|
)
|
||||||
|
|
||||||
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
|
task = PipelineTask(
|
||||||
|
pipeline,
|
||||||
|
PipelineParams(
|
||||||
|
allow_interruptions=True,
|
||||||
|
enable_metrics=True,
|
||||||
|
enable_usage_metrics=True,
|
||||||
|
),
|
||||||
|
)
|
||||||
await task.queue_frame(quiet_frame)
|
await task.queue_frame(quiet_frame)
|
||||||
|
|
||||||
@transport.event_handler("on_first_participant_joined")
|
@transport.event_handler("on_first_participant_joined")
|
||||||
@@ -1,4 +1,6 @@
|
|||||||
DAILY_SAMPLE_ROOM_URL=https://yourdomain.daily.co/yourroom # (for joining the bot to the same room repeatedly for local dev)
|
DAILY_SAMPLE_ROOM_URL=https://yourdomain.daily.co/yourroom # (for joining the bot to the same room repeatedly for local dev)
|
||||||
DAILY_API_KEY=7df...
|
DAILY_API_KEY=7df...
|
||||||
OPENAI_API_KEY=sk-PL...
|
OPENAI_API_KEY=sk-PL...
|
||||||
|
GEMINI_API_KEY=AIza...
|
||||||
ELEVENLABS_API_KEY=aeb...
|
ELEVENLABS_API_KEY=aeb...
|
||||||
|
BOT_IMPLEMENTATION= # Options: 'openai' or 'gemini'
|
||||||
@@ -4,14 +4,16 @@
|
|||||||
# SPDX-License-Identifier: BSD 2-Clause License
|
# SPDX-License-Identifier: BSD 2-Clause License
|
||||||
#
|
#
|
||||||
|
|
||||||
import aiohttp
|
|
||||||
import argparse
|
import argparse
|
||||||
import os
|
import os
|
||||||
|
|
||||||
|
import aiohttp
|
||||||
|
|
||||||
from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper
|
from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper
|
||||||
|
|
||||||
|
|
||||||
async def configure(aiohttp_session: aiohttp.ClientSession):
|
async def configure(aiohttp_session: aiohttp.ClientSession):
|
||||||
|
"""Configure the Daily room and Daily REST helper."""
|
||||||
parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample")
|
parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample")
|
||||||
parser.add_argument(
|
parser.add_argument(
|
||||||
"-u", "--url", type=str, required=False, help="URL of the Daily room to join"
|
"-u", "--url", type=str, required=False, help="URL of the Daily room to join"
|
||||||
|
|||||||
@@ -4,8 +4,7 @@
|
|||||||
# SPDX-License-Identifier: BSD 2-Clause License
|
# SPDX-License-Identifier: BSD 2-Clause License
|
||||||
#
|
#
|
||||||
|
|
||||||
"""
|
"""RTVI Bot Server Implementation.
|
||||||
RTVI Bot Server Implementation
|
|
||||||
|
|
||||||
This FastAPI server manages RTVI bot instances and provides endpoints for both
|
This FastAPI server manages RTVI bot instances and provides endpoints for both
|
||||||
direct browser access and RTVI client connections. It handles:
|
direct browser access and RTVI client connections. It handles:
|
||||||
@@ -49,8 +48,8 @@ daily_helpers = {}
|
|||||||
|
|
||||||
|
|
||||||
def cleanup():
|
def cleanup():
|
||||||
"""
|
"""Cleanup function to terminate all bot processes.
|
||||||
Cleanup function to terminate all bot processes.
|
|
||||||
Called during server shutdown.
|
Called during server shutdown.
|
||||||
"""
|
"""
|
||||||
for entry in bot_procs.values():
|
for entry in bot_procs.values():
|
||||||
@@ -59,10 +58,22 @@ def cleanup():
|
|||||||
proc.wait()
|
proc.wait()
|
||||||
|
|
||||||
|
|
||||||
|
def get_bot_file():
|
||||||
|
bot_implementation = os.getenv("BOT_IMPLEMENTATION", "openai").lower().strip()
|
||||||
|
# If blank or None, default to openai
|
||||||
|
if not bot_implementation:
|
||||||
|
bot_implementation = "openai"
|
||||||
|
if bot_implementation not in ["openai", "gemini"]:
|
||||||
|
raise ValueError(
|
||||||
|
f"Invalid BOT_IMPLEMENTATION: {bot_implementation}. Must be 'openai' or 'gemini'"
|
||||||
|
)
|
||||||
|
return f"bot-{bot_implementation}"
|
||||||
|
|
||||||
|
|
||||||
@asynccontextmanager
|
@asynccontextmanager
|
||||||
async def lifespan(app: FastAPI):
|
async def lifespan(app: FastAPI):
|
||||||
"""
|
"""FastAPI lifespan manager that handles startup and shutdown tasks.
|
||||||
FastAPI lifespan manager that handles startup and shutdown tasks:
|
|
||||||
- Creates aiohttp session
|
- Creates aiohttp session
|
||||||
- Initializes Daily API helper
|
- Initializes Daily API helper
|
||||||
- Cleans up resources on shutdown
|
- Cleans up resources on shutdown
|
||||||
@@ -92,8 +103,7 @@ app.add_middleware(
|
|||||||
|
|
||||||
|
|
||||||
async def create_room_and_token() -> tuple[str, str]:
|
async def create_room_and_token() -> tuple[str, str]:
|
||||||
"""
|
"""Helper function to create a Daily room and generate an access token.
|
||||||
Helper function to create a Daily room and generate an access token.
|
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
tuple[str, str]: A tuple containing (room_url, token)
|
tuple[str, str]: A tuple containing (room_url, token)
|
||||||
@@ -114,8 +124,8 @@ async def create_room_and_token() -> tuple[str, str]:
|
|||||||
|
|
||||||
@app.get("/")
|
@app.get("/")
|
||||||
async def start_agent(request: Request):
|
async def start_agent(request: Request):
|
||||||
"""
|
"""Endpoint for direct browser access to the bot.
|
||||||
Endpoint for direct browser access to the bot.
|
|
||||||
Creates a room, starts a bot instance, and redirects to the Daily room URL.
|
Creates a room, starts a bot instance, and redirects to the Daily room URL.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
@@ -137,8 +147,9 @@ async def start_agent(request: Request):
|
|||||||
|
|
||||||
# Spawn a new bot process
|
# Spawn a new bot process
|
||||||
try:
|
try:
|
||||||
|
bot_file = get_bot_file()
|
||||||
proc = subprocess.Popen(
|
proc = subprocess.Popen(
|
||||||
[f"python3 -m bot -u {room_url} -t {token}"],
|
[f"python3 -m {bot_file} -u {room_url} -t {token}"],
|
||||||
shell=True,
|
shell=True,
|
||||||
bufsize=1,
|
bufsize=1,
|
||||||
cwd=os.path.dirname(os.path.abspath(__file__)),
|
cwd=os.path.dirname(os.path.abspath(__file__)),
|
||||||
@@ -152,8 +163,8 @@ async def start_agent(request: Request):
|
|||||||
|
|
||||||
@app.post("/connect")
|
@app.post("/connect")
|
||||||
async def rtvi_connect(request: Request) -> Dict[Any, Any]:
|
async def rtvi_connect(request: Request) -> Dict[Any, Any]:
|
||||||
"""
|
"""RTVI connect endpoint that creates a room and returns connection credentials.
|
||||||
RTVI connect endpoint that creates a room and returns connection credentials.
|
|
||||||
This endpoint is called by RTVI clients to establish a connection.
|
This endpoint is called by RTVI clients to establish a connection.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
@@ -168,8 +179,9 @@ async def rtvi_connect(request: Request) -> Dict[Any, Any]:
|
|||||||
|
|
||||||
# Start the bot process
|
# Start the bot process
|
||||||
try:
|
try:
|
||||||
|
bot_file = get_bot_file()
|
||||||
proc = subprocess.Popen(
|
proc = subprocess.Popen(
|
||||||
[f"python3 -m bot -u {room_url} -t {token}"],
|
[f"python3 -m {bot_file} -u {room_url} -t {token}"],
|
||||||
shell=True,
|
shell=True,
|
||||||
bufsize=1,
|
bufsize=1,
|
||||||
cwd=os.path.dirname(os.path.abspath(__file__)),
|
cwd=os.path.dirname(os.path.abspath(__file__)),
|
||||||
@@ -184,8 +196,7 @@ async def rtvi_connect(request: Request) -> Dict[Any, Any]:
|
|||||||
|
|
||||||
@app.get("/status/{pid}")
|
@app.get("/status/{pid}")
|
||||||
def get_status(pid: int):
|
def get_status(pid: int):
|
||||||
"""
|
"""Get the status of a specific bot process.
|
||||||
Get the status of a specific bot process.
|
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
pid (int): Process ID of the bot
|
pid (int): Process ID of the bot
|
||||||
|
|||||||
Reference in New Issue
Block a user