wx44wx/pipecat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #831 from pipecat-ai/mb/gemini-simple-chatbot

Browse Source 
Gemini updates to the simple-chatbot demo
This commit is contained in:
Mark Backman
2024-12-11 11:15:15 -05:00
committed by GitHub
parent 62ec2f5d1e 2846d6f461
commit 3341c4f608
10 changed files with 387 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files

53
examples/simple-chatbot/README.md
View File

@@ -2,7 +2,18 @@
<img src="image.png" width="420px">
This repository demonstrates a simple AI chatbot with real-time audio/video interaction, implemented in three different ways. The bot server remains the same, but you can connect to it using three different client approaches.
This repository demonstrates a simple AI chatbot with real-time audio/video interaction, implemented in three different ways. The bot server supports multiple AI backends, and you can connect to it using three different client approaches.
## Two Bot Options
1. **OpenAI Bot** (Default)
- Uses gpt-4o for conversation
- Requires OpenAI API key
2. **Gemini Bot**
- Uses Google's Gemini Multimodal Live model
- Requires Gemini API key
## Three Ways to Connect
@@ -13,13 +24,13 @@ This repository demonstrates a simple AI chatbot with real-time audio/video inte
2. **JavaScript**
- Basic implementation using RTVI JavaScript SDK
- Basic implementation using [Pipecat JavaScript SDK](https://docs.pipecat.ai/client/reference/js/introduction)
- No framework dependencies
- Good for learning the fundamentals
3. **React**
- Basic impelmentation using RTVI React SDK
- Demonstrates the basic client principles with RTVI React
- Basic impelmentation using [Pipecat React SDK](https://docs.pipecat.ai/client/reference/react/introduction)
- Demonstrates the basic client principles with Pipecat React
## Quick Start
@@ -38,8 +49,12 @@ This repository demonstrates a simple AI chatbot with real-time audio/video inte
```bash
pip install -r requirements.txt
```
4. Copy env.example to .env and add your credentials
4. Copy env.example to .env and configure:
- Add your API keys
- Choose your bot implementation:
```ini
BOT_IMPLEMENTATION= # Options: 'openai' (default) or 'gemini'
```
5. Start the server:
```bash
python server.py
@@ -48,7 +63,7 @@ This repository demonstrates a simple AI chatbot with real-time audio/video inte
### Next, connect using your preferred client app:
- [Daily Prebuilt](examples/prebuilt/README.md)
- [Vanilla JavaScript Guide](examples/javascript/README.md)
- [JavaScript Guide](examples/javascript/README.md)
- [React Guide](examples/react/README.md)
## Important Note
@@ -60,21 +75,23 @@ The bot server must be running for any of the client implementations to work. St
- Python 3.10+
- Node.js 16+ (for JavaScript and React implementations)
- Daily API key
- OpenAI API key
- Cartesia API key
- OpenAI API key (for OpenAI bot)
- Gemini API key (for Gemini bot)
- ElevenLabs API key
- Modern web browser with WebRTC support
## Project Structure
```
simple-chatbot-full-stack/
├── server/ # Bot server implementation
│ ├── bot.py # Bot logic and media handling
│ ├── runner.py # Server runner utilities
│ ├── server.py # FastAPI server
simple-chatbot/
├── server/ # Bot server implementation
│ ├── bot-openai.py # OpenAI bot implementation
│ ├── bot-gemini.py # Gemini bot implementation
│ ├── runner.py # Server runner utilities
│ ├── server.py # FastAPI server
│ └── requirements.txt
└── examples/ # Client implementations
├── prebuilt/ # Daily Prebuilt connection
├── javascript/ # JavaScript RTVI client
└── react/ # React RTVI client
└── examples/ # Client implementations
├── prebuilt/ # Daily Prebuilt connection
├── javascript/ # Pipecat JavaScript client
└── react/ # Pipecat React client
```

4
examples/simple-chatbot/examples/javascript/README.md
View File

@@ -1,10 +1,10 @@
# JavaScript Implementation
Basic implementation using the RTVI JavaScript SDK.
Basic implementation using the [Pipecat JavaScript SDK](https://docs.pipecat.ai/client/reference/js/introduction).
## Setup
1. Run the bot server; see [README](../../README).
1. Run the bot server. See the [server README](../../README).
2. Navigate to the `examples/javascript` directory:

2
examples/simple-chatbot/examples/react/README.md
View File

@@ -1,6 +1,6 @@
# React Implementation
Basic implementation using the RTVI React SDK.
Basic implementation using the [Pipecat React SDK](https://docs.pipecat.ai/client/reference/react/introduction).
## Setup

2
examples/simple-chatbot/examples/react/index.html
View File

@@ -4,7 +4,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>RTVI React Client</title>
<title>Pipecat React Client</title>
</head>
<body>

31
examples/simple-chatbot/server/README.md
View File

@@ -1,11 +1,11 @@
# Simple Chatbot Server
A FastAPI server that manages bot instances and provides endpoints for both Daily Prebuilt and RTVI client connections.
A FastAPI server that manages bot instances and provides endpoints for both Daily Prebuilt and Pipecat client connections.
## Endpoints
- `GET /` - Direct browser access, redirects to a Daily Prebuilt room
- `POST /connect` - RTVI client connection endpoint
- `POST /connect` - Pipecat client connection endpoint
- `GET /status/{pid}` - Get status of a specific bot process
## Environment Variables
@@ -13,14 +13,37 @@ A FastAPI server that manages bot instances and provides endpoints for both Dail
Copy `env.example` to `.env` and configure:
```ini
# Required API Keys
DAILY_API_KEY= # Your Daily API key
OPENAI_API_KEY= # Your OpenAI API key (required for OpenAI bot)
GEMINI_API_KEY= # Your Gemini API key (required for Gemini bot)
ELEVENLABS_API_KEY= # Your ElevenLabs API key
# Bot Selection
BOT_IMPLEMENTATION= # Options: 'openai' or 'gemini'
# Optional Configuration
DAILY_API_URL= # Optional: Daily API URL (defaults to https://api.daily.co/v1)
OPENAI_API_KEY= # Your OpenAI API key
CARTESIA_API_KEY= # Your Cartesia API key
DAILY_SAMPLE_ROOM_URL= # Optional: Fixed room URL for development
HOST= # Optional: Host address (defaults to 0.0.0.0)
FAST_API_PORT= # Optional: Port number (defaults to 7860)
```
## Available Bots
The server supports two bot implementations:
1. **OpenAI Bot** (Default)
- Uses GPT-4 for conversation
- Requires OPENAI_API_KEY
2. **Gemini Bot**
- Uses Google's Gemini model
- Requires GEMINI_API_KEY
Select your preferred bot by setting `BOT_IMPLEMENTATION` in your `.env` file.
## Running the Server
Set up and activate your virtual environment:

223
examples/simple-chatbot/server/bot-gemini.py Normal file
View File

@@ -0,0 +1,223 @@
#
# Copyright (c) 2024, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
"""Gemini Bot Implementation.
This module implements a chatbot using Google's Gemini Multimodal Live model.
It includes:
- Real-time audio/video interaction through Daily
- Animated robot avatar
- Speech-to-speech model
The bot runs as part of a pipeline that processes audio/video frames and manages
the conversation flow using Gemini's streaming capabilities.
"""
import asyncio
import os
import sys
import aiohttp
from dotenv import load_dotenv
from loguru import logger
from PIL import Image
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
from pipecat.frames.frames import (
BotStartedSpeakingFrame,
BotStoppedSpeakingFrame,
EndFrame,
Frame,
OutputImageRawFrame,
SpriteFrame,
)
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.processors.frameworks.rtvi import (
RTVIBotTranscriptionProcessor,
RTVIMetricsProcessor,
RTVISpeakingProcessor,
RTVIUserTranscriptionProcessor,
)
from pipecat.services.elevenlabs import ElevenLabsTTSService
from pipecat.services.gemini_multimodal_live.gemini import GeminiMultimodalLiveLLMService
from pipecat.services.openai import OpenAILLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport
load_dotenv(override=True)
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
sprites = []
script_dir = os.path.dirname(__file__)
for i in range(1, 26):
# Build the full path to the image file
full_path = os.path.join(script_dir, f"assets/robot0{i}.png")
# Get the filename without the extension to use as the dictionary key
# Open the image and convert it to bytes
with Image.open(full_path) as img:
sprites.append(OutputImageRawFrame(image=img.tobytes(), size=img.size, format=img.format))
# Create a smooth animation by adding reversed frames
flipped = sprites[::-1]
sprites.extend(flipped)
# Define static and animated states
quiet_frame = sprites[0] # Static frame for when bot is listening
talking_frame = SpriteFrame(images=sprites) # Animation sequence for when bot is talking
class TalkingAnimation(FrameProcessor):
"""Manages the bot's visual animation states.
Switches between static (listening) and animated (talking) states based on
the bot's current speaking status.
"""
def __init__(self):
super().__init__()
self._is_talking = False
async def process_frame(self, frame: Frame, direction: FrameDirection):
"""Process incoming frames and update animation state.
Args:
frame: The incoming frame to process
direction: The direction of frame flow in the pipeline
"""
await super().process_frame(frame, direction)
# Switch to talking animation when bot starts speaking
if isinstance(frame, BotStartedSpeakingFrame):
if not self._is_talking:
await self.push_frame(talking_frame)
self._is_talking = True
# Return to static frame when bot stops speaking
elif isinstance(frame, BotStoppedSpeakingFrame):
await self.push_frame(quiet_frame)
self._is_talking = False
await self.push_frame(frame, direction)
async def main():
"""Main bot execution function.
Sets up and runs the bot pipeline including:
- Daily video transport with specific audio parameters
- Gemini Live multimodal model integration
- Voice activity detection
- Animation processing
- RTVI event handling
"""
async with aiohttp.ClientSession() as session:
(room_url, token) = await configure(session)
# Set up Daily transport with specific audio/video parameters for Gemini
transport = DailyTransport(
room_url,
token,
"Chatbot",
DailyParams(
audio_in_sample_rate=16000,
audio_out_sample_rate=24000,
audio_out_enabled=True,
camera_out_enabled=True,
camera_out_width=1024,
camera_out_height=576,
vad_enabled=True,
vad_audio_passthrough=True,
vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.5)),
),
)
# Initialize the Gemini Multimodal Live model
llm = GeminiMultimodalLiveLLMService(
api_key=os.getenv("GEMINI_API_KEY"),
voice_id="Puck", # Aoede, Charon, Fenrir, Kore, Puck
transcribe_user_audio=True,
transcribe_model_audio=True,
)
messages = [
{
"role": "user",
"content": "You are Chatbot, a friendly, helpful robot. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by introducing yourself.",
},
]
# Set up conversation context and management
# The context_aggregator will automatically collect conversation context
context = OpenAILLMContext(messages)
context_aggregator = llm.create_context_aggregator(context)
ta = TalkingAnimation()
#
# RTVI events for Pipecat client UI
#
# This will send `user-*-speaking` and `bot-*-speaking` messages.
rtvi_speaking = RTVISpeakingProcessor()
# This will emit UserTranscript events.
rtvi_user_transcription = RTVIUserTranscriptionProcessor()
# This will emit BotTranscript events.
rtvi_bot_transcription = RTVIBotTranscriptionProcessor()
# This will send `metrics` messages.
rtvi_metrics = RTVIMetricsProcessor()
pipeline = Pipeline(
[
transport.input(),
context_aggregator.user(),
llm,
rtvi_speaking,
rtvi_user_transcription,
rtvi_bot_transcription,
ta,
rtvi_metrics,
transport.output(),
context_aggregator.assistant(),
]
)
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
),
)
await task.queue_frame(quiet_frame)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
print(f"Participant left: {participant}")
await task.queue_frame(EndFrame())
runner = PipelineRunner()
await runner.run(task)
if __name__ == "__main__":
asyncio.run(main())

75
examples/simple-chatbot/server/bot.py → examples/simple-chatbot/server/bot-openai.py
View File

@@ -4,6 +4,19 @@
# SPDX-License-Identifier: BSD 2-Clause License
#
"""OpenAI Bot Implementation.
This module implements a chatbot using OpenAI's GPT-4 model for natural language
processing. It includes:
- Real-time audio/video interaction through Daily
- Animated robot avatar
- Text-to-speech using ElevenLabs
- Support for both English and Spanish
The bot runs as part of a pipeline that processes audio/video frames and manages
the conversation flow.
"""
import asyncio
import os
import sys
@@ -31,6 +44,8 @@ from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.processors.frameworks.rtvi import (
RTVIBotTranscriptionProcessor,
RTVIMetricsProcessor,
RTVISpeakingProcessor,
RTVIUserTranscriptionProcessor,
)
from pipecat.services.elevenlabs import ElevenLabsTTSService
@@ -38,14 +53,13 @@ from pipecat.services.openai import OpenAILLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport
load_dotenv(override=True)
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
sprites = []
script_dir = os.path.dirname(__file__)
# Load sequential animation frames
for i in range(1, 26):
# Build the full path to the image file
full_path = os.path.join(script_dir, f"assets/robot0{i}.png")
@@ -54,18 +68,20 @@ for i in range(1, 26):
with Image.open(full_path) as img:
sprites.append(OutputImageRawFrame(image=img.tobytes(), size=img.size, format=img.format))
# Create a smooth animation by adding reversed frames
flipped = sprites[::-1]
sprites.extend(flipped)
# When the bot isn't talking, show a static image of the cat listening
quiet_frame = sprites[0]
talking_frame = SpriteFrame(images=sprites)
# Define static and animated states
quiet_frame = sprites[0] # Static frame for when bot is listening
talking_frame = SpriteFrame(images=sprites) # Animation sequence for when bot is talking
class TalkingAnimation(FrameProcessor):
"""
This class starts a talking animation when it receives an first AudioFrame,
and then returns to a "quiet" sprite when it sees a TTSStoppedFrame.
"""Manages the bot's visual animation states.
Switches between static (listening) and animated (talking) states based on
the bot's current speaking status.
"""
def __init__(self):
@@ -73,12 +89,20 @@ class TalkingAnimation(FrameProcessor):
self._is_talking = False
async def process_frame(self, frame: Frame, direction: FrameDirection):
"""Process incoming frames and update animation state.
Args:
frame: The incoming frame to process
direction: The direction of frame flow in the pipeline
"""
await super().process_frame(frame, direction)
# Switch to talking animation when bot starts speaking
if isinstance(frame, BotStartedSpeakingFrame):
if not self._is_talking:
await self.push_frame(talking_frame)
self._is_talking = True
# Return to static frame when bot stops speaking
elif isinstance(frame, BotStoppedSpeakingFrame):
await self.push_frame(quiet_frame)
self._is_talking = False
@@ -87,9 +111,19 @@ class TalkingAnimation(FrameProcessor):
async def main():
"""Main bot execution function.
Sets up and runs the bot pipeline including:
- Daily video transport
- Speech-to-text and text-to-speech services
- Language model integration
- Animation processing
- RTVI event handling
"""
async with aiohttp.ClientSession() as session:
(room_url, token) = await configure(session)
# Set up Daily transport with video/audio parameters
transport = DailyTransport(
room_url,
token,
@@ -113,6 +147,7 @@ async def main():
),
)
# Initialize text-to-speech service
tts = ElevenLabsTTSService(
api_key=os.getenv("ELEVENLABS_API_KEY"),
#
@@ -126,6 +161,7 @@ async def main():
# voice_id="gD1IexrzCvsXPHUuT0s3",
)
# Initialize LLM service
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
messages = [
@@ -142,12 +178,19 @@ async def main():
},
]
# Set up conversation context and management
# The context_aggregator will automatically collect conversation context
context = OpenAILLMContext(messages)
context_aggregator = llm.create_context_aggregator(context)
ta = TalkingAnimation()
# RTVI
#
# RTVI events for Pipecat client UI
#
# This will send `user-*-speaking` and `bot-*-speaking` messages.
rtvi_speaking = RTVISpeakingProcessor()
# This will emit UserTranscript events.
rtvi_user_transcription = RTVIUserTranscriptionProcessor()
@@ -155,21 +198,33 @@ async def main():
# This will emit BotTranscript events.
rtvi_bot_transcription = RTVIBotTranscriptionProcessor()
# This will send `metrics` messages.
rtvi_metrics = RTVIMetricsProcessor()
pipeline = Pipeline(
[
transport.input(),
rtvi_speaking,
rtvi_user_transcription,
context_aggregator.user(),
llm,
rtvi_bot_transcription,
tts,
ta,
rtvi_metrics,
transport.output(),
context_aggregator.assistant(),
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
),
)
await task.queue_frame(quiet_frame)
@transport.event_handler("on_first_participant_joined")

4
examples/simple-chatbot/server/env.example
View File

@@ -1,4 +1,6 @@
DAILY_SAMPLE_ROOM_URL=https://yourdomain.daily.co/yourroom # (for joining the bot to the same room repeatedly for local dev)
DAILY_API_KEY=7df...
OPENAI_API_KEY=sk-PL...
ELEVENLABS_API_KEY=aeb...
GEMINI_API_KEY=AIza...
ELEVENLABS_API_KEY=aeb...
BOT_IMPLEMENTATION= # Options: 'openai' or 'gemini'

4
examples/simple-chatbot/server/runner.py
View File

@@ -4,14 +4,16 @@
# SPDX-License-Identifier: BSD 2-Clause License
#
import aiohttp
import argparse
import os
import aiohttp
from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper
async def configure(aiohttp_session: aiohttp.ClientSession):
"""Configure the Daily room and Daily REST helper."""
parser = argparse.ArgumentParser(description="Daily AI SDK Bot Sample")
parser.add_argument(
"-u", "--url", type=str, required=False, help="URL of the Daily room to join"

43
examples/simple-chatbot/server/server.py
View File

@@ -4,8 +4,7 @@
# SPDX-License-Identifier: BSD 2-Clause License
#
"""
RTVI Bot Server Implementation
"""RTVI Bot Server Implementation.
This FastAPI server manages RTVI bot instances and provides endpoints for both
direct browser access and RTVI client connections. It handles:
@@ -49,8 +48,8 @@ daily_helpers = {}
def cleanup():
"""
Cleanup function to terminate all bot processes.
"""Cleanup function to terminate all bot processes.
Called during server shutdown.
"""
for entry in bot_procs.values():
@@ -59,10 +58,22 @@ def cleanup():
proc.wait()
def get_bot_file():
bot_implementation = os.getenv("BOT_IMPLEMENTATION", "openai").lower().strip()
# If blank or None, default to openai
if not bot_implementation:
bot_implementation = "openai"
if bot_implementation not in ["openai", "gemini"]:
raise ValueError(
f"Invalid BOT_IMPLEMENTATION: {bot_implementation}. Must be 'openai' or 'gemini'"
)
return f"bot-{bot_implementation}"
@asynccontextmanager
async def lifespan(app: FastAPI):
"""
FastAPI lifespan manager that handles startup and shutdown tasks:
"""FastAPI lifespan manager that handles startup and shutdown tasks.
- Creates aiohttp session
- Initializes Daily API helper
- Cleans up resources on shutdown
@@ -92,8 +103,7 @@ app.add_middleware(
async def create_room_and_token() -> tuple[str, str]:
"""
Helper function to create a Daily room and generate an access token.
"""Helper function to create a Daily room and generate an access token.
Returns:
tuple[str, str]: A tuple containing (room_url, token)
@@ -114,8 +124,8 @@ async def create_room_and_token() -> tuple[str, str]:
@app.get("/")
async def start_agent(request: Request):
"""
Endpoint for direct browser access to the bot.
"""Endpoint for direct browser access to the bot.
Creates a room, starts a bot instance, and redirects to the Daily room URL.
Returns:
@@ -137,8 +147,9 @@ async def start_agent(request: Request):
# Spawn a new bot process
try:
bot_file = get_bot_file()
proc = subprocess.Popen(
[f"python3 -m bot -u {room_url} -t {token}"],
[f"python3 -m {bot_file} -u {room_url} -t {token}"],
shell=True,
bufsize=1,
cwd=os.path.dirname(os.path.abspath(__file__)),
@@ -152,8 +163,8 @@ async def start_agent(request: Request):
@app.post("/connect")
async def rtvi_connect(request: Request) -> Dict[Any, Any]:
"""
RTVI connect endpoint that creates a room and returns connection credentials.
"""RTVI connect endpoint that creates a room and returns connection credentials.
This endpoint is called by RTVI clients to establish a connection.
Returns:
@@ -168,8 +179,9 @@ async def rtvi_connect(request: Request) -> Dict[Any, Any]:
# Start the bot process
try:
bot_file = get_bot_file()
proc = subprocess.Popen(
[f"python3 -m bot -u {room_url} -t {token}"],
[f"python3 -m {bot_file} -u {room_url} -t {token}"],
shell=True,
bufsize=1,
cwd=os.path.dirname(os.path.abspath(__file__)),
@@ -184,8 +196,7 @@ async def rtvi_connect(request: Request) -> Dict[Any, Any]:
@app.get("/status/{pid}")
def get_status(pid: int):
"""
Get the status of a specific bot process.
"""Get the status of a specific bot process.
Args:
pid (int): Process ID of the bot