missing no longer necessary to call super().process_frame(frame, direction)

transport(output): fix non-audio frames sync after audio frames

.gitignore

@@ -28,4 +28,11 @@ share/python-wheels/
 MANIFEST
 .DS_Store
 .env
-fly.toml
+fly.toml
+
+# Example files
+pipecat/examples/twilio-chatbot/templates/streams.xml
+
+# Documentation
+docs/api/_build/
+docs/api/api

.readthedocs.yaml

@@ -0,0 +1,36 @@
+version: 2
+
+build:
+  os: ubuntu-22.04
+  tools:
+    python: '3.12'
+  apt_packages:
+    - portaudio19-dev
+    - python3-dev
+    - libasound2-dev
+  jobs:
+    pre_build:
+      - python -m pip install --upgrade pip
+      - pip install wheel setuptools
+    post_build:
+      - echo "Build completed"
+
+sphinx:
+  configuration: docs/api/conf.py
+  fail_on_warning: false
+
+python:
+  install:
+    - requirements: docs/api/requirements.txt
+    - method: pip
+      path: .
+
+search:
+  ranking:
+    api/*: 5
+    getting-started/*: 4
+    guides/*: 3
+
+submodules:
+  include: all
+  recursive: true

CHANGELOG.md

@@ -9,12 +9,86 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
-- `GroqLLMService` and `GrokLLMService` for Groq and Grok API integration, with
-  OpenAI-compatible interface.
+- Add support for more languages to ElevenLabs (Arabic, Croatian, Filipino,
+  Tamil) and PlayHT (Afrikans, Albanian, Amharic, Arabic, Bengali, Croatian,
+  Galician, Hebrew, Mandarin, Serbian, Tagalog, Urdu, Xhosa).
+
+### Changed
+
+- It's no longer necessary to call `super().start/stop/cancel(frame)` if you
+  subclass and implement `AIService.start/stop/cancel()`. This is all now done
+  internally and will avoid possible issues if you forget to add it.
+
+- It's no longer necessary to call `super().process_frame(frame, direction)` if
+  you subclass and implement `FrameProcessor.process_frame()`. This is all now
+  done internally and will avoid possible issues if you forget to add it.
+
+### Deprecated
+
+- `AWSTTSService` is now deprecated, use `PollyTTSService` instead.
+
+### Fixed
+
+- Fixed a `BaseOutputTransport` issue that was causing non-audio frames being
+  processed before the previous audio frames were played. This will allow, for
+  example, sending a frame `A` after a `TTSSpeakFrame` and the frame `A` will
+  only be pushed downstream after the audio generated from `TTSSpeakFrame` has
+  been spoken.
+
+## [0.0.50] - 2024-12-11
+
+### Added
+
+- Added `GeminiMultimodalLiveLLMService`. This is an integration for Google's
+  Gemini Multimodal Live API, supporting:
+
+  - Real-time audio and video input processing
+  - Streaming text responses with TTS
+  - Audio transcription for both user and bot speech
+  - Function calling
+  - System instructions and context management
+  - Dynamic parameter updates (temperature, top_p, etc.)
+
+- Added `AudioTranscriber` utility class for handling audio transcription with
+  Gemini models.
+
+- Added new context classes for Gemini:
+
+  - `GeminiMultimodalLiveContext`
+  - `GeminiMultimodalLiveUserContextAggregator`
+  - `GeminiMultimodalLiveAssistantContextAggregator`
+  - `GeminiMultimodalLiveContextAggregatorPair`
+
+- Added new foundational examples for `GeminiMultimodalLiveLLMService`:
+
+  - `26-gemini-multimodal-live.py`
+  - `26a-gemini-multimodal-live-transcription.py`
+  - `26b-gemini-multimodal-live-video.py`
+  - `26c-gemini-multimodal-live-video.py`
+
+- Added `SimliVideoService`. This is an integration for Simli AI avatars.
+  (see https://www.simli.com)
+
+- Added NVIDIA Riva's `FastPitchTTSService` and `ParakeetSTTService`.
+  (see https://www.nvidia.com/en-us/ai-data-science/products/riva/)
+
+- Added `IdentityFilter`. This is the simplest frame filter that lets through
+  all incoming frames.
+
+- New `STTMuteStrategy` called `FUNCTION_CALL` which mutes the STT service
+  during LLM function calls.
+
+- `DeepgramSTTService` now exposes two event handlers `on_speech_started` and
+  `on_utterance_end` that could be used to implement interruptions. See new
+  example `examples/foundational/07c-interruptible-deepgram-vad.py`.
+
+- Added `GroqLLMService`, `GrokLLMService`, and `NimLLMService` for Groq, Grok,
+  and NVIDIA NIM API integration, with an OpenAI-compatible interface.
 
 - New examples demonstrating function calling with Groq, Grok, Azure OpenAI,
-  and Fireworks: `14f-function-calling-groq.py`, `14g-function-calling-grok.py`,
-  `14h-function-calling-azure.py`, and `14i-function-calling-fireworks.py`.
+  Fireworks, and NVIDIA NIM: `14f-function-calling-groq.py`,
+  `14g-function-calling-grok.py`, `14h-function-calling-azure.py`,
+  `14i-function-calling-fireworks.py`, and `14j-function-calling-nvidia.py`.
 
 - In order to obtain the audio stored by the `AudioBufferProcessor` you can now
   also register an `on_audio_data` event handler. The `on_audio_data` handler
@@ -33,8 +107,16 @@ async def on_audio_data(processor, audio, sample_rate, num_channels):
 
 ### Changed
 
-- All input frames (text, audio, image, etc.) are now system frames. This means
-  they are processed immediately by all processors instead of being queued
+- `STTMuteFilter` now supports multiple simultaneous muting strategies.
+
+- `XTTSService` language now defaults to `Language.EN`.
+
+- `SoundfileMixer` doesn't resample input files anymore to avoid startup
+  delays. The sample rate of the provided sound files now need to match the
+  sample rate of the output transport.
+
+- Input frames (audio, image and transport messages) are now system frames. This
+  means they are processed immediately by all processors instead of being queued
   internally.
 
 - Expanded the transcriptions.language module to support a superset of
@@ -49,6 +131,9 @@ async def on_audio_data(processor, audio, sample_rate, num_channels):
 - Updated the `FireworksLLMService` to use the `OpenAILLMService`. Updated the
   default model to `accounts/fireworks/models/firefunction-v2`.
 
+- Updated the `simple-chatbot` example to include a Javascript and React client
+  example, using RTVI JS and React.
+
 ### Removed
 
 - Removed `AppFrame`. This was used as a special user custom frame, but there's
@@ -56,6 +141,27 @@ async def on_audio_data(processor, audio, sample_rate, num_channels):
 
 ### Fixed
 
+- Fixed a `ParallelPipeline` issue that would cause system frames to be queued.
+
+- Fixed `FastAPIWebsocketTransport` so it can work with binary data (e.g. using
+  the protobuf serializer).
+
+- Fixed an issue in `CartesiaTTSService` that could cause previous audio to be
+  received after an interruption.
+
+- Fixed Cartesia, ElevenLabs, LMNT and PlayHT TTS websocket
+  reconnection. Before, if an error occurred no reconnection was happening.
+
+- Fixed a `BaseOutputTransport` issue that was causing audio to be discarded
+  after an `EndFrame` was received.
+
+- Fixed an issue in `WebsocketServerTransport` and `FastAPIWebsocketTransport`
+  that would cause a busy loop when using audio mixer.
+
+- Fixed a `DailyTransport` and `LiveKitTransport` issue where connections were
+  being closed in the input transport prematurely. This was causing frames
+  queued inside the pipeline being discarded.
+
 - Fixed an issue in `DailyTransport` that would cause some internal callbacks to
   not be executed.
 

README.md

@@ -55,17 +55,17 @@ pip install "pipecat-ai[option,...]"
 
 Available options include:
 
-| Category            | Services                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | Install Command Example               |
-| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- |
-| Speech-to-Text      | [AssemblyAI](https://docs.pipecat.ai/api-reference/services/stt/assemblyai), [Azure](https://docs.pipecat.ai/api-reference/services/stt/azure), [Deepgram](https://docs.pipecat.ai/api-reference/services/stt/deepgram), [Gladia](https://docs.pipecat.ai/api-reference/services/stt/gladia), [Whisper](https://docs.pipecat.ai/api-reference/services/stt/whisper)                                                                                                                                                                                                                                                                                                                                                                                                               | `pip install "pipecat-ai[deepgram]"`  |
-| LLMs                | [Anthropic](https://docs.pipecat.ai/api-reference/services/llm/anthropic), [Azure](https://docs.pipecat.ai/api-reference/services/llm/azure), [Fireworks AI](https://docs.pipecat.ai/api-reference/services/llm/fireworks), [Gemini](https://docs.pipecat.ai/api-reference/services/llm/gemini), [Grok](https://docs.pipecat.ai/api-reference/services/llm/grok), [Groq](https://docs.pipecat.ai/api-reference/services/llm/groq) [Ollama](https://docs.pipecat.ai/api-reference/services/llm/ollama), [OpenAI](https://docs.pipecat.ai/api-reference/services/llm/openai), [Together AI](https://docs.pipecat.ai/api-reference/services/llm/together)                                                                                                                            | `pip install "pipecat-ai[openai]"`    |
-| Text-to-Speech      | [AWS](https://docs.pipecat.ai/api-reference/services/tts/aws), [Azure](https://docs.pipecat.ai/api-reference/services/tts/azure), [Cartesia](https://docs.pipecat.ai/api-reference/services/tts/cartesia), [Deepgram](https://docs.pipecat.ai/api-reference/services/tts/deepgram), [ElevenLabs](https://docs.pipecat.ai/api-reference/services/tts/elevenlabs), [Google](https://docs.pipecat.ai/api-reference/services/tts/google), [LMNT](https://docs.pipecat.ai/api-reference/services/tts/lmnt), [OpenAI](https://docs.pipecat.ai/api-reference/services/tts/openai), [PlayHT](https://docs.pipecat.ai/api-reference/services/tts/playht), [Rime](https://docs.pipecat.ai/api-reference/services/tts/rime), [XTTS](https://docs.pipecat.ai/api-reference/services/tts/xtts) | `pip install "pipecat-ai[cartesia]"`  |
-| Speech-to-Speech    | [OpenAI Realtime](https://docs.pipecat.ai/api-reference/services/s2s/openai)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | `pip install "pipecat-ai[openai]"`    |
-| Transport           | [Daily (WebRTC)](https://docs.pipecat.ai/api-reference/services/transport/daily), WebSocket, Local                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `pip install "pipecat-ai[daily]"`     |
-| Video               | [Tavus](https://docs.pipecat.ai/api-reference/services/video/tavus)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | `pip install "pipecat-ai[tavus]"`     |
-| Vision & Image      | [Moondream](https://docs.pipecat.ai/api-reference/services/vision/moondream), [fal](https://docs.pipecat.ai/api-reference/services/image-generation/fal)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | `pip install "pipecat-ai[moondream]"` |
-| Audio Processing    | [Silero VAD](https://docs.pipecat.ai/api-reference/utilities/audio/silero-vad-analyzer), [Krisp](https://docs.pipecat.ai/api-reference/utilities/audio/krisp-filter), [Noisereduce](https://docs.pipecat.ai/api-reference/utilities/audio/noisereduce-filter)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | `pip install "pipecat-ai[silero]"`    |
-| Analytics & Metrics | [Canonical AI](https://docs.pipecat.ai/api-reference/services/analytics/canonical), [Sentry](https://docs.pipecat.ai/api-reference/services/analytics/sentry)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | `pip install "pipecat-ai[canonical]"` |
+| Category            | Services                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | Install Command Example                 |
+| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- |
+| Speech-to-Text      | [AssemblyAI](https://docs.pipecat.ai/api-reference/services/stt/assemblyai), [Azure](https://docs.pipecat.ai/api-reference/services/stt/azure), [Deepgram](https://docs.pipecat.ai/api-reference/services/stt/deepgram), [Gladia](https://docs.pipecat.ai/api-reference/services/stt/gladia), [Whisper](https://docs.pipecat.ai/api-reference/services/stt/whisper)                                                                                                                                                                                                                                                                                                                                                                                                               | `pip install "pipecat-ai[deepgram]"`    |
+| LLMs                | [Anthropic](https://docs.pipecat.ai/api-reference/services/llm/anthropic), [Azure](https://docs.pipecat.ai/api-reference/services/llm/azure), [Fireworks AI](https://docs.pipecat.ai/api-reference/services/llm/fireworks), [Gemini](https://docs.pipecat.ai/api-reference/services/llm/gemini), [Grok](https://docs.pipecat.ai/api-reference/services/llm/grok), [Groq](https://docs.pipecat.ai/api-reference/services/llm/groq), [NVIDIA NIM](https://docs.pipecat.ai/api-reference/services/llm/nim), [Ollama](https://docs.pipecat.ai/api-reference/services/llm/ollama), [OpenAI](https://docs.pipecat.ai/api-reference/services/llm/openai), [Together AI](https://docs.pipecat.ai/api-reference/services/llm/together)                                                     | `pip install "pipecat-ai[openai]"`      |
+| Text-to-Speech      | [AWS](https://docs.pipecat.ai/api-reference/services/tts/aws), [Azure](https://docs.pipecat.ai/api-reference/services/tts/azure), [Cartesia](https://docs.pipecat.ai/api-reference/services/tts/cartesia), [Deepgram](https://docs.pipecat.ai/api-reference/services/tts/deepgram), [ElevenLabs](https://docs.pipecat.ai/api-reference/services/tts/elevenlabs), [Google](https://docs.pipecat.ai/api-reference/services/tts/google), [LMNT](https://docs.pipecat.ai/api-reference/services/tts/lmnt), [OpenAI](https://docs.pipecat.ai/api-reference/services/tts/openai), [PlayHT](https://docs.pipecat.ai/api-reference/services/tts/playht), [Rime](https://docs.pipecat.ai/api-reference/services/tts/rime), [XTTS](https://docs.pipecat.ai/api-reference/services/tts/xtts) | `pip install "pipecat-ai[cartesia]"`    |
+| Speech-to-Speech    | [Gemini Multimodal Live](https://docs.pipecat.ai/server/services/s2s/gemini), [OpenAI Realtime](https://docs.pipecat.ai/api-reference/services/s2s/openai)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | `pip install "pipecat-ai[openai]"`      |
+| Transport           | [Daily (WebRTC)](https://docs.pipecat.ai/api-reference/services/transport/daily), WebSocket, Local                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `pip install "pipecat-ai[daily]"`       |
+| Video               | [Tavus](https://docs.pipecat.ai/api-reference/services/video/tavus), [Simli](https://docs.pipecat.ai/api-reference/services/video/simli)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | `pip install "pipecat-ai[tavus,simli]"` |
+| Vision & Image      | [Moondream](https://docs.pipecat.ai/api-reference/services/vision/moondream), [fal](https://docs.pipecat.ai/api-reference/services/image-generation/fal)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | `pip install "pipecat-ai[moondream]"`   |
+| Audio Processing    | [Silero VAD](https://docs.pipecat.ai/api-reference/utilities/audio/silero-vad-analyzer), [Krisp](https://docs.pipecat.ai/api-reference/utilities/audio/krisp-filter), [Noisereduce](https://docs.pipecat.ai/api-reference/utilities/audio/noisereduce-filter)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | `pip install "pipecat-ai[silero]"`      |
+| Analytics & Metrics | [Canonical AI](https://docs.pipecat.ai/api-reference/services/analytics/canonical), [Sentry](https://docs.pipecat.ai/api-reference/services/analytics/sentry)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | `pip install "pipecat-ai[canonical]"`   |
 
 📚 [View full services documentation →](https://docs.pipecat.ai/api-reference/services/supported-services)
 

dev-requirements.txt

@@ -1,5 +1,5 @@
 build~=1.2.1
-grpcio-tools~=1.62.2
+grpcio-tools~=1.65.4
 pip-tools~=7.4.1
 pyright~=1.1.376
 pytest~=8.3.2

docs/api/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/api/README.md

@@ -0,0 +1,109 @@
+# Pipecat Documentation
+
+This directory contains the source files for auto-generating Pipecat's server API reference documentation.
+
+## Setup
+
+1. Install documentation dependencies:
+
+```bash
+pip install -r requirements.txt
+```
+
+2. Make the build scripts executable:
+
+```bash
+chmod +x build-docs.sh rtd-test.py
+```
+
+## Building Documentation
+
+From this directory, you can build the documentation in several ways:
+
+### Local Build
+
+```bash
+# Using the build script (automatically opens docs when done)
+./build-docs.sh
+
+# Or directly with sphinx-build
+sphinx-build -b html . _build/html -W --keep-going
+```
+
+### ReadTheDocs Test Build
+
+To test the documentation build process exactly as it would run on ReadTheDocs:
+
+```bash
+./rtd-test.py
+```
+
+This script:
+
+- Creates a fresh virtual environment
+- Installs all dependencies as specified in requirements files
+- Handles conflicting dependencies (like grpcio versions for Riva and PlayHT)
+- Builds the documentation in an isolated environment
+- Provides detailed logging of the build process
+
+Use this script to verify your documentation will build correctly on ReadTheDocs before pushing changes.
+
+## Viewing Documentation
+
+The built documentation will be available at `_build/html/index.html`. To open:
+
+```bash
+# On MacOS
+open _build/html/index.html
+
+# On Linux
+xdg-open _build/html/index.html
+
+# On Windows
+start _build/html/index.html
+```
+
+## Directory Structure
+
+```
+.
+├── api/            # Auto-generated API documentation
+├── _build/         # Built documentation
+├── _static/        # Static files (images, css, etc.)
+├── conf.py         # Sphinx configuration
+├── index.rst       # Main documentation entry point
+├── requirements-base.txt    # Base documentation dependencies
+├── requirements-riva.txt    # Riva-specific dependencies
+├── requirements-playht.txt  # PlayHT-specific dependencies
+├── build-docs.sh   # Local build script
+└── rtd-test.py     # ReadTheDocs test build script
+```
+
+## Notes
+
+- Documentation is auto-generated from Python docstrings
+- Service modules are automatically detected and included
+- The build process matches our ReadTheDocs configuration
+- Warnings are treated as errors (-W flag) to maintain consistency
+- The --keep-going flag ensures all errors are reported
+- Dependencies are split into multiple requirements files to handle version conflicts
+
+## Troubleshooting
+
+If you encounter missing service modules:
+
+1. Verify the service is installed with its extras: `pip install pipecat-ai[service-name]`
+2. Check the build logs for import errors
+3. Ensure the service module is properly initialized in the package
+4. Run `./rtd-test.py` to test in an isolated environment matching ReadTheDocs
+
+For dependency conflicts:
+
+1. Check the requirements files for version specifications
+2. Use `rtd-test.py` to verify dependency resolution
+3. Consider adding service-specific requirements files if needed
+
+For more information:
+
+- [ReadTheDocs Configuration](.readthedocs.yaml)
+- [Sphinx Documentation](https://www.sphinx-doc.org/)

docs/api/build-docs.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+
+# Clean previous build
+rm -rf _build
+
+# Build docs matching ReadTheDocs configuration
+sphinx-build -b html -d _build/doctrees . _build/html -W --keep-going
+
+# Open docs (MacOS)
+open _build/html/index.html

docs/api/conf.py

@@ -0,0 +1,252 @@
+import logging
+import sys
+from pathlib import Path
+
+# Configure logging
+logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")
+logger = logging.getLogger("sphinx-build")
+
+# Add source directory to path
+docs_dir = Path(__file__).parent
+project_root = docs_dir.parent.parent
+sys.path.insert(0, str(project_root / "src"))
+
+# Project information
+project = "pipecat-ai"
+copyright = "2024, Daily"
+author = "Daily"
+
+# General configuration
+extensions = [
+    "sphinx.ext.autodoc",
+    "sphinx.ext.napoleon",
+    "sphinx.ext.viewcode",
+    "sphinx.ext.intersphinx",
+]
+
+# Napoleon settings
+napoleon_google_docstring = True
+napoleon_numpy_docstring = False
+napoleon_include_init_with_doc = True
+
+# AutoDoc settings
+autodoc_default_options = {
+    "members": True,
+    "member-order": "bysource",
+    "special-members": "__init__",
+    "undoc-members": True,
+    "exclude-members": "__weakref__",
+    "no-index": True,
+    "show-inheritance": True,
+}
+
+# Mock imports for optional dependencies
+autodoc_mock_imports = [
+    "riva",
+    "livekit",
+    "pyht",  # Base PlayHT package
+    "pyht.async_client",  # PlayHT specific imports
+    "pyht.client",
+    "pyht.protos",
+    "pyht.protos.api_pb2",
+    "pipecat_ai_playht",  # PlayHT wrapper
+    "anthropic",
+    "assemblyai",
+    "boto3",
+    "azure",
+    "cartesia",
+    "deepgram",
+    "elevenlabs",
+    "fal",
+    "gladia",
+    "google",
+    "krisp",
+    "langchain",
+    "lmnt",
+    "noisereduce",
+    "openai",
+    "openpipe",
+    "simli",
+    "soundfile",
+    # Existing mocks
+    "pipecat_ai_krisp",
+    "pyaudio",
+    "_tkinter",
+    "tkinter",
+    "daily",
+    "daily_python",
+    "pydantic.BaseModel",
+    "pydantic.Field",
+    "pydantic._internal._model_construction",
+    "pydantic._internal._fields",
+]
+
+# HTML output settings
+html_theme = "sphinx_rtd_theme"
+html_static_path = ["_static"]
+autodoc_typehints = "description"
+html_show_sphinx = False
+
+
+def verify_modules():
+    """Verify that required modules are available."""
+    required_modules = {
+        "services": [
+            "assemblyai",
+            "aws",
+            "cartesia",
+            "deepgram",
+            "google",
+            "lmnt",
+            "riva",
+            "simli",
+        ],
+        "serializers": ["livekit"],
+        "vad": ["silero", "vad_analyzer"],
+        "transports": {
+            "services": ["daily", "livekit"],
+            "local": ["audio", "tk"],
+            "network": ["fastapi_websocket", "websocket_server"],
+        },
+    }
+
+    missing = []
+    for category, modules in required_modules.items():
+        if isinstance(modules, dict):
+            # Handle nested structure
+            for subcategory, submodules in modules.items():
+                for module in submodules:
+                    try:
+                        __import__(f"pipecat.{category}.{subcategory}.{module}")
+                        logger.info(
+                            f"Successfully imported pipecat.{category}.{subcategory}.{module}"
+                        )
+                    except (ImportError, TypeError, NameError) as e:
+                        missing.append(f"pipecat.{category}.{subcategory}.{module}")
+                        logger.warning(
+                            f"Optional module not available: pipecat.{category}.{subcategory}.{module} - {str(e)}"
+                        )
+        else:
+            # Handle flat structure
+            for module in modules:
+                try:
+                    __import__(f"pipecat.{category}.{module}")
+                    logger.info(f"Successfully imported pipecat.{category}.{module}")
+                except (ImportError, TypeError, NameError) as e:
+                    missing.append(f"pipecat.{category}.{module}")
+                    logger.warning(
+                        f"Optional module not available: pipecat.{category}.{module} - {str(e)}"
+                    )
+
+    if missing:
+        logger.warning(f"Some optional modules are not available: {missing}")
+
+
+def clean_title(title: str) -> str:
+    """Automatically clean module titles."""
+    # Remove everything after space (like 'module', 'processor', etc.)
+    title = title.split(" ")[0]
+
+    # Get the last part of the dot-separated path
+    parts = title.split(".")
+    title = parts[-1]
+
+    # Special cases for service names and common acronyms
+    special_cases = {
+        "ai": "AI",
+        "aws": "AWS",
+        "api": "API",
+        "vad": "VAD",
+        "assemblyai": "AssemblyAI",
+        "deepgram": "Deepgram",
+        "elevenlabs": "ElevenLabs",
+        "openai": "OpenAI",
+        "openpipe": "OpenPipe",
+        "playht": "PlayHT",
+        "xtts": "XTTS",
+        "lmnt": "LMNT",
+    }
+
+    # Check if the entire title is a special case
+    if title.lower() in special_cases:
+        return special_cases[title.lower()]
+
+    # Otherwise, capitalize each word
+    words = title.split("_")
+    cleaned_words = []
+    for word in words:
+        if word.lower() in special_cases:
+            cleaned_words.append(special_cases[word.lower()])
+        else:
+            cleaned_words.append(word.capitalize())
+
+    return " ".join(cleaned_words)
+
+
+def setup(app):
+    """Generate API documentation during Sphinx build."""
+    from sphinx.ext.apidoc import main
+
+    docs_dir = Path(__file__).parent
+    project_root = docs_dir.parent.parent
+    output_dir = str(docs_dir / "api")
+    source_dir = str(project_root / "src" / "pipecat")
+
+    # Clean existing files
+    if Path(output_dir).exists():
+        import shutil
+
+        shutil.rmtree(output_dir)
+        logger.info(f"Cleaned existing documentation in {output_dir}")
+
+    logger.info(f"Generating API documentation...")
+    logger.info(f"Output directory: {output_dir}")
+    logger.info(f"Source directory: {source_dir}")
+
+    excludes = [
+        str(project_root / "src/pipecat/pipeline/to_be_updated"),
+        str(project_root / "src/pipecat/processors/gstreamer"),
+        str(project_root / "src/pipecat/services/to_be_updated"),
+        str(project_root / "src/pipecat/vad"),  # deprecated
+        "**/test_*.py",
+        "**/tests/*.py",
+    ]
+
+    try:
+        main(
+            [
+                "-f",  # Force overwriting
+                "-e",  # Don't generate empty files
+                "-M",  # Put module documentation before submodule documentation
+                "--no-toc",  # Don't create a table of contents file
+                "--separate",  # Put documentation for each module in its own page
+                "--module-first",  # Module documentation before submodule documentation
+                "--implicit-namespaces",  # Added: Handle implicit namespace packages
+                "-o",
+                output_dir,
+                source_dir,
+            ]
+            + excludes
+        )
+
+        logger.info("API documentation generated successfully!")
+
+        # Process generated RST files to update titles
+        for rst_file in Path(output_dir).glob("**/*.rst"):  # Changed to recursive glob
+            content = rst_file.read_text()
+            lines = content.split("\n")
+
+            # Find and clean up the title
+            if lines and "=" in lines[1]:  # Title is typically the first line
+                old_title = lines[0]
+                new_title = clean_title(old_title)
+                content = content.replace(old_title, new_title)
+                rst_file.write_text(content)
+                logger.info(f"Updated title: {old_title} -> {new_title}")
+
+    except Exception as e:
+        logger.error(f"Error generating API documentation: {e}", exc_info=True)
+
+
+# Run module verification
+verify_modules()

docs/api/index.rst

@@ -0,0 +1,77 @@
+Pipecat API Reference Docs
+==========================
+
+Welcome to Pipecat's API reference documentation!
+
+Pipecat is an open source framework for building voice and multimodal assistants.
+It provides a flexible pipeline architecture for connecting various AI services,
+audio processing, and transport layers.
+
+Quick Links
+-----------
+
+* `GitHub Repository <https://github.com/pipecat-ai/pipecat>`_
+* `Website <https://pipecat.ai>`_
+
+API Reference
+-------------
+
+Core Components
+~~~~~~~~~~~~~~~
+
+* :mod:`Frames <pipecat.frames>`
+* :mod:`Processors <pipecat.processors>`
+* :mod:`Pipeline <pipecat.pipeline>`
+
+Audio Processing
+~~~~~~~~~~~~~~~~
+
+* :mod:`Audio <pipecat.audio>`
+
+Services
+~~~~~~~~
+
+* :mod:`Services <pipecat.services>`
+
+Transport & Serialization
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* :mod:`Transports <pipecat.transports>`
+   * :mod:`Local <pipecat.transports.local>`
+   * :mod:`Network <pipecat.transports.network>`
+   * :mod:`Services <pipecat.transports.services>`
+* :mod:`Serializers <pipecat.serializers>`
+
+Utilities
+~~~~~~~~~
+
+* :mod:`Clocks <pipecat.clocks>`
+* :mod:`Metrics <pipecat.metrics>`
+* :mod:`Sync <pipecat.sync>`
+* :mod:`Transcriptions <pipecat.transcriptions>`
+* :mod:`Utils <pipecat.utils>`
+
+.. toctree::
+   :maxdepth: 3
+   :caption: API Reference
+   :hidden:
+
+   Audio <api/pipecat.audio>
+   Clocks <api/pipecat.clocks>
+   Frames <api/pipecat.frames>
+   Metrics <api/pipecat.metrics>
+   Pipeline <api/pipecat.pipeline>
+   Processors <api/pipecat.processors>
+   Serializers <api/pipecat.serializers>
+   Services <api/pipecat.services>
+   Sync <api/pipecat.sync>
+   Transcriptions <api/pipecat.transcriptions>
+   Transports <api/pipecat.transports>
+   Utils <api/pipecat.utils>
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`

docs/api/make.bat

@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=.
+set BUILDDIR=_build
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.https://www.sphinx-doc.org/
+	exit /b 1
+)
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd

docs/api/requirements.txt

@@ -0,0 +1,40 @@
+# Sphinx dependencies
+sphinx>=8.1.3
+sphinx-rtd-theme
+sphinx-markdown-builder
+sphinx-autodoc-typehints
+toml
+
+# Install all extras individually to ensure they're properly resolved
+pipecat-ai[anthropic]
+pipecat-ai[assemblyai]
+pipecat-ai[aws]
+pipecat-ai[azure]
+pipecat-ai[canonical]
+pipecat-ai[cartesia]
+pipecat-ai[daily]
+pipecat-ai[deepgram]
+pipecat-ai[elevenlabs]
+pipecat-ai[fal]
+pipecat-ai[fireworks]
+pipecat-ai[gladia]
+pipecat-ai[google]
+pipecat-ai[grok]
+pipecat-ai[groq]
+# pipecat-ai[krisp] # Mocked instead
+pipecat-ai[langchain]
+pipecat-ai[livekit]
+pipecat-ai[lmnt]
+pipecat-ai[local]
+pipecat-ai[moondream]
+pipecat-ai[nim]
+pipecat-ai[noisereduce]
+pipecat-ai[openai]
+# pipecat-ai[openpipe]
+# pipecat-ai[playht] # Mocked due to grpcio conflict with riva
+pipecat-ai[riva]
+pipecat-ai[silero]
+pipecat-ai[simli]
+pipecat-ai[soundfile]
+pipecat-ai[websocket]
+pipecat-ai[whisper]

docs/api/rtd-test.sh

@@ -0,0 +1,38 @@
+#!/bin/bash
+set -e
+
+# Configuration
+DOCS_DIR=$(pwd)
+PROJECT_ROOT=$(cd ../../ && pwd)
+TEST_DIR="/tmp/rtd-test-$(date +%Y%m%d_%H%M%S)"
+
+echo "Creating test directory: $TEST_DIR"
+mkdir -p "$TEST_DIR"
+cd "$TEST_DIR"
+
+# Create virtual environment
+python -m venv venv
+source venv/bin/activate
+
+echo "Installing build dependencies..."
+pip install --upgrade pip wheel setuptools
+
+echo "Installing documentation dependencies..."
+pip install -r "$DOCS_DIR/requirements.txt"
+
+echo "Building documentation..."
+cd "$DOCS_DIR"
+sphinx-build -b html . "_build/html"
+
+echo "Build complete. Check _build/html directory for output."
+
+# Print summary
+echo -e "\n=== Build Summary ==="
+echo "Documentation: $DOCS_DIR/_build/html"
+echo "Test environment: $TEST_DIR"
+echo -e "\nTo view the documentation:"
+echo "open $DOCS_DIR/_build/html/index.html"
+
+# Print installed packages for verification
+echo -e "\n=== Installed Packages ==="
+pip freeze | grep -E "sphinx|pipecat"

dot-env.template

@@ -54,5 +54,9 @@ TAVUS_API_KEY=...
 TAVUS_REPLICA_ID=...
 TAVUS_PERSONA_ID=...
 
-#Krisp
-KRISP_MODEL_PATH=...
+# Simli
+SIMLI_API_KEY=...
+SIMLI_FACE_ID=...
+
+# Krisp
+KRISP_MODEL_PATH=...

examples/deployment/modal-example/requirements.txt

@@ -2,4 +2,4 @@ python-dotenv==1.0.1
 modal==0.65.48
 pipecat-ai[daily,silero,cartesia,openai]==0.0.48
 fastapi==0.115.4
-aiohttp==3.10.10
+aiohttp==3.11.9

examples/foundational/01c-fastpitch.py

@@ -0,0 +1,56 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import asyncio
+import aiohttp
+import os
+import sys
+
+from pipecat.frames.frames import EndFrame, TTSSpeakFrame
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.task import PipelineTask
+from pipecat.pipeline.runner import PipelineRunner
+from pipecat.services.riva import FastPitchTTSService
+from pipecat.transports.services.daily import DailyParams, DailyTransport
+
+from runner import configure
+
+from loguru import logger
+
+from dotenv import load_dotenv
+
+load_dotenv(override=True)
+
+logger.remove(0)
+logger.add(sys.stderr, level="DEBUG")
+
+
+async def main():
+    async with aiohttp.ClientSession() as session:
+        (room_url, _) = await configure(session)
+
+        transport = DailyTransport(
+            room_url, None, "Say One Thing", DailyParams(audio_out_enabled=True)
+        )
+
+        tts = FastPitchTTSService(api_key=os.getenv("NVIDIA_API_KEY"))
+
+        runner = PipelineRunner()
+
+        task = PipelineTask(Pipeline([tts, transport.output()]))
+
+        # Register an event handler so we can play the audio when the
+        # participant joins.
+        @transport.event_handler("on_first_participant_joined")
+        async def on_first_participant_joined(transport, participant):
+            participant_name = participant.get("info", {}).get("userName", "")
+            await task.queue_frames([TTSSpeakFrame(f"Aloha, {participant_name}!"), EndFrame()])
+
+        await runner.run(task)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

examples/foundational/05-sync-speech-and-image.py

@@ -56,8 +56,6 @@ class MonthPrepender(FrameProcessor):
         self.prepend_to_next_text_frame = False
 
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         if isinstance(frame, MonthFrame):
             self.most_recent_month = frame.month
         elif self.prepend_to_next_text_frame and isinstance(frame, TextFrame):

examples/foundational/05a-local-sync-speech-and-image.py

@@ -62,8 +62,6 @@ async def main():
                     self.text = ""
 
                 async def process_frame(self, frame: Frame, direction: FrameDirection):
-                    await super().process_frame(frame, direction)
-
                     if isinstance(frame, TextFrame):
                         self.text = frame.text
                     await self.push_frame(frame, direction)
@@ -75,8 +73,6 @@ async def main():
                     self.frame = None
 
                 async def process_frame(self, frame: Frame, direction: FrameDirection):
-                    await super().process_frame(frame, direction)
-
                     if isinstance(frame, TTSAudioRawFrame):
                         self.audio.extend(frame.audio)
                         self.frame = OutputAudioRawFrame(
@@ -90,8 +86,6 @@ async def main():
                     self.frame = None
 
                 async def process_frame(self, frame: Frame, direction: FrameDirection):
-                    await super().process_frame(frame, direction)
-
                     if isinstance(frame, URLImageRawFrame):
                         self.frame = frame
                     await self.push_frame(frame, direction)

examples/foundational/06a-image-sync.py

@@ -47,8 +47,6 @@ class ImageSyncAggregator(FrameProcessor):
         self._waiting_image_bytes = self._waiting_image.tobytes()
 
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         if not isinstance(frame, SystemFrame) and direction == FrameDirection.DOWNSTREAM:
             await self.push_frame(
                 OutputImageRawFrame(

examples/foundational/07c-interruptible-deepgram-vad.py

@@ -0,0 +1,105 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import asyncio
+import os
+import sys
+
+import aiohttp
+from deepgram import LiveOptions
+from dotenv import load_dotenv
+from loguru import logger
+from runner import configure
+
+from pipecat.frames.frames import (
+    BotInterruptionFrame,
+    LLMMessagesFrame,
+    StopInterruptionFrame,
+    UserStartedSpeakingFrame,
+    UserStoppedSpeakingFrame,
+)
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.runner import PipelineRunner
+from pipecat.pipeline.task import PipelineParams, PipelineTask
+from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
+from pipecat.services.deepgram import DeepgramSTTService, DeepgramTTSService
+from pipecat.services.openai import OpenAILLMService
+from pipecat.transports.services.daily import DailyParams, DailyTransport
+
+load_dotenv(override=True)
+
+logger.remove(0)
+logger.add(sys.stderr, level="DEBUG")
+
+
+async def main():
+    async with aiohttp.ClientSession() as session:
+        (room_url, _) = await configure(session)
+
+        transport = DailyTransport(
+            room_url,
+            None,
+            "Respond bot",
+            DailyParams(
+                audio_in_enabled=True,
+                audio_out_enabled=True,
+            ),
+        )
+
+        stt = DeepgramSTTService(
+            api_key=os.getenv("DEEPGRAM_API_KEY"),
+            live_options=LiveOptions(vad_events=True, utterance_end_ms="1000"),
+        )
+
+        tts = DeepgramTTSService(api_key=os.getenv("DEEPGRAM_API_KEY"), voice="aura-helios-en")
+
+        llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
+
+        messages = [
+            {
+                "role": "system",
+                "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
+            },
+        ]
+
+        context = OpenAILLMContext(messages)
+        context_aggregator = llm.create_context_aggregator(context)
+
+        pipeline = Pipeline(
+            [
+                transport.input(),  # Transport user input
+                stt,  # STT
+                context_aggregator.user(),  # User responses
+                llm,  # LLM
+                tts,  # TTS
+                transport.output(),  # Transport bot output
+                context_aggregator.assistant(),  # Assistant spoken responses
+            ]
+        )
+
+        task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
+
+        @stt.event_handler("on_speech_started")
+        async def on_speech_started(stt, *args, **kwargs):
+            await task.queue_frames([BotInterruptionFrame(), UserStartedSpeakingFrame()])
+
+        @stt.event_handler("on_utterance_end")
+        async def on_utterance_end(stt, *args, **kwargs):
+            await task.queue_frames([StopInterruptionFrame(), UserStoppedSpeakingFrame()])
+
+        @transport.event_handler("on_first_participant_joined")
+        async def on_first_participant_joined(transport, participant):
+            # Kick off the conversation.
+            messages.append({"role": "system", "content": "Please introduce yourself to the user."})
+            await task.queue_frames([LLMMessagesFrame(messages)])
+
+        runner = PipelineRunner()
+
+        await runner.run(task)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

examples/foundational/07i-interruptible-xtts.py

@@ -50,7 +50,6 @@ async def main():
         tts = XTTSService(
             aiohttp_session=session,
             voice_id="Claribel Dervla",
-            language="en",
             base_url="http://localhost:8000",
         )
 

examples/foundational/07m-interruptible-polly.py

@@ -19,7 +19,7 @@ from pipecat.pipeline.pipeline import Pipeline
 from pipecat.pipeline.runner import PipelineRunner
 from pipecat.pipeline.task import PipelineParams, PipelineTask
 from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
-from pipecat.services.aws import AWSTTSService
+from pipecat.services.aws import PollyTTSService
 from pipecat.services.deepgram import DeepgramSTTService
 from pipecat.services.openai import OpenAILLMService
 from pipecat.transports.services.daily import DailyParams, DailyTransport
@@ -48,12 +48,12 @@ async def main():
 
         stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
 
-        tts = AWSTTSService(
+        tts = PollyTTSService(
             api_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
             aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
             region=os.getenv("AWS_REGION"),
             voice_id="Amy",
-            params=AWSTTSService.InputParams(engine="neural", language="en-GB", rate="1.05"),
+            params=PollyTTSService.InputParams(engine="neural", language="en-GB", rate="1.05"),
         )
 
         llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")

examples/foundational/07r-interruptible-riva-nim.py

@@ -0,0 +1,92 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import asyncio
+import os
+import sys
+
+import aiohttp
+from dotenv import load_dotenv
+from loguru import logger
+from runner import configure
+
+from pipecat.audio.vad.silero import SileroVADAnalyzer
+from pipecat.frames.frames import LLMMessagesFrame
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.runner import PipelineRunner
+from pipecat.pipeline.task import PipelineParams, PipelineTask
+from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
+from pipecat.services.nim import NimLLMService
+from pipecat.services.riva import FastPitchTTSService, ParakeetSTTService
+from pipecat.transports.services.daily import DailyParams, DailyTransport
+
+load_dotenv(override=True)
+
+logger.remove(0)
+logger.add(sys.stderr, level="DEBUG")
+
+
+async def main():
+    async with aiohttp.ClientSession() as session:
+        (room_url, _) = await configure(session)
+
+        transport = DailyTransport(
+            room_url,
+            None,
+            "Respond bot",
+            DailyParams(
+                audio_out_enabled=True,
+                vad_enabled=True,
+                vad_analyzer=SileroVADAnalyzer(),
+                vad_audio_passthrough=True,
+            ),
+        )
+
+        stt = ParakeetSTTService(api_key=os.getenv("NVIDIA_API_KEY"))
+
+        llm = NimLLMService(
+            api_key=os.getenv("NVIDIA_API_KEY"), model="meta/llama-3.1-405b-instruct"
+        )
+
+        tts = FastPitchTTSService(api_key=os.getenv("NVIDIA_API_KEY"))
+
+        messages = [
+            {
+                "role": "system",
+                "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
+            },
+        ]
+
+        context = OpenAILLMContext(messages)
+        context_aggregator = llm.create_context_aggregator(context)
+
+        pipeline = Pipeline(
+            [
+                transport.input(),  # Transport user input
+                stt,  # STT
+                context_aggregator.user(),  # User responses
+                llm,  # LLM
+                tts,  # TTS
+                transport.output(),  # Transport bot output
+                context_aggregator.assistant(),  # Assistant spoken responses
+            ]
+        )
+
+        task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
+
+        @transport.event_handler("on_first_participant_joined")
+        async def on_first_participant_joined(transport, participant):
+            # Kick off the conversation.
+            messages.append({"role": "system", "content": "Please introduce yourself to the user."})
+            await task.queue_frames([LLMMessagesFrame(messages)])
+
+        runner = PipelineRunner()
+
+        await runner.run(task)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

examples/foundational/07s-interruptible-google-audio-in.py

@@ -82,8 +82,6 @@ class UserAudioCollector(FrameProcessor):
         self._user_speaking = False
 
     async def process_frame(self, frame, direction):
-        await super().process_frame(frame, direction)
-
         if isinstance(frame, TranscriptionFrame):
             # We could gracefully handle both audio input and text/transcription input ...
             # but let's leave that as an exercise to the reader. :-)
@@ -126,7 +124,6 @@ class TranscriptExtractor(FrameProcessor):
         self._accumulating_transcript = False
 
     async def process_frame(self, frame, direction):
-        await super().process_frame(frame, direction)
         if isinstance(frame, LLMFullResponseStartFrame):
             self._processing_llm_response = True
             self._accumulating_transcript = True
@@ -180,8 +177,6 @@ class TanscriptionContextFixup(FrameProcessor):
             self._context.messages[-1].parts[-1].text += f"\n\n{marker}\n{self._transcript}\n"
 
     async def process_frame(self, frame, direction):
-        await super().process_frame(frame, direction)
-
         if isinstance(frame, MagicDemoTranscriptionFrame):
             self._transcript = frame.text
         elif isinstance(frame, LLMFullResponseEndFrame) or isinstance(

examples/foundational/09-mirror.py

@@ -35,8 +35,6 @@ logger.add(sys.stderr, level="DEBUG")
 
 class MirrorProcessor(FrameProcessor):
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         if isinstance(frame, InputAudioRawFrame):
             await self.push_frame(
                 OutputAudioRawFrame(

examples/foundational/09a-local-mirror.py

@@ -39,8 +39,6 @@ logger.add(sys.stderr, level="DEBUG")
 
 class MirrorProcessor(FrameProcessor):
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         if isinstance(frame, InputAudioRawFrame):
             await self.push_frame(
                 OutputAudioRawFrame(

examples/foundational/11-sound-effects.py

@@ -14,16 +14,18 @@ from pipecat.audio.vad.silero import SileroVADAnalyzer
 from pipecat.frames.frames import (
     Frame,
     LLMFullResponseEndFrame,
-    LLMMessagesFrame,
     OutputAudioRawFrame,
 )
 from pipecat.pipeline.pipeline import Pipeline
 from pipecat.pipeline.runner import PipelineRunner
 from pipecat.pipeline.task import PipelineTask
-from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
+from pipecat.processors.aggregators.openai_llm_context import (
+    OpenAILLMContext,
+    OpenAILLMContextFrame,
+)
 from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
 from pipecat.processors.logger import FrameLogger
-from pipecat.services.cartesia import CartesiaHttpTTSService
+from pipecat.services.cartesia import CartesiaTTSService
 from pipecat.services.openai import OpenAILLMService
 from pipecat.transports.services.daily import DailyParams, DailyTransport
 
@@ -58,8 +60,6 @@ for file in sound_files:
 
 class OutboundSoundEffectWrapper(FrameProcessor):
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         if isinstance(frame, LLMFullResponseEndFrame):
             await self.push_frame(sounds["ding1.wav"])
             # In case anything else downstream needs it
@@ -70,9 +70,7 @@ class OutboundSoundEffectWrapper(FrameProcessor):
 
 class InboundSoundEffectWrapper(FrameProcessor):
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
-        if isinstance(frame, LLMMessagesFrame):
+        if isinstance(frame, OpenAILLMContextFrame):
             await self.push_frame(sounds["ding2.wav"])
             # In case anything else downstream needs it
             await self.push_frame(frame, direction)
@@ -98,7 +96,7 @@ async def main():
 
         llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
 
-        tts = CartesiaHttpTTSService(
+        tts = CartesiaTTSService(
             api_key=os.getenv("CARTESIA_API_KEY"),
             voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22",  # British Lady
         )

examples/foundational/12-describe-video.py

@@ -42,8 +42,6 @@ class UserImageRequester(FrameProcessor):
         self._participant_id = participant_id
 
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         if self._participant_id and isinstance(frame, TextFrame):
             await self.push_frame(
                 UserImageRequestFrame(self._participant_id), FrameDirection.UPSTREAM

examples/foundational/12a-describe-video-gemini-flash.py

@@ -42,8 +42,6 @@ class UserImageRequester(FrameProcessor):
         self._participant_id = participant_id
 
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         if self._participant_id and isinstance(frame, TextFrame):
             await self.push_frame(
                 UserImageRequestFrame(self._participant_id), FrameDirection.UPSTREAM

examples/foundational/12b-describe-video-gpt-4o.py

@@ -42,8 +42,6 @@ class UserImageRequester(FrameProcessor):
         self._participant_id = participant_id
 
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         if self._participant_id and isinstance(frame, TextFrame):
             await self.push_frame(
                 UserImageRequestFrame(self._participant_id), FrameDirection.UPSTREAM

examples/foundational/12c-describe-video-anthropic.py

@@ -42,8 +42,6 @@ class UserImageRequester(FrameProcessor):
         self._participant_id = participant_id
 
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         if self._participant_id and isinstance(frame, TextFrame):
             await self.push_frame(
                 UserImageRequestFrame(self._participant_id), FrameDirection.UPSTREAM

examples/foundational/13-whisper-transcription.py

@@ -30,8 +30,6 @@ logger.add(sys.stderr, level="DEBUG")
 
 class TranscriptionLogger(FrameProcessor):
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         if isinstance(frame, TranscriptionFrame):
             print(f"Transcription: {frame.text}")
 

examples/foundational/13a-whisper-local.py

@@ -28,8 +28,6 @@ logger.add(sys.stderr, level="DEBUG")
 
 class TranscriptionLogger(FrameProcessor):
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         if isinstance(frame, TranscriptionFrame):
             print(f"Transcription: {frame.text}")
 

examples/foundational/13b-deepgram-transcription.py

@@ -31,8 +31,6 @@ logger.add(sys.stderr, level="DEBUG")
 
 class TranscriptionLogger(FrameProcessor):
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         if isinstance(frame, TranscriptionFrame):
             print(f"Transcription: {frame.text}")
 

examples/foundational/13c-gladia-transcription.py

@@ -29,8 +29,6 @@ logger.add(sys.stderr, level="DEBUG")
 
 class TranscriptionLogger(FrameProcessor):
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         if isinstance(frame, TranscriptionFrame):
             print(f"Transcription: {frame.text}")
 

examples/foundational/13d-assemblyai-transcription.py

@@ -29,8 +29,6 @@ logger.add(sys.stderr, level="DEBUG")
 
 class TranscriptionLogger(FrameProcessor):
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         if isinstance(frame, TranscriptionFrame):
             print(f"Transcription: {frame.text}")
 

examples/foundational/14j-function-calling-nim.py

@@ -0,0 +1,140 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import asyncio
+import os
+import sys
+
+import aiohttp
+from dotenv import load_dotenv
+from loguru import logger
+from openai.types.chat import ChatCompletionToolParam
+from runner import configure
+
+from pipecat.audio.vad.silero import SileroVADAnalyzer
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.runner import PipelineRunner
+from pipecat.pipeline.task import PipelineParams, PipelineTask
+from pipecat.services.cartesia import CartesiaTTSService
+from pipecat.services.nim import NimLLMService
+from pipecat.services.openai import OpenAILLMContext
+from pipecat.transports.services.daily import DailyParams, DailyTransport
+
+load_dotenv(override=True)
+
+logger.remove(0)
+logger.add(sys.stderr, level="DEBUG")
+
+
+async def start_fetch_weather(function_name, llm, context):
+    # note: we can't push a frame to the LLM here. the bot
+    # can interrupt itself and/or cause audio overlapping glitches.
+    # possible question for Aleix and Chad about what the right way
+    # to trigger speech is, now, with the new queues/async/sync refactors.
+    # await llm.push_frame(TextFrame("Let me check on that."))
+    logger.debug(f"Starting fetch_weather_from_api with function_name: {function_name}")
+
+
+async def fetch_weather_from_api(function_name, tool_call_id, args, llm, context, result_callback):
+    await result_callback({"conditions": "nice", "temperature": "75"})
+
+
+async def main():
+    async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
+        transport = DailyTransport(
+            room_url,
+            token,
+            "Respond bot",
+            DailyParams(
+                audio_out_enabled=True,
+                transcription_enabled=True,
+                vad_enabled=True,
+                vad_analyzer=SileroVADAnalyzer(),
+            ),
+        )
+
+        tts = CartesiaTTSService(
+            api_key=os.getenv("CARTESIA_API_KEY"),
+            voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22",  # British Lady
+            # text_filter=MarkdownTextFilter(),
+        )
+
+        llm = NimLLMService(
+            api_key=os.getenv("NVIDIA_API_KEY"), model="meta/llama-3.1-405b-instruct"
+        )
+        # Register a function_name of None to get all functions
+        # sent to the same callback with an additional function_name parameter.
+        llm.register_function(None, fetch_weather_from_api, start_callback=start_fetch_weather)
+
+        tools = [
+            ChatCompletionToolParam(
+                type="function",
+                function={
+                    "name": "get_current_weather",
+                    "description": "Get the current weather",
+                    "parameters": {
+                        "type": "object",
+                        "properties": {
+                            "location": {
+                                "type": "string",
+                                "description": "The city and state, e.g. San Francisco, CA",
+                            },
+                            "format": {
+                                "type": "string",
+                                "enum": ["celsius", "fahrenheit"],
+                                "description": "The temperature unit to use. Infer this from the users location.",
+                            },
+                        },
+                        "required": ["location", "format"],
+                    },
+                },
+            )
+        ]
+        messages = [
+            {
+                "role": "system",
+                "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
+            },
+        ]
+
+        context = OpenAILLMContext(messages, tools)
+        context_aggregator = llm.create_context_aggregator(context)
+
+        pipeline = Pipeline(
+            [
+                transport.input(),
+                context_aggregator.user(),
+                llm,
+                tts,
+                transport.output(),
+                context_aggregator.assistant(),
+            ]
+        )
+
+        task = PipelineTask(
+            pipeline,
+            PipelineParams(
+                allow_interruptions=True,
+                enable_metrics=True,
+                enable_usage_metrics=True,
+            ),
+        )
+
+        @transport.event_handler("on_first_participant_joined")
+        async def on_first_participant_joined(transport, participant):
+            await transport.capture_participant_transcription(participant["id"])
+            # Kick off the conversation.
+            await task.queue_frames([context_aggregator.user().get_context_frame()])
+
+        runner = PipelineRunner()
+
+        await runner.run(task)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

examples/foundational/15a-switch-languages.py

@@ -9,8 +9,10 @@ import aiohttp
 import os
 import sys
 
+from deepgram import LiveOptions
+
 from pipecat.audio.vad.silero import SileroVADAnalyzer
-from pipecat.frames.frames import LLMMessagesFrame, TTSUpdateSettingsFrame
+from pipecat.frames.frames import LLMMessagesFrame
 from pipecat.pipeline.pipeline import Pipeline
 from pipecat.pipeline.parallel_pipeline import ParallelPipeline
 from pipecat.pipeline.runner import PipelineRunner
@@ -18,6 +20,7 @@ from pipecat.pipeline.task import PipelineParams, PipelineTask
 from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
 from pipecat.processors.filters.function_filter import FunctionFilter
 from pipecat.services.cartesia import CartesiaTTSService
+from pipecat.services.deepgram import DeepgramSTTService
 from pipecat.services.openai import OpenAILLMService
 from pipecat.transports.services.daily import DailyParams, DailyTransport
 
@@ -61,13 +64,16 @@ async def main():
             "Pipecat",
             DailyParams(
                 audio_out_enabled=True,
-                transcription_enabled=True,
                 vad_enabled=True,
                 vad_analyzer=SileroVADAnalyzer(),
                 vad_audio_passthrough=True,
             ),
         )
 
+        stt = DeepgramSTTService(
+            api_key=os.getenv("DEEPGRAM_API_KEY"), live_options=LiveOptions(language="multi")
+        )
+
         english_tts = CartesiaTTSService(
             api_key=os.getenv("CARTESIA_API_KEY"),
             voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22",  # British Lady
@@ -113,6 +119,7 @@ async def main():
         pipeline = Pipeline(
             [
                 transport.input(),  # Transport user input
+                stt,  # STT
                 context_aggregator.user(),  # User responses
                 llm,  # LLM
                 ParallelPipeline(  # TTS (bot will speak the chosen language)

examples/foundational/18-gstreamer-filesrc.py

@@ -53,7 +53,7 @@ async def main():
             out_params=GStreamerPipelineSource.OutputParams(
                 video_width=1280,
                 video_height=720,
-                audio_sample_rate=16000,
+                audio_sample_rate=24000,
                 audio_channels=1,
             ),
         )

examples/foundational/22b-natural-conversation-proposal.py

@@ -64,7 +64,6 @@ class StatementJudgeContextFilter(FrameProcessor):
         self._notifier = notifier
 
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
         # We must not block system frames.
         if isinstance(frame, SystemFrame):
             await self.push_frame(frame, direction)
@@ -118,7 +117,6 @@ class CompletenessCheck(FrameProcessor):
         self._notifier = notifier
 
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
         if isinstance(frame, TextFrame) and frame.text == "YES":
             logger.debug("Completeness check YES")
             await self.push_frame(UserStoppedSpeakingFrame())
@@ -141,8 +139,6 @@ class OutputGate(FrameProcessor):
         self._gate_open = True
 
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         # We must not block system frames.
         if isinstance(frame, SystemFrame):
             if isinstance(frame, StartFrame):

examples/foundational/22c-natural-conversation-mixed-llms.py

@@ -101,12 +101,12 @@ HIGH PRIORITY SIGNALS:
 
 Examples:
 # Complete Wh-question
-[{"role": "assistant", "content": "I can help you learn."}, 
+[{"role": "assistant", "content": "I can help you learn."},
  {"role": "user", "content": "What's the fastest way to learn Spanish"}]
 Output: YES
 
 # Complete Yes/No question despite STT error
-[{"role": "assistant", "content": "I know about planets."}, 
+[{"role": "assistant", "content": "I know about planets."},
  {"role": "user", "content": "Is is Jupiter the biggest planet"}]
 Output: YES
 
@@ -118,12 +118,12 @@ Output: YES
 
 Examples:
 # Direct instruction
-[{"role": "assistant", "content": "I can explain many topics."}, 
+[{"role": "assistant", "content": "I can explain many topics."},
  {"role": "user", "content": "Tell me about black holes"}]
 Output: YES
 
 # Action demand
-[{"role": "assistant", "content": "I can help with math."}, 
+[{"role": "assistant", "content": "I can help with math."},
  {"role": "user", "content": "Solve this equation x plus 5 equals 12"}]
 Output: YES
 
@@ -134,12 +134,12 @@ Output: YES
 
 Examples:
 # Specific answer
-[{"role": "assistant", "content": "What's your favorite color?"}, 
+[{"role": "assistant", "content": "What's your favorite color?"},
  {"role": "user", "content": "I really like blue"}]
 Output: YES
 
 # Option selection
-[{"role": "assistant", "content": "Would you prefer morning or evening?"}, 
+[{"role": "assistant", "content": "Would you prefer morning or evening?"},
  {"role": "user", "content": "Morning"}]
 Output: YES
 
@@ -153,17 +153,17 @@ MEDIUM PRIORITY SIGNALS:
 
 Examples:
 # Self-correction reaching completion
-[{"role": "assistant", "content": "What would you like to know?"}, 
+[{"role": "assistant", "content": "What would you like to know?"},
  {"role": "user", "content": "Tell me about... no wait, explain how rainbows form"}]
 Output: YES
 
 # Topic change with complete thought
-[{"role": "assistant", "content": "The weather is nice today."}, 
+[{"role": "assistant", "content": "The weather is nice today."},
  {"role": "user", "content": "Actually can you tell me who invented the telephone"}]
 Output: YES
 
 # Mid-sentence completion
-[{"role": "assistant", "content": "Hello I'm ready."}, 
+[{"role": "assistant", "content": "Hello I'm ready."},
  {"role": "user", "content": "What's the capital of? France"}]
 Output: YES
 
@@ -175,12 +175,12 @@ Output: YES
 
 Examples:
 # Acknowledgment
-[{"role": "assistant", "content": "Should we talk about history?"}, 
+[{"role": "assistant", "content": "Should we talk about history?"},
  {"role": "user", "content": "Sure"}]
 Output: YES
 
 # Disagreement with completion
-[{"role": "assistant", "content": "Is that what you meant?"}, 
+[{"role": "assistant", "content": "Is that what you meant?"},
  {"role": "user", "content": "No not really"}]
 Output: YES
 
@@ -194,12 +194,12 @@ LOW PRIORITY SIGNALS:
 
 Examples:
 # Word repetition but complete
-[{"role": "assistant", "content": "I can help with that."}, 
+[{"role": "assistant", "content": "I can help with that."},
  {"role": "user", "content": "What what is the time right now"}]
 Output: YES
 
 # Missing punctuation but complete
-[{"role": "assistant", "content": "I can explain that."}, 
+[{"role": "assistant", "content": "I can explain that."},
  {"role": "user", "content": "Please tell me how computers work"}]
 Output: YES
 
@@ -211,12 +211,12 @@ Output: YES
 
 Examples:
 # Filler words but complete
-[{"role": "assistant", "content": "What would you like to know?"}, 
+[{"role": "assistant", "content": "What would you like to know?"},
  {"role": "user", "content": "Um uh how do airplanes fly"}]
 Output: YES
 
 # Thinking pause but incomplete
-[{"role": "assistant", "content": "I can explain anything."}, 
+[{"role": "assistant", "content": "I can explain anything."},
  {"role": "user", "content": "Well um I want to know about the"}]
 Output: NO
 
@@ -241,17 +241,17 @@ DECISION RULES:
 
 Examples:
 # Incomplete despite corrections
-[{"role": "assistant", "content": "What would you like to know about?"}, 
+[{"role": "assistant", "content": "What would you like to know about?"},
  {"role": "user", "content": "Can you tell me about"}]
 Output: NO
 
 # Complete despite multiple artifacts
-[{"role": "assistant", "content": "I can help you learn."}, 
+[{"role": "assistant", "content": "I can help you learn."},
  {"role": "user", "content": "How do you I mean what's the best way to learn programming"}]
 Output: YES
 
 # Trailing off incomplete
-[{"role": "assistant", "content": "I can explain anything."}, 
+[{"role": "assistant", "content": "I can explain anything."},
  {"role": "user", "content": "I was wondering if you could tell me why"}]
 Output: NO
 """
@@ -268,7 +268,6 @@ class StatementJudgeContextFilter(FrameProcessor):
         self._notifier = notifier
 
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
         # We must not block system frames.
         if isinstance(frame, SystemFrame):
             await self.push_frame(frame, direction)
@@ -320,8 +319,6 @@ class CompletenessCheck(FrameProcessor):
         self._notifier = notifier
 
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         if isinstance(frame, TextFrame) and frame.text == "YES":
             logger.debug("!!! Completeness check YES")
             await self.push_frame(UserStoppedSpeakingFrame())
@@ -344,8 +341,6 @@ class OutputGate(FrameProcessor):
         self._gate_open = True
 
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         # We must not block system frames.
         if isinstance(frame, SystemFrame):
             if isinstance(frame, StartFrame):

examples/foundational/22d-natural-conversation-gemini-audio.py

@@ -90,8 +90,6 @@ class StatementJudgeAudioContextAccumulator(FrameProcessor):
         self._user_speaking = False
 
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         # ignore context frame
         if isinstance(frame, OpenAILLMContextFrame):
             return
@@ -133,8 +131,6 @@ class CompletenessCheck(FrameProcessor):
         self._audio_accumulator = audio_accumulator
 
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         if isinstance(frame, TextFrame) and frame.text.startswith("YES"):
             logger.debug("Completeness check YES")
             await self.push_frame(UserStoppedSpeakingFrame())
@@ -159,8 +155,6 @@ class OutputGate(FrameProcessor):
         self._gate_open = True
 
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         # We must not block system frames.
         if isinstance(frame, SystemFrame):
             if isinstance(frame, StartFrame):

examples/foundational/24-stt-mute-filter.py

@@ -11,12 +11,11 @@ import sys
 import aiohttp
 from dotenv import load_dotenv
 from loguru import logger
+from openai.types.chat import ChatCompletionToolParam
 from runner import configure
 
 from pipecat.audio.vad.silero import SileroVADAnalyzer
-from pipecat.frames.frames import (
-    LLMMessagesFrame,
-)
+from pipecat.frames.frames import LLMMessagesFrame
 from pipecat.pipeline.pipeline import Pipeline
 from pipecat.pipeline.runner import PipelineRunner
 from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -32,6 +31,18 @@ logger.remove(0)
 logger.add(sys.stderr, level="DEBUG")
 
 
+async def start_fetch_weather(function_name, llm, context):
+    logger.debug(f"Starting fetch_weather_from_api with function_name: {function_name}")
+
+
+async def fetch_weather_from_api(function_name, tool_call_id, args, llm, context, result_callback):
+    # Add a delay to test interruption during function calls
+    logger.info("Weather API call starting...")
+    await asyncio.sleep(5)  # 5-second delay
+    logger.info("Weather API call completed")
+    await result_callback({"conditions": "nice", "temperature": "75"})
+
+
 async def main():
     async with aiohttp.ClientSession() as session:
         (room_url, _) = await configure(session)
@@ -49,23 +60,52 @@ async def main():
         )
 
         stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
-        # Configure the mute processor to mute only during first speech
+        # Configure the mute processor with both strategies
         stt_mute_processor = STTMuteFilter(
-            stt_service=stt, config=STTMuteConfig(strategy=STTMuteStrategy.FIRST_SPEECH)
+            stt_service=stt,
+            config=STTMuteConfig(
+                strategies={STTMuteStrategy.FIRST_SPEECH, STTMuteStrategy.FUNCTION_CALL}
+            ),
         )
 
         tts = DeepgramTTSService(api_key=os.getenv("DEEPGRAM_API_KEY"), voice="aura-helios-en")
 
         llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
+        llm.register_function(None, fetch_weather_from_api, start_callback=start_fetch_weather)
+
+        tools = [
+            ChatCompletionToolParam(
+                type="function",
+                function={
+                    "name": "get_current_weather",
+                    "description": "Get the current weather",
+                    "parameters": {
+                        "type": "object",
+                        "properties": {
+                            "location": {
+                                "type": "string",
+                                "description": "The city and state, e.g. San Francisco, CA",
+                            },
+                            "format": {
+                                "type": "string",
+                                "enum": ["celsius", "fahrenheit"],
+                                "description": "The temperature unit to use. Infer this from the users location.",
+                            },
+                        },
+                        "required": ["location", "format"],
+                    },
+                },
+            )
+        ]
 
         messages = [
             {
                 "role": "system",
-                "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
+                "content": "You are a helpful assistant who can check the weather. Always check the weather when a location is mentioned. Respond concisely and naturally. Your output will be converted to audio so use only simple words and punctuation.",
             },
         ]
 
-        context = OpenAILLMContext(messages)
+        context = OpenAILLMContext(messages, tools)
         context_aggregator = llm.create_context_aggregator(context)
 
         pipeline = Pipeline(
@@ -85,8 +125,13 @@ async def main():
 
         @transport.event_handler("on_first_participant_joined")
         async def on_first_participant_joined(transport, participant):
-            # Kick off the conversation.
-            messages.append({"role": "system", "content": "Please introduce yourself to the user."})
+            # Kick off the conversation with a weather-related prompt
+            messages.append(
+                {
+                    "role": "system",
+                    "content": "Ask the user what city they'd like to know the weather for.",
+                }
+            )
             await task.queue_frames([LLMMessagesFrame(messages)])
 
         runner = PipelineRunner()

examples/foundational/25-google-audio-in.py

@@ -0,0 +1,366 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import aiohttp
+import asyncio
+import os
+import sys
+
+import google.ai.generativelanguage as glm
+
+from dataclasses import dataclass
+from dotenv import load_dotenv
+from loguru import logger
+from runner import configure
+
+from pipecat.audio.vad.silero import SileroVADAnalyzer
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.parallel_pipeline import ParallelPipeline
+from pipecat.pipeline.runner import PipelineRunner
+from pipecat.pipeline.task import PipelineParams, PipelineTask
+from pipecat.processors.aggregators.openai_llm_context import (
+    OpenAILLMContext,
+    OpenAILLMContextFrame,
+)
+from pipecat.services.cartesia import CartesiaTTSService
+from pipecat.services.google import GoogleLLMService, GoogleLLMContext
+from pipecat.processors.frame_processor import FrameProcessor
+from pipecat.transports.services.daily import DailyParams, DailyTransport
+from pipecat.frames.frames import (
+    Frame,
+    InputAudioRawFrame,
+    LLMFullResponseEndFrame,
+    MetricsFrame,
+    SystemFrame,
+    TextFrame,
+    TranscriptionFrame,
+    UserStartedSpeakingFrame,
+    UserStoppedSpeakingFrame,
+)
+
+load_dotenv(override=True)
+
+logger.remove(0)
+logger.add(sys.stderr, level="DEBUG")
+
+#
+# The system prompt for the main conversation.
+#
+conversation_system_message = """
+You are a helpful LLM in a WebRTC call. Your goals are to be helpful and brief in your responses. Respond with one or two sentences at most, unless you are asked to
+respond at more length. Your output will be converted to audio so don't include special characters in your answers.
+"""
+
+#
+# The system prompt for the LLM doing the audio transcription.
+#
+# Note that we could provide additional instructions per-conversation, here, if that's helpful
+# for our use case. For example, names of people so that the transcription gets the spelling
+# right.
+#
+# A possible future improvement would be to use structured output so that we can include a
+# language tag and perhaps other analytic information.
+#
+transcriber_system_message = """
+You are an audio transcriber. You are receiving audio from a user. Your job is to
+transcribe the input audio to text exactly as it was said by the user..
+
+You will receive the full conversation history before the audio input, to help with context. Use the full history only to help improve the accuracy of your transcription.
+
+Rules:
+  - Respond with an exact transcription of the audio input.
+  - Do not include any text other than the transcription.
+  - Do not explain or add to your response.
+  - Transcribe the audio input simply and precisely.
+  - If the audio is not clear, emit the special string "EMPTY".
+  - No response other than exact transcription, or "EMPTY", is allowed.
+"""
+
+
+class UserAudioCollector(FrameProcessor):
+    """
+    This FrameProcessor collects audio frames in a buffer, then adds them to the
+    LLM context when the user stops speaking.
+    """
+
+    def __init__(self, context, user_context_aggregator):
+        super().__init__()
+        self._context = context
+        self._user_context_aggregator = user_context_aggregator
+        self._audio_frames = []
+        self._start_secs = 0.2  # this should match VAD start_secs (hardcoding for now)
+        self._user_speaking = False
+
+    async def process_frame(self, frame, direction):
+        if isinstance(frame, TranscriptionFrame):
+            # We could gracefully handle both audio input and text/transcription input ...
+            # but let's leave that as an exercise to the reader. :-)
+            return
+        if isinstance(frame, UserStartedSpeakingFrame):
+            self._user_speaking = True
+        elif isinstance(frame, UserStoppedSpeakingFrame):
+            self._user_speaking = False
+            self._context.add_audio_frames_message(audio_frames=self._audio_frames)
+            await self._user_context_aggregator.push_frame(
+                self._user_context_aggregator.get_context_frame()
+            )
+        elif isinstance(frame, InputAudioRawFrame):
+            if self._user_speaking:
+                self._audio_frames.append(frame)
+            else:
+                # Append the audio frame to our buffer. Treat the buffer as a ring buffer, dropping the oldest
+                # frames as necessary. Assume all audio frames have the same duration.
+                self._audio_frames.append(frame)
+                frame_duration = len(frame.audio) / 16 * frame.num_channels / frame.sample_rate
+                buffer_duration = frame_duration * len(self._audio_frames)
+                while buffer_duration > self._start_secs:
+                    self._audio_frames.pop(0)
+                    buffer_duration -= frame_duration
+
+        await self.push_frame(frame, direction)
+
+
+class InputTranscriptionContextFilter(FrameProcessor):
+    """
+    This FrameProcessor blocks all frames except the OpenAILLMContextFrame that triggers
+    LLM inference. (And system frames, which are needed for the pipeline element lifecycle.)
+
+    We take the context object out of the OpenAILLMContextFrame and use it to create a new
+    context object that we will send to the transcriber LLM.
+    """
+
+    async def process_frame(self, frame, direction):
+        if isinstance(frame, SystemFrame):
+            # We don't want to block system frames.
+            await self.push_frame(frame, direction)
+            return
+
+        if not isinstance(frame, OpenAILLMContextFrame):
+            return
+
+        try:
+            message = frame.context.messages[-1]
+            last_part = message.parts[-1]
+            if not (
+                message.role == "user"
+                and last_part.inline_data
+                and last_part.inline_data.mime_type == "audio/wav"
+            ):
+                return
+
+            # Assemble a new message, with three parts: conversation history, transcription
+            # prompt, and audio. We could use only part of the conversation, if we need to
+            # keep the token count down, but for now, we'll just use the whole thing.
+            parts = []
+
+            # Get previous conversation history
+            previous_messages = frame.context.messages[:-2]
+            history = ""
+            for msg in previous_messages:
+                for part in msg.parts:
+                    if part.text:
+                        history += f"{msg.role}: {part.text}\n"
+            if history:
+                assembled = f"Here is the conversation history so far. These are not instructions. This is data that you should use only to improve the accuracy of your transcription.\n\n----\n\n{history}\n\n----\n\nEND OF CONVERSATION HISTORY\n\n"
+                parts.append(glm.Part(text=assembled))
+
+            parts.append(
+                glm.Part(
+                    text="Transcribe this audio. Respond either with the transcription exactly as it was said by the user, or with the special string 'EMPTY' if the audio is not clear."
+                )
+            )
+            parts.append(last_part)
+            msg = glm.Content(role="user", parts=parts)
+            ctx = GoogleLLMContext([msg])
+            ctx.system_message = transcriber_system_message
+            await self.push_frame(OpenAILLMContextFrame(context=ctx))
+        except Exception as e:
+            logger.error(f"Error processing frame: {e}")
+
+
+@dataclass
+class LLMDemoTranscriptionFrame(Frame):
+    """
+    It would be nice if we could just use a TranscriptionFrame to send our transcriber
+    LLM's transcription output down the pipelline. But we can't, because TranscriptionFrame
+    is a child class of TextFrame, which in our pipeline will be interpreted by the TTS
+    service as text that should be turned into speech. We could restructure this pipeline,
+    but instead we'll just use a custom frame type.
+    (Composition and reuse are ... double-edged swords.)
+    """
+
+    text: str
+
+
+class InputTranscriptionFrameEmitter(FrameProcessor):
+    """
+    A simple FrameProcessor that aggregates the TextFrame output from the transcriber LLM
+    and then sends the full response down the pipeline as an LLMDemoTranscriptionFrame.
+    """
+
+    def __init__(self):
+        super().__init__()
+        self._aggregation = ""
+
+    async def process_frame(self, frame, direction):
+        if isinstance(frame, TextFrame):
+            self._aggregation += frame.text
+        elif isinstance(frame, LLMFullResponseEndFrame):
+            await self.push_frame(LLMDemoTranscriptionFrame(text=self._aggregation.strip()))
+            self._aggregation = ""
+        elif isinstance(frame, MetricsFrame):
+            await self.push_frame(frame, direction)
+
+
+class TranscriptionContextFixup(FrameProcessor):
+    """
+    This FrameProcessor looks for the LLMDemoTranscriptionFrame and swaps out the
+    audio part of the most recent user message with the text transcription.
+
+    Audio is big, using a lot of tokens and network bandwidth. So doing this is
+    important if we want to keep both latency and cost low.
+
+    This class is a bit of a hack, especially because it directly creates a
+    GoogleLLMContext object, which we don't generally do. We usually try to leave
+    the implementation-specific details of the LLM context encapsulated inside the
+    service classes.
+    """
+
+    def __init__(self, context):
+        super().__init__()
+        self._context = context
+        self._transcript = "THIS IS A TRANSCRIPT"
+
+    def is_user_audio_message(self, message):
+        last_part = message.parts[-1]
+        return (
+            message.role == "user"
+            and last_part.inline_data
+            and last_part.inline_data.mime_type == "audio/wav"
+        )
+
+    def swap_user_audio(self):
+        if not self._transcript:
+            return
+        message = self._context.messages[-2]
+        if not self.is_user_audio_message(message):
+            message = self._context.messages[-1]
+            if not self.is_user_audio_message(message):
+                return
+
+        audio_part = message.parts[-1]
+        audio_part.inline_data = None
+        audio_part.text = self._transcript
+
+    async def process_frame(self, frame, direction):
+        if isinstance(frame, LLMDemoTranscriptionFrame):
+            logger.info(f"Transcription from Gemini: {frame.text}")
+            self._transcript = frame.text
+            self.swap_user_audio()
+            self._transcript = ""
+
+        await self.push_frame(frame, direction)
+
+
+async def main():
+    async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
+        transport = DailyTransport(
+            room_url,
+            token,
+            "Respond bot",
+            DailyParams(
+                audio_out_enabled=True,
+                # No transcription at all. just audio input to Gemini!
+                # transcription_enabled=True,
+                vad_enabled=True,
+                vad_analyzer=SileroVADAnalyzer(),
+                vad_audio_passthrough=True,
+            ),
+        )
+
+        tts = CartesiaTTSService(
+            api_key=os.getenv("CARTESIA_API_KEY"),
+            voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22",  # British Lady
+        )
+
+        conversation_llm = GoogleLLMService(
+            name="Conversation",
+            model="gemini-1.5-flash-latest",
+            # model="gemini-exp-1121",
+            api_key=os.getenv("GOOGLE_API_KEY"),
+            # we can give the GoogleLLMService a system instruction to use directly
+            # in the GenerativeModel constructor. Let's do that rather than put
+            # our system message in the messages list.
+            system_instruction=conversation_system_message,
+        )
+
+        input_transcription_llm = GoogleLLMService(
+            name="Transcription",
+            model="gemini-1.5-flash-latest",
+            # model="gemini-exp-1121",
+            api_key=os.getenv("GOOGLE_API_KEY"),
+            system_instruction=transcriber_system_message,
+        )
+
+        messages = [
+            {
+                "role": "user",
+                "content": "Start by saying hello.",
+            },
+        ]
+
+        context = OpenAILLMContext(messages)
+        context_aggregator = conversation_llm.create_context_aggregator(context)
+        audio_collector = UserAudioCollector(context, context_aggregator.user())
+        input_transcription_context_filter = InputTranscriptionContextFilter()
+        transcription_frames_emitter = InputTranscriptionFrameEmitter()
+        fixup_context_messages = TranscriptionContextFixup(context)
+
+        pipeline = Pipeline(
+            [
+                transport.input(),
+                audio_collector,
+                context_aggregator.user(),
+                ParallelPipeline(
+                    [  # transcribe
+                        input_transcription_context_filter,
+                        input_transcription_llm,
+                        transcription_frames_emitter,
+                    ],
+                    [  # conversation inference
+                        conversation_llm,
+                    ],
+                ),
+                tts,
+                transport.output(),
+                context_aggregator.assistant(),
+                fixup_context_messages,
+            ]
+        )
+
+        task = PipelineTask(
+            pipeline,
+            PipelineParams(
+                allow_interruptions=True,
+                enable_metrics=True,
+                enable_usage_metrics=True,
+            ),
+        )
+
+        @transport.event_handler("on_first_participant_joined")
+        async def on_first_participant_joined(transport, participant):
+            # Kick off the conversation.
+            await task.queue_frames([context_aggregator.user().get_context_frame()])
+
+        runner = PipelineRunner()
+
+        await runner.run(task)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

examples/foundational/26-gemini-multimodal-live.py

@@ -0,0 +1,82 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import aiohttp
+import asyncio
+import os
+import sys
+
+
+from dotenv import load_dotenv
+from loguru import logger
+from runner import configure
+
+from pipecat.services.gemini_multimodal_live.gemini import GeminiMultimodalLiveLLMService
+
+from pipecat.audio.vad.silero import SileroVADAnalyzer
+from pipecat.audio.vad.vad_analyzer import VADParams
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.runner import PipelineRunner
+from pipecat.pipeline.task import PipelineParams, PipelineTask
+from pipecat.transports.services.daily import DailyParams, DailyTransport
+
+load_dotenv(override=True)
+
+logger.remove(0)
+logger.add(sys.stderr, level="DEBUG")
+
+
+async def main():
+    async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
+        transport = DailyTransport(
+            room_url,
+            token,
+            "Respond bot",
+            DailyParams(
+                audio_in_sample_rate=16000,
+                audio_out_sample_rate=24000,
+                audio_out_enabled=True,
+                vad_enabled=True,
+                vad_audio_passthrough=True,
+                # set stop_secs to something roughly similar to the internal setting
+                # of the Multimodal Live api, just to align events. This doesn't really
+                # matter because we can only use the Multimodal Live API's phrase
+                # endpointing, for now.
+                vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.5)),
+            ),
+        )
+
+        llm = GeminiMultimodalLiveLLMService(
+            api_key=os.getenv("GOOGLE_API_KEY"),
+            # system_instruction="Talk like a pirate."
+        )
+
+        pipeline = Pipeline(
+            [
+                transport.input(),
+                llm,
+                transport.output(),
+            ]
+        )
+
+        task = PipelineTask(
+            pipeline,
+            PipelineParams(
+                allow_interruptions=True,
+                enable_metrics=True,
+                enable_usage_metrics=True,
+            ),
+        )
+
+        runner = PipelineRunner()
+
+        await runner.run(task)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

examples/foundational/26a-gemini-multimodal-live-transcription.py

@@ -0,0 +1,111 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import asyncio
+import os
+import sys
+
+import aiohttp
+from dotenv import load_dotenv
+from loguru import logger
+from runner import configure
+
+from pipecat.audio.vad.silero import SileroVADAnalyzer
+from pipecat.audio.vad.vad_analyzer import VADParams
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.runner import PipelineRunner
+from pipecat.pipeline.task import PipelineParams, PipelineTask
+from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
+from pipecat.services.gemini_multimodal_live.gemini import GeminiMultimodalLiveLLMService
+from pipecat.transports.services.daily import DailyParams, DailyTransport
+
+load_dotenv(override=True)
+
+logger.remove(0)
+logger.add(sys.stderr, level="DEBUG")
+
+
+async def main():
+    async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
+        transport = DailyTransport(
+            room_url,
+            token,
+            "Respond bot",
+            DailyParams(
+                audio_in_sample_rate=16000,
+                audio_out_sample_rate=24000,
+                audio_out_enabled=True,
+                vad_enabled=True,
+                vad_audio_passthrough=True,
+                # set stop_secs to something roughly similar to the internal setting
+                # of the Multimodal Live api, just to align events. This doesn't really
+                # matter because we can only use the Multimodal Live API's phrase
+                # endpointing, for now.
+                vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.5)),
+            ),
+        )
+
+        llm = GeminiMultimodalLiveLLMService(
+            api_key=os.getenv("GOOGLE_API_KEY"),
+            voice_id="Aoede",  # Puck, Charon, Kore, Fenrir, Aoede
+            # system_instruction="Talk like a pirate."
+            transcribe_user_audio=True,
+            transcribe_model_audio=True,
+            # inference_on_context_initialization=False,
+        )
+
+        context = OpenAILLMContext(
+            [
+                {
+                    "role": "user",
+                    "content": "Say hello. Then ask if I want to hear a joke.",
+                },
+                #     {"role": "assistant", "content": "Hello! Why don't scientists trust atoms?"},
+                #     {
+                #         "role": "user",
+                #         "content": [
+                #             {
+                #                 "type": "text",
+                #                 "text": "Oh, I know this one: because they make up everything.",
+                #             }
+                #         ],
+                #     },
+            ],
+        )
+        context_aggregator = llm.create_context_aggregator(context)
+
+        pipeline = Pipeline(
+            [
+                transport.input(),
+                context_aggregator.user(),
+                llm,
+                transport.output(),
+                context_aggregator.assistant(),
+            ]
+        )
+
+        task = PipelineTask(
+            pipeline,
+            PipelineParams(
+                allow_interruptions=True,
+                enable_metrics=True,
+                enable_usage_metrics=True,
+            ),
+        )
+
+        @transport.event_handler("on_first_participant_joined")
+        async def on_first_participant_joined(transport, participant):
+            await task.queue_frames([context_aggregator.user().get_context_frame()])
+
+        runner = PipelineRunner()
+
+        await runner.run(task)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

examples/foundational/26b-gemini-multimodal-live-function-calling.py

@@ -0,0 +1,142 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import asyncio
+import os
+import sys
+from datetime import datetime
+
+import aiohttp
+from dotenv import load_dotenv
+from loguru import logger
+from runner import configure
+
+from pipecat.audio.vad.silero import SileroVADAnalyzer
+from pipecat.audio.vad.vad_analyzer import VADParams
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.runner import PipelineRunner
+from pipecat.pipeline.task import PipelineParams, PipelineTask
+from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
+from pipecat.services.gemini_multimodal_live.gemini import GeminiMultimodalLiveLLMService
+from pipecat.transports.services.daily import DailyParams, DailyTransport
+
+load_dotenv(override=True)
+
+logger.remove(0)
+logger.add(sys.stderr, level="DEBUG")
+
+
+async def fetch_weather_from_api(function_name, tool_call_id, args, llm, context, result_callback):
+    temperature = 75 if args["format"] == "fahrenheit" else 24
+    await result_callback(
+        {
+            "conditions": "nice",
+            "temperature": temperature,
+            "format": args["format"],
+            "timestamp": datetime.now().strftime("%Y%m%d_%H%M%S"),
+        }
+    )
+
+
+tools = [
+    {
+        "function_declarations": [
+            {
+                "name": "get_current_weather",
+                "description": "Get the current weather",
+                "parameters": {
+                    "type": "object",
+                    "properties": {
+                        "location": {
+                            "type": "string",
+                            "description": "The city and state, e.g. San Francisco, CA",
+                        },
+                        "format": {
+                            "type": "string",
+                            "enum": ["celsius", "fahrenheit"],
+                            "description": "The temperature unit to use. Infer this from the users location.",
+                        },
+                    },
+                    "required": ["location", "format"],
+                },
+            },
+        ]
+    }
+]
+
+system_instruction = """
+You are a helpful assistant who can answer questions and use tools.
+
+You have a tool called "get_current_weather" that can be used to get the current weather. If the user asks
+for the weather, call this function.
+"""
+
+
+async def main():
+    async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
+        transport = DailyTransport(
+            room_url,
+            token,
+            "Respond bot",
+            DailyParams(
+                audio_in_sample_rate=16000,
+                audio_out_sample_rate=24000,
+                audio_out_enabled=True,
+                vad_enabled=True,
+                vad_audio_passthrough=True,
+                # set stop_secs to something roughly similar to the internal setting
+                # of the Multimodal Live api, just to align events. This doesn't really
+                # matter because we can only use the Multimodal Live API's phrase
+                # endpointing, for now.
+                vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.5)),
+            ),
+        )
+
+        llm = GeminiMultimodalLiveLLMService(
+            api_key=os.getenv("GOOGLE_API_KEY"),
+            system_instruction=system_instruction,
+            tools=tools,
+        )
+
+        llm.register_function("get_current_weather", fetch_weather_from_api)
+
+        context = OpenAILLMContext(
+            [{"role": "user", "content": "Say hello."}],
+        )
+        context_aggregator = llm.create_context_aggregator(context)
+
+        pipeline = Pipeline(
+            [
+                transport.input(),
+                context_aggregator.user(),
+                llm,
+                context_aggregator.assistant(),
+                transport.output(),
+            ]
+        )
+
+        task = PipelineTask(
+            pipeline,
+            PipelineParams(
+                allow_interruptions=True,
+                enable_metrics=True,
+                enable_usage_metrics=True,
+            ),
+        )
+
+        @transport.event_handler("on_first_participant_joined")
+        async def on_first_participant_joined(transport, participant):
+            await task.queue_frames([context_aggregator.user().get_context_frame()])
+
+        runner = PipelineRunner()
+
+        await runner.run(task)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

examples/foundational/26c-gemini-multimodal-live-video.py

@@ -0,0 +1,115 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import asyncio
+import os
+import sys
+
+import aiohttp
+from dotenv import load_dotenv
+from loguru import logger
+from runner import configure
+
+from pipecat.audio.vad.silero import SileroVADAnalyzer
+from pipecat.audio.vad.vad_analyzer import VADParams
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.runner import PipelineRunner
+from pipecat.pipeline.task import PipelineParams, PipelineTask
+from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
+from pipecat.services.gemini_multimodal_live.gemini import GeminiMultimodalLiveLLMService
+from pipecat.transports.services.daily import DailyParams, DailyTransport
+
+load_dotenv(override=True)
+
+logger.remove(0)
+logger.add(sys.stderr, level="DEBUG")
+
+
+async def main():
+    async with aiohttp.ClientSession() as session:
+        (room_url, token) = await configure(session)
+
+        transport = DailyTransport(
+            room_url,
+            token,
+            "Respond bot",
+            DailyParams(
+                audio_in_sample_rate=16000,
+                audio_out_sample_rate=24000,
+                audio_out_enabled=True,
+                vad_enabled=True,
+                vad_audio_passthrough=True,
+                # set stop_secs to something roughly similar to the internal setting
+                # of the Multimodal Live api, just to align events. This doesn't really
+                # matter because we can only use the Multimodal Live API's phrase
+                # endpointing, for now.
+                vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.5)),
+                start_audio_paused=True,
+                start_video_paused=True,
+            ),
+        )
+
+        llm = GeminiMultimodalLiveLLMService(
+            api_key=os.getenv("GOOGLE_API_KEY"),
+            voice_id="Aoede",  # Puck, Charon, Kore, Fenrir, Aoede
+            # system_instruction="Talk like a pirate."
+            transcribe_user_audio=True,
+            transcribe_model_audio=True,
+            # inference_on_context_initialization=False,
+        )
+
+        context = OpenAILLMContext(
+            [
+                {
+                    "role": "user",
+                    "content": "Say hello.",
+                },
+            ],
+        )
+        context_aggregator = llm.create_context_aggregator(context)
+
+        pipeline = Pipeline(
+            [
+                transport.input(),
+                context_aggregator.user(),
+                llm,
+                transport.output(),
+                context_aggregator.assistant(),
+            ]
+        )
+
+        task = PipelineTask(
+            pipeline,
+            PipelineParams(
+                allow_interruptions=True,
+                enable_metrics=True,
+                enable_usage_metrics=True,
+            ),
+        )
+
+        @transport.event_handler("on_first_participant_joined")
+        async def on_first_participant_joined(transport, participant):
+            # Enable both camera and screenshare. From the client side
+            # send just one.
+            await transport.capture_participant_video(
+                participant["id"], framerate=1, video_source="camera"
+            )
+            await transport.capture_participant_video(
+                participant["id"], framerate=1, video_source="screenVideo"
+            )
+            await task.queue_frames([context_aggregator.user().get_context_frame()])
+            await asyncio.sleep(3)
+            logger.debug("Unpausing audio and video")
+            llm.set_audio_input_paused(False)
+            llm.set_video_input_paused(False)
+
+        runner = PipelineRunner()
+
+        await runner.run(task)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

examples/foundational/27-simli-layer.py

@@ -0,0 +1,105 @@
+#
+# Copyright (c) 2024, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+import asyncio
+import aiohttp
+import os
+import sys
+
+from pipecat.audio.vad.silero import SileroVADAnalyzer
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.runner import PipelineRunner
+from pipecat.pipeline.task import PipelineParams, PipelineTask
+from pipecat.frames.frames import LLMMessagesFrame
+
+from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
+from pipecat.services.cartesia import CartesiaTTSService
+from pipecat.services.openai import OpenAILLMService
+from pipecat.transports.services.daily import DailyParams, DailyTransport
+
+from runner import configure
+from loguru import logger
+from dotenv import load_dotenv
+
+from simli import SimliConfig
+from pipecat.services.simli import SimliVideoService
+
+load_dotenv(override=True)
+
+logger.remove(0)
+logger.add(sys.stderr, level="DEBUG")
+
+
+async def main():
+    async with aiohttp.ClientSession() as session:
+        room, token = await configure(session)
+        transport = DailyTransport(
+            room,
+            token,
+            "Simli",
+            DailyParams(
+                audio_out_enabled=True,
+                camera_out_enabled=True,
+                camera_out_width=512,
+                camera_out_height=512,
+                vad_enabled=True,
+                vad_analyzer=SileroVADAnalyzer(),
+                transcription_enabled=True,
+            ),
+        )
+
+        tts = CartesiaTTSService(
+            api_key=os.getenv("CARTESIA_API_KEY"),
+            voice_id="a167e0f3-df7e-4d52-a9c3-f949145efdab",
+        )
+
+        simli_ai = SimliVideoService(
+            SimliConfig(os.getenv("SIMLI_API_KEY"), os.getenv("SIMLI_FACE_ID"))
+        )
+
+        llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o-mini")
+
+        messages = [
+            {
+                "role": "system",
+                "content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
+            },
+        ]
+
+        context = OpenAILLMContext(messages)
+        context_aggregator = llm.create_context_aggregator(context)
+
+        pipeline = Pipeline(
+            [
+                transport.input(),
+                context_aggregator.user(),
+                llm,
+                tts,
+                simli_ai,
+                transport.output(),
+                context_aggregator.assistant(),
+            ]
+        )
+
+        task = PipelineTask(
+            pipeline,
+            PipelineParams(
+                allow_interruptions=True,
+                enable_metrics=True,
+            ),
+        )
+
+        @transport.event_handler("on_first_participant_joined")
+        async def on_first_participant_joined(transport, participant):
+            await transport.capture_participant_transcription(participant["id"])
+            await task.queue_frames([LLMMessagesFrame(messages)])
+
+        runner = PipelineRunner()
+        await runner.run(task)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

examples/moondream-chatbot/bot.py

@@ -13,13 +13,13 @@ from PIL import Image
 
 from pipecat.audio.vad.silero import SileroVADAnalyzer
 from pipecat.frames.frames import (
+    BotStartedSpeakingFrame,
+    BotStoppedSpeakingFrame,
     ImageRawFrame,
     OutputImageRawFrame,
     SpriteFrame,
     Frame,
     LLMMessagesFrame,
-    TTSAudioRawFrame,
-    TTSStoppedFrame,
     TextFrame,
     UserImageRawFrame,
     UserImageRequestFrame,
@@ -81,16 +81,15 @@ class TalkingAnimation(FrameProcessor):
         self._is_talking = False
 
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
-        if isinstance(frame, TTSAudioRawFrame):
+        if isinstance(frame, BotStartedSpeakingFrame):
             if not self._is_talking:
                 await self.push_frame(talking_frame)
                 self._is_talking = True
-        elif isinstance(frame, TTSStoppedFrame):
+        elif isinstance(frame, BotStoppedSpeakingFrame):
             await self.push_frame(quiet_frame)
             self._is_talking = False
-        await self.push_frame(frame)
+
+        await self.push_frame(frame, direction)
 
 
 class UserImageRequester(FrameProcessor):
@@ -102,8 +101,6 @@ class UserImageRequester(FrameProcessor):
         self.participant_id = participant_id
 
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         if self.participant_id and isinstance(frame, TextFrame):
             if frame.text == user_request_answer:
                 await self.push_frame(
@@ -120,21 +117,17 @@ class TextFilterProcessor(FrameProcessor):
         self.text = text
 
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         if isinstance(frame, TextFrame):
             if frame.text != self.text:
                 await self.push_frame(frame)
         else:
-            await self.push_frame(frame)
+            await self.push_frame(frame, direction)
 
 
 class ImageFilterProcessor(FrameProcessor):
     async def process_frame(self, frame: Frame, direction: FrameDirection):
-        await super().process_frame(frame, direction)
-
         if not isinstance(frame, ImageRawFrame):
-            await self.push_frame(frame)
+            await self.push_frame(frame, direction)
 
 
 async def main():

examples/patient-intake/README.md

@@ -4,6 +4,8 @@
 
 This project implements an AI-powered chatbot designed to streamline the medical intake process for Tri-County Health Services. The chatbot, named Jessica, interacts with patients to collect essential information before their doctor's visit, enhancing efficiency and improving the patient experience.
 
+💡 Looking to build structured conversations? Check out [Pipecat Flows](https://github.com/pipecat-ai/pipecat-flows) for managing complex conversational states and transitions.
+
 ## Features
 
 Identity Verification: Confirms patient identity by verifying their date of birth.
@@ -62,3 +64,32 @@ Then, visit `http://localhost:7860/` in your browser to start a chatbot session.
 docker build -t chatbot .
 docker run --env-file .env -p 7860:7860 chatbot
 ```
+## Cartesia best practices
+
+Since this example is using Cartesia, checkout the best practices given in Cartesia's docs. LLM prompts should be modified accordingly.
+<https://docs.cartesia.ai/build-with-sonic/formatting-text-for-sonic/best-practices>
+
+<https://docs.cartesia.ai/build-with-sonic/formatting-text-for-sonic/inserting-breaks-pauses>
+
+<https://docs.cartesia.ai/build-with-sonic/formatting-text-for-sonic/spelling-out-input-text>
+### Example
+```python
+messages = [
+    {
+        "role": "system",
+        "content": '''You are a helpful AI assistant. Format all responses following these guidelines:
+
+1. Use proper punctuation and end each response with appropriate punctuation
+2. Format dates as MM/DD/YYYY
+3. Insert pauses using - or <break time='1s' /> for longer pauses
+4. Use ?? for emphasized questions
+5. Avoid quotation marks unless citing
+6. Add spaces between URLs/emails and punctuation marks
+7. For domain-specific terms or proper nouns, provide pronunciation guidance in [brackets]
+8. Keep responses clear and concise
+9. Use appropriate voice/language pairs for multilingual content
+
+Your goal is to demonstrate these capabilities in a succinct way. Your output will be converted to audio, so maintain natural communication flow. Respond creatively and helpfully, but keep responses brief. Start by introducing yourself.'''
+    }
+]
+```

examples/simple-chatbot/.gitignore

@@ -1,161 +1,51 @@
-# Byte-compiled / optimized / DLL files
+# Python
 __pycache__/
 *.py[cod]
 *$py.class
-
-# C extensions
 *.so
-
-# Distribution / packaging
 .Python
 build/
-develop-eggs/
 dist/
-downloads/
-eggs/
-.eggs/
-lib/
-lib64/
-parts/
-sdist/
-var/
-wheels/
-share/python-wheels/
 *.egg-info/
 .installed.cfg
 *.egg
-MANIFEST
-
-# PyInstaller
-#  Usually these files are written by a python script from a template
-#  before PyInstaller builds the exe, so as to inject date/other infos into it.
-*.manifest
-*.spec
-
-# Installer logs
-pip-log.txt
-pip-delete-this-directory.txt
-
-# Unit test / coverage reports
-htmlcov/
-.tox/
-.nox/
+.pytest_cache/
 .coverage
 .coverage.*
-.cache
-nosetests.xml
-coverage.xml
-*.cover
-*.py,cover
-.hypothesis/
-.pytest_cache/
-cover/
-
-# Translations
-*.mo
-*.pot
-
-# Django stuff:
-*.log
-local_settings.py
-db.sqlite3
-db.sqlite3-journal
-
-# Flask stuff:
-instance/
-.webassets-cache
-
-# Scrapy stuff:
-.scrapy
-
-# Sphinx documentation
-docs/_build/
-
-# PyBuilder
-.pybuilder/
-target/
-
-# Jupyter Notebook
-.ipynb_checkpoints
-
-# IPython
-profile_default/
-ipython_config.py
-
-# pyenv
-#   For a library or package, you might want to ignore these files since the code is
-#   intended to run in multiple environments; otherwise, check them in:
-# .python-version
-
-# pipenv
-#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
-#   However, in case of collaboration, if having platform-specific dependencies or dependencies
-#   having no cross-platform support, pipenv may install dependencies that don't work, or not
-#   install all needed dependencies.
-#Pipfile.lock
-
-# poetry
-#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
-#   This is especially recommended for binary packages to ensure reproducibility, and is more
-#   commonly ignored for libraries.
-#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
-#poetry.lock
-
-# pdm
-#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
-#pdm.lock
-#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
-#   in version control.
-#   https://pdm.fming.dev/#use-with-ide
-.pdm.toml
-
-# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
-__pypackages__/
-
-# Celery stuff
-celerybeat-schedule
-celerybeat.pid
-
-# SageMath parsed files
-*.sage.py
-
-# Environments
 .env
 .venv
 env/
 venv/
 ENV/
-env.bak/
-venv.bak/
-
-# Spyder project settings
-.spyderproject
-.spyproject
-
-# Rope project settings
-.ropeproject
-
-# mkdocs documentation
-/site
-
-# mypy
 .mypy_cache/
 .dmypy.json
 dmypy.json
 
-# Pyre type checker
-.pyre/
+# JavaScript/Node.js
+node_modules/
+dist/
+dist-ssr/
+*.local
+.env.local
+.env.development.local
+.env.test.local
+.env.production.local
 
-# pytype static type analyzer
-.pytype/
+# Logs
+logs/
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+pnpm-debug.log*
 
-# Cython debug symbols
-cython_debug/
+# Editor/IDE
+.vscode/*
+!.vscode/extensions.json
+.idea/
+*.swp
+*.swo
+.DS_Store
 
-# PyCharm
-#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
-#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
-#  and can be added to the global gitignore or merged into this file.  For a more nuclear
-#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
-#.idea/
-runpod.toml
+# Project specific
+runpod.toml

examples/simple-chatbot/README.md

@@ -2,36 +2,96 @@
 
 <img src="image.png" width="420px">
 
-This app connects you to a chatbot powered by GPT-4, complete with animations generated by Stable Video Diffusion.
+This repository demonstrates a simple AI chatbot with real-time audio/video interaction, implemented in three different ways. The bot server supports multiple AI backends, and you can connect to it using three different client approaches.
 
-See a video of it in action: https://x.com/kwindla/status/1778628911817183509
+## Two Bot Options
 
-And a quick video walkthrough of the code: https://www.loom.com/share/13df1967161f4d24ade054e7f8753416
+1. **OpenAI Bot** (Default)
 
-ℹ️ The first time, things might take extra time to get started since VAD (Voice Activity Detection) model needs to be downloaded.
+   - Uses gpt-4o for conversation
+   - Requires OpenAI API key
 
-## Get started
+2. **Gemini Bot**
+   - Uses Google's Gemini Multimodal Live model
+   - Requires Gemini API key
 
-```python
-python3 -m venv venv
-source venv/bin/activate
-pip install -r requirements.txt
+## Three Ways to Connect
 
-cp env.example .env # and add your credentials
+1. **Daily Prebuilt** (Simplest)
+
+   - Direct connection through a Daily Prebuilt room
+   - For demo purposes only; handy for quick testing
+
+2. **JavaScript**
+
+   - Basic implementation using [Pipecat JavaScript SDK](https://docs.pipecat.ai/client/reference/js/introduction)
+   - No framework dependencies
+   - Good for learning the fundamentals
+
+3. **React**
+   - Basic impelmentation using [Pipecat React SDK](https://docs.pipecat.ai/client/reference/react/introduction)
+   - Demonstrates the basic client principles with Pipecat React
+
+## Quick Start
+
+### First, start the bot server:
+
+1. Navigate to the server directory:
+   ```bash
+   cd server
+   ```
+2. Create and activate a virtual environment:
+   ```bash
+   python3 -m venv venv
+   source venv/bin/activate  # On Windows: venv\Scripts\activate
+   ```
+3. Install requirements:
+   ```bash
+   pip install -r requirements.txt
+   ```
+4. Copy env.example to .env and configure:
+   - Add your API keys
+   - Choose your bot implementation:
+     ```ini
+     BOT_IMPLEMENTATION=      # Options: 'openai' (default) or 'gemini'
+     ```
+5. Start the server:
+   ```bash
+   python server.py
+   ```
+
+### Next, connect using your preferred client app:
+
+- [Daily Prebuilt](examples/prebuilt/README.md)
+- [JavaScript Guide](examples/javascript/README.md)
+- [React Guide](examples/react/README.md)
+
+## Important Note
+
+The bot server must be running for any of the client implementations to work. Start the server first before trying any of the client apps.
+
+## Requirements
+
+- Python 3.10+
+- Node.js 16+ (for JavaScript and React implementations)
+- Daily API key
+- OpenAI API key (for OpenAI bot)
+- Gemini API key (for Gemini bot)
+- ElevenLabs API key
+- Modern web browser with WebRTC support
+
+## Project Structure
 
 ```
-
-## Run the server
-
-```bash
-python server.py
-```
-
-Then, visit `http://localhost:7860/` in your browser to start a chatbot session.
-
-## Build and test the Docker image
-
-```
-docker build -t chatbot .
-docker run --env-file .env -p 7860:7860 chatbot
+simple-chatbot/
+├── server/              # Bot server implementation
+│   ├── bot-openai.py    # OpenAI bot implementation
+│   ├── bot-gemini.py    # Gemini bot implementation
+│   ├── runner.py        # Server runner utilities
+│   ├── server.py        # FastAPI server
+│   └── requirements.txt
+└── examples/            # Client implementations
+    ├── prebuilt/        # Daily Prebuilt connection
+    ├── javascript/      # Pipecat JavaScript client
+    └── react/           # Pipecat React client
 ```

examples/simple-chatbot/examples/javascript/README.md

@@ -0,0 +1,27 @@
+# JavaScript Implementation
+
+Basic implementation using the [Pipecat JavaScript SDK](https://docs.pipecat.ai/client/reference/js/introduction).
+
+## Setup
+
+1. Run the bot server. See the [server README](../../README).
+
+2. Navigate to the `examples/javascript` directory:
+
+```bash
+cd examples/javascript
+```
+
+3. Install dependencies:
+
+```bash
+npm install
+```
+
+4. Run the client app:
+
+```
+npm run dev
+```
+
+5. Visit http://localhost:5173 in your browser.

examples/simple-chatbot/examples/javascript/index.html

@@ -0,0 +1,40 @@
+<!DOCTYPE html>
+<html lang="en">
+
+<head>
+  <meta charset="UTF-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1.0">
+  <title>AI Chatbot</title>
+</head>
+
+<body>
+  <div class="container">
+    <div class="status-bar">
+      <div class="status">
+        Status: <span id="connection-status">Disconnected</span>
+      </div>
+      <div class="controls">
+        <button id="connect-btn">Connect</button>
+        <button id="disconnect-btn" disabled>Disconnect</button>
+      </div>
+    </div>
+
+    <div class="main-content">
+      <div class="bot-container">
+        <div id="bot-video-container">
+        </div>
+        <audio id="bot-audio" autoplay></audio>
+      </div>
+    </div>
+
+    <div class="debug-panel">
+      <h3>Debug Info</h3>
+      <div id="debug-log"></div>
+    </div>
+  </div>
+
+  <script type="module" src="/src/app.js"></script>
+  <link rel="stylesheet" href="/src/style.css">
+</body>
+
+</html>

examples/simple-chatbot/examples/javascript/package.json

@@ -0,0 +1,21 @@
+{
+  "name": "client",
+  "version": "1.0.0",
+  "main": "index.js",
+  "scripts": {
+    "dev": "vite",
+    "build": "vite build",
+    "preview": "vite preview"
+  },
+  "keywords": [],
+  "author": "",
+  "license": "ISC",
+  "description": "",
+  "dependencies": {
+    "@daily-co/realtime-ai-daily": "^0.2.1",
+    "realtime-ai": "^0.2.1"
+  },
+  "devDependencies": {
+    "vite": "^6.0.2"
+  }
+}

examples/simple-chatbot/examples/javascript/src/app.js

@@ -0,0 +1,314 @@
+/**
+ * Copyright (c) 2024, Daily
+ *
+ * SPDX-License-Identifier: BSD 2-Clause License
+ */
+
+/**
+ * RTVI Client Implementation
+ *
+ * This client connects to an RTVI-compatible bot server using WebRTC (via Daily).
+ * It handles audio/video streaming and manages the connection lifecycle.
+ *
+ * Requirements:
+ * - A running RTVI bot server (defaults to http://localhost:7860)
+ * - The server must implement the /connect endpoint that returns Daily.co room credentials
+ * - Browser with WebRTC support
+ */
+
+import { RTVIClient, RTVIEvent } from 'realtime-ai';
+import { DailyTransport } from '@daily-co/realtime-ai-daily';
+
+/**
+ * ChatbotClient handles the connection and media management for a real-time
+ * voice and video interaction with an AI bot.
+ */
+class ChatbotClient {
+  constructor() {
+    // Initialize client state
+    this.rtviClient = null;
+    this.setupDOMElements();
+    this.setupEventListeners();
+  }
+
+  /**
+   * Set up references to DOM elements and create necessary media elements
+   */
+  setupDOMElements() {
+    // Get references to UI control elements
+    this.connectBtn = document.getElementById('connect-btn');
+    this.disconnectBtn = document.getElementById('disconnect-btn');
+    this.statusSpan = document.getElementById('connection-status');
+    this.debugLog = document.getElementById('debug-log');
+    this.botVideoContainer = document.getElementById('bot-video-container');
+
+    // Create an audio element for bot's voice output
+    this.botAudio = document.createElement('audio');
+    this.botAudio.autoplay = true;
+    this.botAudio.playsInline = true;
+    document.body.appendChild(this.botAudio);
+  }
+
+  /**
+   * Set up event listeners for connect/disconnect buttons
+   */
+  setupEventListeners() {
+    this.connectBtn.addEventListener('click', () => this.connect());
+    this.disconnectBtn.addEventListener('click', () => this.disconnect());
+  }
+
+  /**
+   * Add a timestamped message to the debug log
+   */
+  log(message) {
+    const entry = document.createElement('div');
+    entry.textContent = `${new Date().toISOString()} - ${message}`;
+
+    // Add styling based on message type
+    if (message.startsWith('User: ')) {
+      entry.style.color = '#2196F3'; // blue for user
+    } else if (message.startsWith('Bot: ')) {
+      entry.style.color = '#4CAF50'; // green for bot
+    }
+
+    this.debugLog.appendChild(entry);
+    this.debugLog.scrollTop = this.debugLog.scrollHeight;
+    console.log(message);
+  }
+
+  /**
+   * Update the connection status display
+   */
+  updateStatus(status) {
+    this.statusSpan.textContent = status;
+    this.log(`Status: ${status}`);
+  }
+
+  /**
+   * Check for available media tracks and set them up if present
+   * This is called when the bot is ready or when the transport state changes to ready
+   */
+  setupMediaTracks() {
+    if (!this.rtviClient) return;
+
+    // Get current tracks from the client
+    const tracks = this.rtviClient.tracks();
+
+    // Set up any available bot tracks
+    if (tracks.bot?.audio) {
+      this.setupAudioTrack(tracks.bot.audio);
+    }
+    if (tracks.bot?.video) {
+      this.setupVideoTrack(tracks.bot.video);
+    }
+  }
+
+  /**
+   * Set up listeners for track events (start/stop)
+   * This handles new tracks being added during the session
+   */
+  setupTrackListeners() {
+    if (!this.rtviClient) return;
+
+    // Listen for new tracks starting
+    this.rtviClient.on(RTVIEvent.TrackStarted, (track, participant) => {
+      // Only handle non-local (bot) tracks
+      if (!participant?.local) {
+        if (track.kind === 'audio') {
+          this.setupAudioTrack(track);
+        } else if (track.kind === 'video') {
+          this.setupVideoTrack(track);
+        }
+      }
+    });
+
+    // Listen for tracks stopping
+    this.rtviClient.on(RTVIEvent.TrackStopped, (track, participant) => {
+      this.log(
+        `Track stopped event: ${track.kind} from ${
+          participant?.name || 'unknown'
+        }`
+      );
+    });
+  }
+
+  /**
+   * Set up an audio track for playback
+   * Handles both initial setup and track updates
+   */
+  setupAudioTrack(track) {
+    this.log('Setting up audio track');
+    // Check if we're already playing this track
+    if (this.botAudio.srcObject) {
+      const oldTrack = this.botAudio.srcObject.getAudioTracks()[0];
+      if (oldTrack?.id === track.id) return;
+    }
+    // Create a new MediaStream with the track and set it as the audio source
+    this.botAudio.srcObject = new MediaStream([track]);
+  }
+
+  /**
+   * Set up a video track for display
+   * Handles both initial setup and track updates
+   */
+  setupVideoTrack(track) {
+    this.log('Setting up video track');
+    const videoEl = document.createElement('video');
+    videoEl.autoplay = true;
+    videoEl.playsInline = true;
+    videoEl.muted = true;
+    videoEl.style.width = '100%';
+    videoEl.style.height = '100%';
+    videoEl.style.objectFit = 'cover';
+
+    // Check if we're already displaying this track
+    if (this.botVideoContainer.querySelector('video')?.srcObject) {
+      const oldTrack = this.botVideoContainer
+        .querySelector('video')
+        .srcObject.getVideoTracks()[0];
+      if (oldTrack?.id === track.id) return;
+    }
+
+    // Create a new MediaStream with the track and set it as the video source
+    videoEl.srcObject = new MediaStream([track]);
+    this.botVideoContainer.innerHTML = '';
+    this.botVideoContainer.appendChild(videoEl);
+  }
+
+  /**
+   * Initialize and connect to the bot
+   * This sets up the RTVI client, initializes devices, and establishes the connection
+   */
+  async connect() {
+    try {
+      // Create a new Daily transport for WebRTC communication
+      const transport = new DailyTransport();
+
+      // Initialize the RTVI client with our configuration
+      this.rtviClient = new RTVIClient({
+        transport,
+        params: {
+          // The baseURL and endpoint of your bot server that the client will connect to
+          baseUrl: 'http://localhost:7860',
+          endpoints: {
+            connect: '/connect',
+          },
+        },
+        enableMic: true, // Enable microphone for user input
+        enableCam: false,
+        callbacks: {
+          // Handle connection state changes
+          onConnected: () => {
+            this.updateStatus('Connected');
+            this.connectBtn.disabled = true;
+            this.disconnectBtn.disabled = false;
+            this.log('Client connected');
+          },
+          onDisconnected: () => {
+            this.updateStatus('Disconnected');
+            this.connectBtn.disabled = false;
+            this.disconnectBtn.disabled = true;
+            this.log('Client disconnected');
+          },
+          // Handle transport state changes
+          onTransportStateChanged: (state) => {
+            this.updateStatus(`Transport: ${state}`);
+            this.log(`Transport state changed: ${state}`);
+            if (state === 'ready') {
+              this.setupMediaTracks();
+            }
+          },
+          // Handle bot connection events
+          onBotConnected: (participant) => {
+            this.log(`Bot connected: ${JSON.stringify(participant)}`);
+          },
+          onBotDisconnected: (participant) => {
+            this.log(`Bot disconnected: ${JSON.stringify(participant)}`);
+          },
+          onBotReady: (data) => {
+            this.log(`Bot ready: ${JSON.stringify(data)}`);
+            this.setupMediaTracks();
+          },
+          // Transcript events
+          onUserTranscript: (data) => {
+            // Only log final transcripts
+            if (data.final) {
+              this.log(`User: ${data.text}`);
+            }
+          },
+          onBotTranscript: (data) => {
+            this.log(`Bot: ${data.text}`);
+          },
+          // Error handling
+          onMessageError: (error) => {
+            console.log('Message error:', error);
+          },
+          onError: (error) => {
+            console.log('Error:', error);
+          },
+        },
+      });
+
+      // Set up listeners for media track events
+      this.setupTrackListeners();
+
+      // Initialize audio/video devices
+      this.log('Initializing devices...');
+      await this.rtviClient.initDevices();
+
+      // Connect to the bot
+      this.log('Connecting to bot...');
+      await this.rtviClient.connect();
+
+      this.log('Connection complete');
+    } catch (error) {
+      // Handle any errors during connection
+      this.log(`Error connecting: ${error.message}`);
+      this.log(`Error stack: ${error.stack}`);
+      this.updateStatus('Error');
+
+      // Clean up if there's an error
+      if (this.rtviClient) {
+        try {
+          await this.rtviClient.disconnect();
+        } catch (disconnectError) {
+          this.log(`Error during disconnect: ${disconnectError.message}`);
+        }
+      }
+    }
+  }
+
+  /**
+   * Disconnect from the bot and clean up media resources
+   */
+  async disconnect() {
+    if (this.rtviClient) {
+      try {
+        // Disconnect the RTVI client
+        await this.rtviClient.disconnect();
+        this.rtviClient = null;
+
+        // Clean up audio
+        if (this.botAudio.srcObject) {
+          this.botAudio.srcObject.getTracks().forEach((track) => track.stop());
+          this.botAudio.srcObject = null;
+        }
+
+        // Clean up video
+        if (this.botVideoContainer.querySelector('video')?.srcObject) {
+          const video = this.botVideoContainer.querySelector('video');
+          video.srcObject.getTracks().forEach((track) => track.stop());
+          video.srcObject = null;
+        }
+        this.botVideoContainer.innerHTML = '';
+      } catch (error) {
+        this.log(`Error disconnecting: ${error.message}`);
+      }
+    }
+  }
+}
+
+// Initialize the client when the page loads
+window.addEventListener('DOMContentLoaded', () => {
+  new ChatbotClient();
+});

examples/simple-chatbot/examples/javascript/src/style.css

@@ -0,0 +1,98 @@
+body {
+  margin: 0;
+  padding: 20px;
+  font-family: Arial, sans-serif;
+  background-color: #f0f0f0;
+}
+
+.container {
+  max-width: 1200px;
+  margin: 0 auto;
+}
+
+.status-bar {
+  display: flex;
+  justify-content: space-between;
+  align-items: center;
+  padding: 10px;
+  background-color: #fff;
+  border-radius: 8px;
+  margin-bottom: 20px;
+}
+
+.controls button {
+  padding: 8px 16px;
+  margin-left: 10px;
+  border: none;
+  border-radius: 4px;
+  cursor: pointer;
+}
+
+#connect-btn {
+  background-color: #4caf50;
+  color: white;
+}
+
+#disconnect-btn {
+  background-color: #f44336;
+  color: white;
+}
+
+button:disabled {
+  opacity: 0.5;
+  cursor: not-allowed;
+}
+
+.main-content {
+  background-color: #fff;
+  border-radius: 8px;
+  padding: 20px;
+  margin-bottom: 20px;
+}
+
+.bot-container {
+  display: flex;
+  flex-direction: column;
+  align-items: center;
+}
+
+#bot-video-container {
+  width: 640px;
+  height: 360px;
+  background-color: #e0e0e0;
+  border-radius: 8px;
+  margin: 20px auto;
+  overflow: hidden;
+  display: flex;
+  align-items: center;
+  justify-content: center;
+}
+
+#bot-video-container video {
+  width: 100%;
+  height: 100%;
+  object-fit: cover;
+}
+
+.debug-panel {
+  background-color: #fff;
+  border-radius: 8px;
+  padding: 20px;
+}
+
+.debug-panel h3 {
+  margin: 0 0 10px 0;
+  font-size: 16px;
+  font-weight: bold;
+}
+
+#debug-log {
+  height: 200px;
+  overflow-y: auto;
+  background-color: #f8f8f8;
+  padding: 10px;
+  border-radius: 4px;
+  font-family: monospace;
+  font-size: 12px;
+  line-height: 1.4;
+}

examples/simple-chatbot/examples/prebuilt/README.md

@@ -0,0 +1,15 @@
+# Daily Prebuilt Connection
+
+The simplest way to connect to the chatbot using Daily's Prebuilt UI.
+
+1. Start the bot server
+
+```bash
+python server/server.py
+```
+
+2. Visit http://localhost:7860
+
+3. Allow microphone access when prompted
+
+4. Start talking with the bot

examples/simple-chatbot/examples/react/.gitignore

@@ -0,0 +1,24 @@
+# Logs
+logs
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+pnpm-debug.log*
+lerna-debug.log*
+
+node_modules
+dist
+dist-ssr
+*.local
+
+# Editor directories and files
+.vscode/*
+!.vscode/extensions.json
+.idea
+.DS_Store
+*.suo
+*.ntvs*
+*.njsproj
+*.sln
+*.sw?

examples/simple-chatbot/examples/react/README.md

@@ -0,0 +1,27 @@
+# React Implementation
+
+Basic implementation using the [Pipecat React SDK](https://docs.pipecat.ai/client/reference/react/introduction).
+
+## Setup
+
+1. Run the bot server; see [README](../../README).
+
+2. Navigate to the `examples/react` directory:
+
+```bash
+cd examples/react
+```
+
+3. Install dependencies:
+
+```bash
+npm install
+```
+
+4. Run the client app:
+
+```
+npm run dev
+```
+
+5. Visit http://localhost:5173 in your browser.

examples/simple-chatbot/examples/react/eslint.config.js

@@ -0,0 +1,28 @@
+import js from '@eslint/js'
+import globals from 'globals'
+import reactHooks from 'eslint-plugin-react-hooks'
+import reactRefresh from 'eslint-plugin-react-refresh'
+import tseslint from 'typescript-eslint'
+
+export default tseslint.config(
+  { ignores: ['dist'] },
+  {
+    extends: [js.configs.recommended, ...tseslint.configs.recommended],
+    files: ['**/*.{ts,tsx}'],
+    languageOptions: {
+      ecmaVersion: 2020,
+      globals: globals.browser,
+    },
+    plugins: {
+      'react-hooks': reactHooks,
+      'react-refresh': reactRefresh,
+    },
+    rules: {
+      ...reactHooks.configs.recommended.rules,
+      'react-refresh/only-export-components': [
+        'warn',
+        { allowConstantExport: true },
+      ],
+    },
+  },
+)

examples/simple-chatbot/examples/react/index.html

@@ -0,0 +1,15 @@
+<!DOCTYPE html>
+<html lang="en">
+
+<head>
+  <meta charset="UTF-8" />
+  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+  <title>Pipecat React Client</title>
+</head>
+
+<body>
+  <div id="root"></div>
+  <script type="module" src="/src/main.tsx"></script>
+</body>
+
+</html>

examples/simple-chatbot/examples/react/package.json

@@ -0,0 +1,32 @@
+{
+  "name": "react",
+  "private": true,
+  "version": "0.0.0",
+  "type": "module",
+  "scripts": {
+    "dev": "vite",
+    "build": "tsc && vite build",
+    "lint": "eslint .",
+    "preview": "vite preview"
+  },
+  "dependencies": {
+    "@daily-co/realtime-ai-daily": "^0.2.1",
+    "react": "^18.3.1",
+    "react-dom": "^18.3.1",
+    "realtime-ai": "^0.2.1",
+    "realtime-ai-react": "^0.2.1"
+  },
+  "devDependencies": {
+    "@eslint/js": "^9.15.0",
+    "@types/react": "^18.3.12",
+    "@types/react-dom": "^18.3.1",
+    "@vitejs/plugin-react": "^4.3.4",
+    "eslint": "^9.15.0",
+    "eslint-plugin-react-hooks": "^5.0.0",
+    "eslint-plugin-react-refresh": "^0.4.14",
+    "globals": "^15.12.0",
+    "typescript": "~5.6.2",
+    "typescript-eslint": "^8.15.0",
+    "vite": "^6.0.1"
+  }
+}

examples/simple-chatbot/examples/react/src/App.css

@@ -0,0 +1,82 @@
+body {
+  margin: 0;
+  padding: 20px;
+  font-family: Arial, sans-serif;
+  background-color: #f0f0f0;
+}
+
+.app {
+  max-width: 1200px;
+  margin: 0 auto;
+}
+
+.status-bar {
+  display: flex;
+  justify-content: space-between;
+  align-items: center;
+  padding: 10px;
+  background-color: #fff;
+  border-radius: 8px;
+  margin-bottom: 20px;
+}
+
+.controls button {
+  padding: 8px 16px;
+  margin-left: 10px;
+  border: none;
+  border-radius: 4px;
+  cursor: pointer;
+}
+
+button:disabled {
+  opacity: 0.5;
+  cursor: not-allowed;
+}
+
+.connect-btn {
+  background-color: #4caf50;
+  color: white;
+}
+
+.disconnect-btn {
+  background-color: #f44336;
+  color: white;
+}
+
+.main-content {
+  background-color: #fff;
+  border-radius: 8px;
+  padding: 20px;
+  margin-bottom: 20px;
+}
+
+.bot-container {
+  display: flex;
+  flex-direction: column;
+  align-items: center;
+}
+
+.video-container {
+  width: 640px;
+  height: 360px;
+  background-color: #ddd;
+  margin-bottom: 20px;
+  border-radius: 8px;
+  overflow: hidden;
+}
+
+.video-container video {
+  width: 100%;
+  height: 100%;
+  object-fit: cover;
+}
+
+.mic-enabled {
+  background-color: #4caf50;
+  color: white;
+}
+
+.mic-disabled {
+  background-color: #f44336;
+  color: white;
+}

examples/simple-chatbot/examples/react/src/App.tsx

@@ -0,0 +1,51 @@
+import {
+  RTVIClientAudio,
+  RTVIClientVideo,
+  useRTVIClientTransportState,
+} from 'realtime-ai-react';
+import { RTVIProvider } from './providers/RTVIProvider';
+import { ConnectButton } from './components/ConnectButton';
+import { StatusDisplay } from './components/StatusDisplay';
+import { DebugDisplay } from './components/DebugDisplay';
+import './App.css';
+
+function BotVideo() {
+  const transportState = useRTVIClientTransportState();
+  const isConnected = transportState !== 'disconnected';
+
+  return (
+    <div className="bot-container">
+      <div className="video-container">
+        {isConnected && <RTVIClientVideo participant="bot" fit="cover" />}
+      </div>
+    </div>
+  );
+}
+
+function AppContent() {
+  return (
+    <div className="app">
+      <div className="status-bar">
+        <StatusDisplay />
+        <ConnectButton />
+      </div>
+
+      <div className="main-content">
+        <BotVideo />
+      </div>
+
+      <DebugDisplay />
+      <RTVIClientAudio />
+    </div>
+  );
+}
+
+function App() {
+  return (
+    <RTVIProvider>
+      <AppContent />
+    </RTVIProvider>
+  );
+}
+
+export default App;

examples/simple-chatbot/examples/react/src/components/ConnectButton.tsx

@@ -0,0 +1,37 @@
+import { useRTVIClient, useRTVIClientTransportState } from 'realtime-ai-react';
+
+export function ConnectButton() {
+  const client = useRTVIClient();
+  const transportState = useRTVIClientTransportState();
+  const isConnected = ['connected', 'ready'].includes(transportState);
+
+  const handleClick = async () => {
+    if (!client) {
+      console.error('RTVI client is not initialized');
+      return;
+    }
+
+    try {
+      if (isConnected) {
+        await client.disconnect();
+      } else {
+        await client.connect();
+      }
+    } catch (error) {
+      console.error('Connection error:', error);
+    }
+  };
+
+  return (
+    <div className="controls">
+      <button
+        className={isConnected ? 'disconnect-btn' : 'connect-btn'}
+        onClick={handleClick}
+        disabled={
+          !client || ['connecting', 'disconnecting'].includes(transportState)
+        }>
+        {isConnected ? 'Disconnect' : 'Connect'}
+      </button>
+    </div>
+  );
+}

examples/simple-chatbot/examples/react/src/components/DebugDisplay.css

@@ -0,0 +1,26 @@
+.debug-panel {
+  background-color: #fff;
+  border-radius: 8px;
+  padding: 20px;
+}
+
+.debug-panel h3 {
+  margin: 0 0 10px 0;
+  font-size: 16px;
+  font-weight: bold;
+}
+
+.debug-log {
+  height: 200px;
+  overflow-y: auto;
+  background-color: #f8f8f8;
+  padding: 10px;
+  border-radius: 4px;
+  font-family: monospace;
+  font-size: 12px;
+  line-height: 1.4;
+}
+
+.debug-log div {
+  margin-bottom: 4px;
+}

examples/simple-chatbot/examples/react/src/components/DebugDisplay.tsx

@@ -0,0 +1,144 @@
+import { useRef, useCallback } from 'react';
+import {
+  Participant,
+  RTVIEvent,
+  TransportState,
+  TranscriptData,
+  BotLLMTextData,
+} from 'realtime-ai';
+import { useRTVIClient, useRTVIClientEvent } from 'realtime-ai-react';
+import './DebugDisplay.css';
+
+export function DebugDisplay() {
+  const debugLogRef = useRef<HTMLDivElement>(null);
+  const client = useRTVIClient();
+
+  const log = useCallback((message: string) => {
+    if (!debugLogRef.current) return;
+
+    const entry = document.createElement('div');
+    entry.textContent = `${new Date().toISOString()} - ${message}`;
+
+    // Add styling based on message type
+    if (message.startsWith('User: ')) {
+      entry.style.color = '#2196F3'; // blue for user
+    } else if (message.startsWith('Bot: ')) {
+      entry.style.color = '#4CAF50'; // green for bot
+    }
+
+    debugLogRef.current.appendChild(entry);
+    debugLogRef.current.scrollTop = debugLogRef.current.scrollHeight;
+  }, []);
+
+  // Log transport state changes
+  useRTVIClientEvent(
+    RTVIEvent.TransportStateChanged,
+    useCallback(
+      (state: TransportState) => {
+        log(`Transport state changed: ${state}`);
+      },
+      [log]
+    )
+  );
+
+  // Log bot connection events
+  useRTVIClientEvent(
+    RTVIEvent.BotConnected,
+    useCallback(
+      (participant?: Participant) => {
+        log(`Bot connected: ${JSON.stringify(participant)}`);
+      },
+      [log]
+    )
+  );
+
+  useRTVIClientEvent(
+    RTVIEvent.BotDisconnected,
+    useCallback(
+      (participant?: Participant) => {
+        log(`Bot disconnected: ${JSON.stringify(participant)}`);
+      },
+      [log]
+    )
+  );
+
+  // Log track events
+  useRTVIClientEvent(
+    RTVIEvent.TrackStarted,
+    useCallback(
+      (track: MediaStreamTrack, participant?: Participant) => {
+        log(
+          `Track started: ${track.kind} from ${participant?.name || 'unknown'}`
+        );
+      },
+      [log]
+    )
+  );
+
+  useRTVIClientEvent(
+    RTVIEvent.TrackedStopped,
+    useCallback(
+      (track: MediaStreamTrack, participant?: Participant) => {
+        log(
+          `Track stopped: ${track.kind} from ${participant?.name || 'unknown'}`
+        );
+      },
+      [log]
+    )
+  );
+
+  // Log bot ready state and check tracks
+  useRTVIClientEvent(
+    RTVIEvent.BotReady,
+    useCallback(() => {
+      log(`Bot ready`);
+
+      if (!client) return;
+
+      const tracks = client.tracks();
+      log(
+        `Available tracks: ${JSON.stringify({
+          local: {
+            audio: !!tracks.local.audio,
+            video: !!tracks.local.video,
+          },
+          bot: {
+            audio: !!tracks.bot?.audio,
+            video: !!tracks.bot?.video,
+          },
+        })}`
+      );
+    }, [client, log])
+  );
+
+  // Log transcripts
+  useRTVIClientEvent(
+    RTVIEvent.UserTranscript,
+    useCallback(
+      (data: TranscriptData) => {
+        // Only log final transcripts
+        if (data.final) {
+          log(`User: ${data.text}`);
+        }
+      },
+      [log]
+    )
+  );
+
+  useRTVIClientEvent(
+    RTVIEvent.BotTranscript,
+    useCallback(
+      (data: BotLLMTextData) => {
+        log(`Bot: ${data.text}`);
+      },
+      [log]
+    )
+  );
+
+  return (
+    <div className="debug-panel">
+      <h3>Debug Info</h3>
+      <div ref={debugLogRef} className="debug-log" />
+    </div>
+  );
+}

examples/simple-chatbot/examples/react/src/components/StatusDisplay.tsx

@@ -0,0 +1,11 @@
+import { useRTVIClientTransportState } from 'realtime-ai-react';
+
+export function StatusDisplay() {
+  const transportState = useRTVIClientTransportState();
+
+  return (
+    <div className="status">
+      Status: <span>{transportState}</span>
+    </div>
+  );
+}

examples/simple-chatbot/examples/react/src/main.tsx

@@ -0,0 +1,9 @@
+import React from 'react';
+import ReactDOM from 'react-dom/client';
+import App from './App';
+
+ReactDOM.createRoot(document.getElementById('root')!).render(
+  <React.StrictMode>
+    <App />
+  </React.StrictMode>
+);

examples/simple-chatbot/examples/react/src/providers/RTVIProvider.tsx

@@ -0,0 +1,22 @@
+import { type PropsWithChildren } from 'react';
+import { RTVIClient } from 'realtime-ai';
+import { DailyTransport } from '@daily-co/realtime-ai-daily';
+import { RTVIClientProvider } from 'realtime-ai-react';
+
+const transport = new DailyTransport();
+
+const client = new RTVIClient({
+  transport,
+  params: {
+    baseUrl: 'http://localhost:7860',
+    endpoints: {
+      connect: '/connect',
+    },
+  },
+  enableMic: true,
+  enableCam: false,
+});
+
+export function RTVIProvider({ children }: PropsWithChildren) {
+  return <RTVIClientProvider client={client}>{children}</RTVIClientProvider>;
+}

examples/simple-chatbot/examples/react/tsconfig.json

@@ -0,0 +1,25 @@
+{
+  "compilerOptions": {
+    "target": "ES2020",
+    "useDefineForClassFields": true,
+    "lib": ["ES2020", "DOM", "DOM.Iterable"],
+    "module": "ESNext",
+    "skipLibCheck": true,
+
+    /* Bundler mode */
+    "moduleResolution": "bundler",
+    "allowImportingTsExtensions": true,
+    "resolveJsonModule": true,
+    "isolatedModules": true,
+    "noEmit": true,
+    "jsx": "react-jsx",
+
+    /* Linting */
+    "strict": true,
+    "noUnusedLocals": true,
+    "noUnusedParameters": true,
+    "noFallthroughCasesInSwitch": true
+  },
+  "include": ["src"],
+  "references": [{ "path": "./tsconfig.node.json" }]
+}

examples/simple-chatbot/examples/react/tsconfig.node.json

@@ -0,0 +1,10 @@
+{
+  "compilerOptions": {
+    "composite": true,
+    "skipLibCheck": true,
+    "module": "ESNext",
+    "moduleResolution": "bundler",
+    "allowSyntheticDefaultImports": true
+  },
+  "include": ["vite.config.ts"]
+}

examples/simple-chatbot/examples/react/vite.config.ts

@@ -0,0 +1,7 @@
+import { defineConfig } from 'vite'
+import react from '@vitejs/plugin-react'
+
+// https://vite.dev/config/
+export default defineConfig({
+  plugins: [react()],
+})

examples/simple-chatbot/requirements.txt

@@ -1,4 +0,0 @@
-python-dotenv
-fastapi[all]
-uvicorn
-pipecat-ai[daily,elevenlabs,openai,silero]

examples/simple-chatbot/server.py

@@ -1,141 +0,0 @@
-#
-# Copyright (c) 2024, Daily
-#
-# SPDX-License-Identifier: BSD 2-Clause License
-#
-
-import aiohttp
-import os
-import argparse
-import subprocess
-
-from contextlib import asynccontextmanager
-
-from fastapi import FastAPI, Request, HTTPException
-from fastapi.middleware.cors import CORSMiddleware
-from fastapi.responses import JSONResponse, RedirectResponse
-
-from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper, DailyRoomParams
-
-from dotenv import load_dotenv
-
-load_dotenv(override=True)
-
-MAX_BOTS_PER_ROOM = 1
-
-# Bot sub-process dict for status reporting and concurrency control
-bot_procs = {}
-
-daily_helpers = {}
-
-
-def cleanup():
-    # Clean up function, just to be extra safe
-    for entry in bot_procs.values():
-        proc = entry[0]
-        proc.terminate()
-        proc.wait()
-
-
-@asynccontextmanager
-async def lifespan(app: FastAPI):
-    aiohttp_session = aiohttp.ClientSession()
-    daily_helpers["rest"] = DailyRESTHelper(
-        daily_api_key=os.getenv("DAILY_API_KEY", ""),
-        daily_api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
-        aiohttp_session=aiohttp_session,
-    )
-    yield
-    await aiohttp_session.close()
-    cleanup()
-
-
-app = FastAPI(lifespan=lifespan)
-
-app.add_middleware(
-    CORSMiddleware,
-    allow_origins=["*"],
-    allow_credentials=True,
-    allow_methods=["*"],
-    allow_headers=["*"],
-)
-
-
-@app.get("/")
-async def start_agent(request: Request):
-    print(f"!!! Creating room")
-    room = await daily_helpers["rest"].create_room(DailyRoomParams())
-    print(f"!!! Room URL: {room.url}")
-    # Ensure the room property is present
-    if not room.url:
-        raise HTTPException(
-            status_code=500,
-            detail="Missing 'room' property in request data. Cannot start agent without a target room!",
-        )
-
-    # Check if there is already an existing process running in this room
-    num_bots_in_room = sum(
-        1 for proc in bot_procs.values() if proc[1] == room.url and proc[0].poll() is None
-    )
-    if num_bots_in_room >= MAX_BOTS_PER_ROOM:
-        raise HTTPException(status_code=500, detail=f"Max bot limited reach for room: {room.url}")
-
-    # Get the token for the room
-    token = await daily_helpers["rest"].get_token(room.url)
-
-    if not token:
-        raise HTTPException(status_code=500, detail=f"Failed to get token for room: {room.url}")
-
-    # Spawn a new agent, and join the user session
-    # Note: this is mostly for demonstration purposes (refer to 'deployment' in README)
-    try:
-        proc = subprocess.Popen(
-            [f"python3 -m bot -u {room.url} -t {token}"],
-            shell=True,
-            bufsize=1,
-            cwd=os.path.dirname(os.path.abspath(__file__)),
-        )
-        bot_procs[proc.pid] = (proc, room.url)
-    except Exception as e:
-        raise HTTPException(status_code=500, detail=f"Failed to start subprocess: {e}")
-
-    return RedirectResponse(room.url)
-
-
-@app.get("/status/{pid}")
-def get_status(pid: int):
-    # Look up the subprocess
-    proc = bot_procs.get(pid)
-
-    # If the subprocess doesn't exist, return an error
-    if not proc:
-        raise HTTPException(status_code=404, detail=f"Bot with process id: {pid} not found")
-
-    # Check the status of the subprocess
-    if proc[0].poll() is None:
-        status = "running"
-    else:
-        status = "finished"
-
-    return JSONResponse({"bot_id": pid, "status": status})
-
-
-if __name__ == "__main__":
-    import uvicorn
-
-    default_host = os.getenv("HOST", "0.0.0.0")
-    default_port = int(os.getenv("FAST_API_PORT", "7860"))
-
-    parser = argparse.ArgumentParser(description="Daily Storyteller FastAPI server")
-    parser.add_argument("--host", type=str, default=default_host, help="Host address")
-    parser.add_argument("--port", type=int, default=default_port, help="Port number")
-    parser.add_argument("--reload", action="store_true", help="Reload code on change")
-
-    config = parser.parse_args()
-
-    uvicorn.run(
-        "server:app",
-        host=config.host,
-        port=config.port,
-        reload=config.reload,
-    )

examples/simple-chatbot/server/README.md

@@ -0,0 +1,66 @@
+# Simple Chatbot Server
+
+A FastAPI server that manages bot instances and provides endpoints for both Daily Prebuilt and Pipecat client connections.
+
+## Endpoints
+
+- `GET /` - Direct browser access, redirects to a Daily Prebuilt room
+- `POST /connect` - Pipecat client connection endpoint
+- `GET /status/{pid}` - Get status of a specific bot process
+
+## Environment Variables
+
+Copy `env.example` to `.env` and configure:
+
+```ini
+# Required API Keys
+DAILY_API_KEY=           # Your Daily API key
+OPENAI_API_KEY=          # Your OpenAI API key (required for OpenAI bot)
+GEMINI_API_KEY=          # Your Gemini API key (required for Gemini bot)
+ELEVENLABS_API_KEY=      # Your ElevenLabs API key
+
+# Bot Selection
+BOT_IMPLEMENTATION=      # Options: 'openai' or 'gemini'
+
+# Optional Configuration
+DAILY_API_URL=           # Optional: Daily API URL (defaults to https://api.daily.co/v1)
+DAILY_SAMPLE_ROOM_URL=   # Optional: Fixed room URL for development
+HOST=                    # Optional: Host address (defaults to 0.0.0.0)
+FAST_API_PORT=           # Optional: Port number (defaults to 7860)
+```
+
+## Available Bots
+
+The server supports two bot implementations:
+
+1. **OpenAI Bot** (Default)
+
+   - Uses GPT-4 for conversation
+   - Requires OPENAI_API_KEY
+
+2. **Gemini Bot**
+   - Uses Google's Gemini model
+   - Requires GEMINI_API_KEY
+
+Select your preferred bot by setting `BOT_IMPLEMENTATION` in your `.env` file.
+
+## Running the Server
+
+Set up and activate your virtual environment:
+
+```bash
+python3 -m venv venv
+source venv/bin/activate  # On Windows: venv\Scripts\activate
+```
+
+Install dependencies:
+
+```bash
+pip install -r requirements.txt
+```
+
+Run the server:
+
+```bash
+python server.py
+```