7596d71460
* initial config * skeleton * Added a README (to be added to). * Payloads coming from the ASR. * doc update * handle the partials and finals * enable diarization in the example * support sending messages to pipecat pipeline * requirements fix in README * updated example (with amusement) * updated example to match master * updated docs * support for diarization tags * logic fix for wrapper * Use an internal SpeechFrame for speaker_id (not user_id). * only include speaker tags on finalised transcript (as this may skew end of utterance detection) * updated docs * correction to docs and updated example * updated requirement * Fix for using default EU server. * Updates from PR comments. * Refactor based on comments in the original PR. Primary focus on documentation, naming conventions and how `user_id` is used. * Check for SMX installed when importing. * Variable name change * Comment correction. * Support for Esporanto and Uyghur * Impoved language support * function name change * Locale fix * intercept * interim changes * pass the pipeline task to the module for adding events to the top of the pipeline * logging for the pipeline * Reduce timeout for content aggregator. * staged update * testing with Azure * Updated context (Azure was dropping punctuation) and using better ElevenLabs model. * Updated to RT 0.3.0 and use OpenAI (not Azure). * Missing OpenAI import; parameter name change for output locale validation. * Revert to `0.2.0` of RT SDK. * fix for assignment of `output_locale_code`. * update Speechmatics library to 0.3.1 * new transcription example * updated asyncio task handling * Updated doc strings * enable OpenTelemetry logging * removed import from stt for __init__ * updated examples and default values * updated examples * prevent lock up when closing the STT connection
Pipecat Foundational Examples
This directory contains examples showing how to build voice and multimodal agents with Pipecat. Each example demonstrates specific features, progressing from basic to advanced concepts.
Learning Paths
Depending on what you're trying to build, these learning paths will guide you through relevant examples:
- New to Pipecat: Start with examples 01, 02, 07
- Building conversational bots: 07, 10, 38
- Common add-on capabilities: 17, 24, 28, 34
- Adding visual capabilities: 03, 12, 26
- Advanced agent capabilities: 14, 20, 37
Quick Start
-
Set up a virtual environment:
python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate -
Install dependencies:
pip install -r requirements.txt -
Create a
.envfile with your API keys. -
Run any example:
python 01-say-one-thing.py -
Open the web interface at http://localhost:7860 and click "Connect"
Running examples with other transports
It is possible to run most of the examples with other transports such as Twilio or Daily.
Daily
You need to create a Daily account at https://dashboard.daily.co/u/signup. Once signed up, you can create your own room from the dashboard and set the environment variables DAILY_SAMPLE_ROOM_URL and DAILY_API_KEY. Alternatively, you can let the example create a room for you (still needs DAILY_API_KEY environment variable). Then, start any example with -t daily:
python 07-interruptible.py -t daily
Twilio
It is also possible to run the example through a Twilio phone number. You will need to setup a few things:
- Install and run ngrok.
ngrok http 7860
- Configure your Twilio phone number. One way is to setup a TwiML app and set the request URL to the ngrok URL from step (1). Then, set your phone number to use the new TwiML app.
Then, run the example with:
python 07-interruptible.py -t twilio -x NGROK_HOST_NAME (no protocol)
Examples by Feature
Basics
- 01-say-one-thing.py: Most basic bot that says one phrase and exits (Transport, TTS, Event handlers)
- 02-llm-say-one-thing.py: Bot generates a response with an LLM (LLM initialization)
- 03-still-frame.py: Displays a static image (Video transport, Image service)
- 04-transport.py: Different transport options (WebRTC, Daily, Livekit)
Conversational AI
- 07-interruptible.py: Basic voice assistant bot (STT, TTS, LLM, Interruptible speech)
- 10-wake-phrase.py: Bot activated by wake phrase (WakeCheckFilter)
- 22-natural-conversation.py: Smart turn detection (Multiple LLMs, Turn management)
- 38-smart-turn-fal.py: ML-based turn detection (Fal service, Local models)
Common Utilities
- 17-detect-user-idle.py: Handle inactive users (UserIdleProcessor)
- 24-stt-mute-filter.py: Selectively mute user input (STTMuteFilter)
- 28-transcription-processor.py: Record conversation text (TranscriptProcessor)
- 30-observer.py: Access frame data (Custom observers)
- 31-heartbeats.py: Detect idle pipelines (Pipeline monitoring)
- 34-audio-recording.py: Record conversation audio (Composite and track-level recording)
Advanced LLM Features
- 14-function-calling.py: Bot with tool usage (Function schemas, Tool registration)
- 20a-persistent-context-openai.py: Persistent conversation context (Memory management)
- 32-gemini-grounding-metadata.py: Web search capabilities (Google search integration)
- 33-gemini-rag.py: Retrieval-augmented generation (Data sources, Grounding)
- 37-mem0.py: Long-term agent memory (Mem0 service integration)
Media Handling
- 05-sync-speech-and-images.py: Synchronized narration with images (Custom processors, SyncParallelPipeline)
- 06a-image-sync.py: Dynamic image updates while speaking (Synchronized A/V pipelines)
- 09-mirror.py: Mirror user's audio and video (Custom frame processors)
- 11-sound-effects.py: Add sounds when bot speaks (Sound playback, Event synchronization)
- 23-bot-background-sound.py: Play background audio (SoundfileMixer)
Vision & Multimodal
- 12a-describe-video-gemini-flash.py: Bot describes user's video (Video input, Multimodal LLMs)
- 26c-gemini-multimodal-live-video.py: Gemini with video input (Streaming video, Function calls)
Voice & Language
- 13-transcription.py: Speech transcription demo (STT providers, Real-time transcription)
- 15-switch-voices.py: Dynamic voice/language changing (ParallelPipelines, FunctionFilters)
- 25-google-audio-in.py: Gemini for speech recognition (Alternative transcription)
- 35-pattern-pair-voice-switching.py: Dynamic TTS voice switching (XML parsing, PatternPairAggregator)
- 36-user-email-gathering.py: Spelling mode for TTS (Confirmation patterns, XML tags)
Integration Examples
- 18-gstreamer-filesrc.py: GStreamer video streaming (Video processing)
- 19-openai-realtime-beta.py: OpenAI Speech-to-Speech (Direct S2S, Function calls)
- 21-tavus-layer-tavus-transport.py: Tavus digital twin (Avatar integration)
- 27-simli-layer.py: Simli avatar integration (Video synchronization)
Performance & Optimization
- 16-gpu-container-local-bot.py: GPU-accelerated local bot (Performance measurement)
Utilities
Advanced Usage
Customizing Network Settings
python <example-name> --host 0.0.0.0 --port 8080
Troubleshooting
- No audio/video: Check browser permissions for microphone and camera
- Connection errors: Verify API keys in
.envfile - Missing dependencies: Run
pip install -r requirements.txt - Port conflicts: Use
--portto change the port
For more examples, visit our GitHub repository.