72d0fb418a
Before the new async-tool mechanism landed, AWSNovaSonicLLMService and OpenAIRealtimeLLMService honored cancel_on_interruption=False by simply not cancelling in-flight function calls on interruption — the eventual result then flowed through the same channel as any synchronous tool result. The new mechanism (which appends started/intermediate/final messages to the LLM context as the underlying task progresses) broke that path: the realtime services didn't know how to interpret those messages, and the eventual result was never delivered to the provider. Restore the flag's behavior by teaching both services to detect async-tool messages in the context and route them appropriately: - started → skipped silently. The provider already issued the tool call and natively awaits a result; nothing to send for the started marker. - final → delivered via the formal tool-result channel. Same path as a synchronous tool result, just delayed. Streamed intermediate results (FunctionCallResultProperties(is_final= False)) are not supported on these realtime services. An intermediate result is logged as an error and surfaced via push_error, then dropped. Use a non-realtime LLM service if a tool needs to stream intermediate results. (Docstrings on register_function, register_direct_function, and FunctionCallResultProperties.is_final updated to call this out.) A new shared module pipecat.processors.aggregators.async_tool_messages is the single source of truth for the on-the-wire payload shape: the aggregator uses its build_*_message functions when injecting messages, and the realtime services use parse_message when scanning the context. Adds two example files exercising a network-delayed weather tool with each service. The plain realtime-aws-nova-sonic.py example is also reverted to a synchronous tool call now that the async variant lives in its own file. Similar fixes for other realtime services are forthcoming.
Pipecat Examples
This directory contains examples showing how to build voice and multimodal agents with Pipecat.
Setup
-
Follow the README steps to get your local environment configured.
Run from root directory: Make sure you are running the steps from the root directory.
Using local audio?: The
LocalAudioTransportrequires a system dependency forportaudio. Install the dependency to use the transport. -
Copy the
env.examplefile and add API keys for services you plan to use:cp env.example .env # Edit .env with your API keys -
Run any example:
uv run python getting-started/01-say-one-thing.py -
Open the web interface at http://localhost:7860/client/ and click "Connect"
Running examples with other transports
Most examples support running with other transports, like Twilio or Daily.
Daily
You need to create a Daily account at https://dashboard.daily.co/u/signup. Once signed up, you can create your own room from the dashboard and set the environment variables DAILY_ROOM_URL and DAILY_API_KEY. Alternatively, you can let the example create a room for you (still needs DAILY_API_KEY environment variable). Then, start any example with -t daily:
uv run getting-started/06-voice-agent.py -t daily
Twilio
It is also possible to run the example through a Twilio phone number. You will need to setup a few things:
- Install and run ngrok.
ngrok http 7860
- Configure your Twilio phone number. One way is to setup a TwiML app and set the request URL to the ngrok URL from step (1). Then, set your phone number to use the new TwiML app.
Then, run the example with:
uv run getting-started/06-voice-agent.py -t twilio -x NGROK_HOST_NAME
Directory Structure
getting-started/
Progressive introduction to Pipecat, from minimal TTS to a full voice agent with function calling.
voice/
Full STT + LLM + TTS voice agent pipelines showcasing different speech service providers (Deepgram, ElevenLabs, Cartesia, etc.)
function-calling/
Function calling with different LLM providers (OpenAI, Anthropic, Google, etc.)
transcription/
Speech-to-text examples with various STT providers.
vision/
Image description and vision capabilities with different multimodal LLMs.
realtime/
Realtime and multimodal live APIs (OpenAI Realtime, Gemini Live, AWS Nova Sonic, Ultravox, Grok).
persistent-context/
Maintaining conversation context across sessions with different providers.
context-summarization/
Summarizing conversation context to manage token limits.
update-settings/
Changing service settings at runtime, organized by service type:
turn-management/
Turn detection, interruption handling, and user input management.
thinking-and-mcp/
LLM thinking/reasoning modes and MCP (Model Context Protocol) tool server integration.
transports/
Transport layer examples (WebRTC, Daily, LiveKit).
video-avatar/
Video avatar integrations (Tavus, HeyGen, Simli, LemonSlice).
video-processing/
Video processing, mirroring, GStreamer, and custom video tracks.
audio/
Audio recording, background sounds, and sound effects.
observability/
Pipeline monitoring: observers, heartbeats, and Sentry metrics.
rag/
Retrieval-augmented generation, grounding, and long-term memory (Mem0, Gemini).
features/
Miscellaneous features: wake phrases, live translation, service switching, voice switching, and more.
Advanced Usage
Customizing Network Settings
uv run python <example-name> --host 0.0.0.0 --port 8080
Troubleshooting
- No audio/video: Check browser permissions for microphone and camera
- Connection errors: Verify API keys in
.envfile - Port conflicts: Use
--portto change the port
For more examples, visit the pipecat-examples repository.