d8f5c0be71
Adds XAITTSService in the existing xai/tts.py module, alongside the existing XAIHttpTTSService. Connects to xAI's streaming endpoint at wss://api.x.ai/v1/tts, streams text.delta chunks up and base64 audio.delta chunks down on the same connection so audio starts flowing before the full utterance is synthesized. Extends InterruptibleTTSService since xAI's protocol is strictly sequential per connection and exposes neither a cancel verb nor a context ID — the only way to stop an in-flight utterance is to tear down the WebSocket, which is exactly what InterruptibleTTSService does on interruption when the bot is speaking. Voice, language, codec, and sample_rate are passed as query-string params at connect time; runtime setting changes reconnect the socket. Defaults to raw PCM so emitted TTSAudioRawFrame objects need no decoding downstream. Splits the existing example into voice-xai.py (WebSocket) and voice-xai-http.py (batch HTTP) so each variant has its own entry point. Promotes the xai extra to depend on pipecat-ai[websockets-base] since the new service imports the websockets library.
Pipecat Examples
This directory contains examples showing how to build voice and multimodal agents with Pipecat.
Setup
-
Follow the README steps to get your local environment configured.
Run from root directory: Make sure you are running the steps from the root directory.
Using local audio?: The
LocalAudioTransportrequires a system dependency forportaudio. Install the dependency to use the transport. -
Copy the
env.examplefile and add API keys for services you plan to use:cp env.example .env # Edit .env with your API keys -
Run any example:
uv run python getting-started/01-say-one-thing.py -
Open the web interface at http://localhost:7860/client/ and click "Connect"
Running examples with other transports
Most examples support running with other transports, like Twilio or Daily.
Daily
You need to create a Daily account at https://dashboard.daily.co/u/signup. Once signed up, you can create your own room from the dashboard and set the environment variables DAILY_ROOM_URL and DAILY_API_KEY. Alternatively, you can let the example create a room for you (still needs DAILY_API_KEY environment variable). Then, start any example with -t daily:
uv run getting-started/06-voice-agent.py -t daily
Twilio
It is also possible to run the example through a Twilio phone number. You will need to setup a few things:
- Install and run ngrok.
ngrok http 7860
- Configure your Twilio phone number. One way is to setup a TwiML app and set the request URL to the ngrok URL from step (1). Then, set your phone number to use the new TwiML app.
Then, run the example with:
uv run getting-started/06-voice-agent.py -t twilio -x NGROK_HOST_NAME
Directory Structure
getting-started/
Progressive introduction to Pipecat, from minimal TTS to a full voice agent with function calling.
voice/
Full STT + LLM + TTS voice agent pipelines showcasing different speech service providers (Deepgram, ElevenLabs, Cartesia, etc.)
function-calling/
Function calling with different LLM providers (OpenAI, Anthropic, Google, etc.)
transcription/
Speech-to-text examples with various STT providers.
vision/
Image description and vision capabilities with different multimodal LLMs.
realtime/
Realtime and multimodal live APIs (OpenAI Realtime, Gemini Live, AWS Nova Sonic, Ultravox, Grok).
persistent-context/
Maintaining conversation context across sessions with different providers.
context-summarization/
Summarizing conversation context to manage token limits.
update-settings/
Changing service settings at runtime, organized by service type:
turn-management/
Turn detection, interruption handling, and user input management.
thinking-and-mcp/
LLM thinking/reasoning modes and MCP (Model Context Protocol) tool server integration.
transports/
Transport layer examples (WebRTC, Daily, LiveKit).
video-avatar/
Video avatar integrations (Tavus, HeyGen, Simli, LemonSlice).
video-processing/
Video processing, mirroring, GStreamer, and custom video tracks.
audio/
Audio recording, background sounds, and sound effects.
observability/
Pipeline monitoring: observers, heartbeats, and Sentry metrics.
rag/
Retrieval-augmented generation, grounding, and long-term memory (Mem0, Gemini).
features/
Miscellaneous features: wake phrases, live translation, service switching, voice switching, and more.
Advanced Usage
Customizing Network Settings
uv run python <example-name> --host 0.0.0.0 --port 8080
Troubleshooting
- No audio/video: Check browser permissions for microphone and camera
- Connection errors: Verify API keys in
.envfile - Port conflicts: Use
--portto change the port
For more examples, visit the pipecat-examples repository.