diff --git a/README.md b/README.md index f92178e5d..783015fb8 100644 --- a/README.md +++ b/README.md @@ -1,43 +1,72 @@

-ย pipecat + pipecat

[![PyPI](https://img.shields.io/pypi/v/pipecat-ai)](https://pypi.org/project/pipecat-ai) ![Tests](https://github.com/pipecat-ai/pipecat/actions/workflows/tests.yaml/badge.svg) [![codecov](https://codecov.io/gh/pipecat-ai/pipecat/graph/badge.svg?token=LNVUIVO4Y9)](https://codecov.io/gh/pipecat-ai/pipecat) [![Docs](https://img.shields.io/badge/Documentation-blue)](https://docs.pipecat.ai) [![Discord](https://img.shields.io/discord/1239284677165056021)](https://discord.gg/pipecat) -Pipecat is an open source Python framework for building voice and multimodal conversational agents. It handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions, letting you focus on creating engaging experiences. +# ๐ŸŽ™๏ธ Pipecat: Real-Time Voice & Multimodal AI Agents -## What you can build +**Pipecat** is an open-source Python framework for building real-time voice and multimodal conversational agents. Orchestrate audio and video, AI services, different transports, and conversation pipelines effortlesslyโ€”so you can focus on what makes your agent unique. -- **Voice Assistants**: [Natural, real-time conversations with AI](https://demo.dailybots.ai/) -- **Interactive Agents**: Personal coaches and meeting assistants -- **Multimodal Apps**: Combine voice, video, images, and text -- **Creative Tools**: [Story-telling experiences](https://storytelling-chatbot.fly.dev/) and social companions -- **Business Solutions**: [Customer intake flows](https://www.youtube.com/watch?v=lDevgsp9vn0) and support bots -- **Complex conversational flows**: [Refer to Pipecat Flows](https://github.com/pipecat-ai/pipecat-flows) to learn more +## ๐Ÿš€ What You Can Build -## See it in action +- **Voice Assistants** โ€“ natural, streaming conversations with AI +- **AI Companions** โ€“ coaches, meeting assistants, characters +- **Multimodal Interfaces** โ€“ voice, video, images, and more +- **Interactive Storytelling** โ€“ creative tools with generative media +- **Business Agents** โ€“ customer intake, support bots, guided flows +- **Complex Dialog Systems** โ€“ design logic with structured conversations + +๐Ÿงญ Looking to build structured conversations? Check out [Pipecat Flows](https://github.com/pipecat-ai/pipecat-flows) for managing complex conversational states and transitions. + +## ๐Ÿง  Why Pipecat? + +- **Voice-first**: Integrates speech recognition, text-to-speech, and conversation handling +- **Pluggable**: Supports many AI services and tools +- **Composable Pipelines**: Build complex behavior from modular components +- **Real-Time**: Ultra-low latency interaction with different transports (e.g. WebSockets or WebRTC) + +## ๐ŸŽฌ See it in action

-   - +   +
-   - +   +

-## Key features +## ๐Ÿ“ฑ Client SDKs -- **Voice-first Design**: Built-in speech recognition, TTS, and conversation handling -- **Flexible Integration**: Works with popular AI services (OpenAI, ElevenLabs, etc.) -- **Pipeline Architecture**: Build complex apps from simple, reusable components -- **Real-time Processing**: Frame-based pipeline architecture for fluid interactions -- **Production Ready**: Enterprise-grade WebRTC and Websocket support +You can connect to Pipecat from any platform using our official SDKs: -๐Ÿ’ก Looking to build structured conversations? Check out [Pipecat Flows](https://github.com/pipecat-ai/pipecat-flows) for managing complex conversational states and transitions. +| Platform | SDK Repo | Description | +| -------- | ------------------------------------------------------------------------------ | -------------------------------- | +| Web | [pipecat-client-web](https://github.com/pipecat-ai/pipecat-client-web) | JavaScript and React client SDKs | +| iOS | [pipecat-client-ios](https://github.com/pipecat-ai/pipecat-client-ios) | Swift SDK for iOS | +| Android | [pipecat-client-android](https://github.com/pipecat-ai/pipecat-client-android) | Kotlin SDK for Android | +| C++ | [pipecat-client-cxx](https://github.com/pipecat-ai/pipecat-client-cxx) | C++ client SDK | -## Getting started +## ๐Ÿงฉ Available services -You can get started with Pipecat running on your local machine, then move your agent processes to the cloud when youโ€™re ready. You can also add a ๐Ÿ“ž telephone number, ๐Ÿ–ผ๏ธ image output, ๐Ÿ“บ video input, use different LLMs, and more. +| Category | Services | +| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Speech-to-Text | [AssemblyAI](https://docs.pipecat.ai/server/services/stt/assemblyai), [Azure](https://docs.pipecat.ai/server/services/stt/azure), [Deepgram](https://docs.pipecat.ai/server/services/stt/deepgram), [Fal Wizper](https://docs.pipecat.ai/server/services/stt/fal), [Gladia](https://docs.pipecat.ai/server/services/stt/gladia), [Google](https://docs.pipecat.ai/server/services/stt/google), [Groq (Whisper)](https://docs.pipecat.ai/server/services/stt/groq), [OpenAI (Whisper)](https://docs.pipecat.ai/server/services/stt/openai), [Parakeet (NVIDIA)](https://docs.pipecat.ai/server/services/stt/parakeet), [Ultravox](https://docs.pipecat.ai/server/services/stt/ultravox), [Whisper](https://docs.pipecat.ai/server/services/stt/whisper) | +| LLMs | [Anthropic](https://docs.pipecat.ai/server/services/llm/anthropic), [Azure](https://docs.pipecat.ai/server/services/llm/azure), [Cerebras](https://docs.pipecat.ai/server/services/llm/cerebras), [DeepSeek](https://docs.pipecat.ai/server/services/llm/deepseek), [Fireworks AI](https://docs.pipecat.ai/server/services/llm/fireworks), [Gemini](https://docs.pipecat.ai/server/services/llm/gemini), [Grok](https://docs.pipecat.ai/server/services/llm/grok), [Groq](https://docs.pipecat.ai/server/services/llm/groq), [NVIDIA NIM](https://docs.pipecat.ai/server/services/llm/nim), [Ollama](https://docs.pipecat.ai/server/services/llm/ollama), [OpenAI](https://docs.pipecat.ai/server/services/llm/openai), [OpenRouter](https://docs.pipecat.ai/server/services/llm/openrouter), [Perplexity](https://docs.pipecat.ai/server/services/llm/perplexity), [Qwen](https://docs.pipecat.ai/server/services/llm/qwen), [Together AI](https://docs.pipecat.ai/server/services/llm/together) | +| Text-to-Speech | [AWS](https://docs.pipecat.ai/server/services/tts/aws), [Azure](https://docs.pipecat.ai/server/services/tts/azure), [Cartesia](https://docs.pipecat.ai/server/services/tts/cartesia), [Deepgram](https://docs.pipecat.ai/server/services/tts/deepgram), [ElevenLabs](https://docs.pipecat.ai/server/services/tts/elevenlabs), [FastPitch (NVIDIA)](https://docs.pipecat.ai/server/services/tts/fastpitch), [Fish](https://docs.pipecat.ai/server/services/tts/fish), [Google](https://docs.pipecat.ai/server/services/tts/google), [LMNT](https://docs.pipecat.ai/server/services/tts/lmnt), [Neuphonic](https://docs.pipecat.ai/server/services/tts/neuphonic), [OpenAI](https://docs.pipecat.ai/server/services/tts/openai), [Piper](https://docs.pipecat.ai/server/services/tts/piper), [PlayHT](https://docs.pipecat.ai/server/services/tts/playht), [Rime](https://docs.pipecat.ai/server/services/tts/rime), [XTTS](https://docs.pipecat.ai/server/services/tts/xtts) | +| Speech-to-Speech | [Gemini Multimodal Live](https://docs.pipecat.ai/server/services/s2s/gemini), [OpenAI Realtime](https://docs.pipecat.ai/server/services/s2s/openai) | +| Transport | [Daily (WebRTC)](https://docs.pipecat.ai/server/services/transport/daily), [FastAPI Websocket](https://docs.pipecat.ai/server/services/transport/fastapi-websocket), [SmallWebRTCTransport](https://docs.pipecat.ai/server/services/transport/small-webrtc), [WebSocket Server](https://docs.pipecat.ai/server/services/transport/websocket-server), Local | +| Video | [Tavus](https://docs.pipecat.ai/server/services/video/tavus), [Simli](https://docs.pipecat.ai/server/services/video/simli) | +| Memory | [mem0](https://docs.pipecat.ai/server/services/memory/mem0) | +| Vision & Image | [fal](https://docs.pipecat.ai/server/services/image-generation/fal), [Google Imagen](https://docs.pipecat.ai/server/services/image-generation/fal), [Moondream](https://docs.pipecat.ai/server/services/vision/moondream) | +| Audio Processing | [Silero VAD](https://docs.pipecat.ai/server/utilities/audio/silero-vad-analyzer), [Krisp](https://docs.pipecat.ai/server/utilities/audio/krisp-filter), [Koala](https://docs.pipecat.ai/server/utilities/audio/koala-filter), [Noisereduce](https://docs.pipecat.ai/server/utilities/audio/noisereduce-filter) | +| Analytics & Metrics | [Canonical AI](https://docs.pipecat.ai/server/services/analytics/canonical), [Sentry](https://docs.pipecat.ai/server/services/analytics/sentry) | + +๐Ÿ“š [View full services documentation โ†’](https://docs.pipecat.ai/server/services/supported-services) + +## โšก Getting started + +You can get started with Pipecat running on your local machine, then move your agent processes to the cloud when youโ€™re ready. ```shell # Install the module @@ -53,141 +82,51 @@ To keep things lightweight, only the core framework is included by default. If y pip install "pipecat-ai[option,...]" ``` -### Available services - -| Category | Services | Install Command Example | -| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | -| Speech-to-Text | [AssemblyAI](https://docs.pipecat.ai/server/services/stt/assemblyai), [Azure](https://docs.pipecat.ai/server/services/stt/azure), [Deepgram](https://docs.pipecat.ai/server/services/stt/deepgram), [Fal Wizper](https://docs.pipecat.ai/server/services/stt/fal), [Gladia](https://docs.pipecat.ai/server/services/stt/gladia), [Google](https://docs.pipecat.ai/server/services/stt/google), [Groq (Whisper)](https://docs.pipecat.ai/server/services/stt/groq), [OpenAI (Whisper)](https://docs.pipecat.ai/server/services/stt/openai), [Parakeet (NVIDIA)](https://docs.pipecat.ai/server/services/stt/parakeet), [Ultravox](https://docs.pipecat.ai/server/services/stt/ultravox), [Whisper](https://docs.pipecat.ai/server/services/stt/whisper) | `pip install "pipecat-ai[deepgram]"` | -| LLMs | [Anthropic](https://docs.pipecat.ai/server/services/llm/anthropic), [Azure](https://docs.pipecat.ai/server/services/llm/azure), [Cerebras](https://docs.pipecat.ai/server/services/llm/cerebras), [DeepSeek](https://docs.pipecat.ai/server/services/llm/deepseek), [Fireworks AI](https://docs.pipecat.ai/server/services/llm/fireworks), [Gemini](https://docs.pipecat.ai/server/services/llm/gemini), [Grok](https://docs.pipecat.ai/server/services/llm/grok), [Groq](https://docs.pipecat.ai/server/services/llm/groq), [NVIDIA NIM](https://docs.pipecat.ai/server/services/llm/nim), [Ollama](https://docs.pipecat.ai/server/services/llm/ollama), [OpenAI](https://docs.pipecat.ai/server/services/llm/openai), [OpenRouter](https://docs.pipecat.ai/server/services/llm/openrouter), [Perplexity](https://docs.pipecat.ai/server/services/llm/perplexity), [Qwen](https://docs.pipecat.ai/server/services/llm/qwen), [Together AI](https://docs.pipecat.ai/server/services/llm/together) | `pip install "pipecat-ai[openai]"` | -| Text-to-Speech | [AWS](https://docs.pipecat.ai/server/services/tts/aws), [Azure](https://docs.pipecat.ai/server/services/tts/azure), [Cartesia](https://docs.pipecat.ai/server/services/tts/cartesia), [Deepgram](https://docs.pipecat.ai/server/services/tts/deepgram), [ElevenLabs](https://docs.pipecat.ai/server/services/tts/elevenlabs), [FastPitch (NVIDIA)](https://docs.pipecat.ai/server/services/tts/fastpitch), [Fish](https://docs.pipecat.ai/server/services/tts/fish), [Google](https://docs.pipecat.ai/server/services/tts/google), [LMNT](https://docs.pipecat.ai/server/services/tts/lmnt), [Neuphonic](https://docs.pipecat.ai/server/services/tts/neuphonic), [OpenAI](https://docs.pipecat.ai/server/services/tts/openai), [Piper](https://docs.pipecat.ai/server/services/tts/piper), [PlayHT](https://docs.pipecat.ai/server/services/tts/playht), [Rime](https://docs.pipecat.ai/server/services/tts/rime), [XTTS](https://docs.pipecat.ai/server/services/tts/xtts) | `pip install "pipecat-ai[cartesia]"` | -| Speech-to-Speech | [Gemini Multimodal Live](https://docs.pipecat.ai/server/services/s2s/gemini), [OpenAI Realtime](https://docs.pipecat.ai/server/services/s2s/openai) | `pip install "pipecat-ai[google]"` | -| Transport | [Daily (WebRTC)](https://docs.pipecat.ai/server/services/transport/daily), [FastAPI Websocket](https://docs.pipecat.ai/server/services/transport/fastapi-websocket), [SmallWebRTCTransport](https://docs.pipecat.ai/server/services/transport/small-webrtc), [WebSocket Server](https://docs.pipecat.ai/server/services/transport/websocket-server), Local | `pip install "pipecat-ai[daily]"` | -| Video | [Tavus](https://docs.pipecat.ai/server/services/video/tavus), [Simli](https://docs.pipecat.ai/server/services/video/simli) | `pip install "pipecat-ai[tavus,simli]"` | -| Memory | [mem0](https://docs.pipecat.ai/server/services/memory/mem0) | `pip install "pipecat-ai[mem0]"` | -| Vision & Image | [fal](https://docs.pipecat.ai/server/services/image-generation/fal), [Google Imagen](https://docs.pipecat.ai/server/services/image-generation/fal), [Moondream](https://docs.pipecat.ai/server/services/vision/moondream) | `pip install "pipecat-ai[moondream]"` | -| Audio Processing | [Silero VAD](https://docs.pipecat.ai/server/utilities/audio/silero-vad-analyzer), [Krisp](https://docs.pipecat.ai/server/utilities/audio/krisp-filter), [Koala](https://docs.pipecat.ai/server/utilities/audio/koala-filter), [Noisereduce](https://docs.pipecat.ai/server/utilities/audio/noisereduce-filter) | `pip install "pipecat-ai[silero]"` | -| Analytics & Metrics | [Canonical AI](https://docs.pipecat.ai/server/services/analytics/canonical), [Sentry](https://docs.pipecat.ai/server/services/analytics/sentry) | `pip install "pipecat-ai[canonical]"` | - -๐Ÿ“š [View full services documentation โ†’](https://docs.pipecat.ai/server/services/supported-services) - -## Code examples +## ๐Ÿงช Code examples - [Foundational](https://github.com/pipecat-ai/pipecat/tree/main/examples/foundational) โ€” small snippets that build on each other, introducing one or two concepts at a time - [Example apps](https://github.com/pipecat-ai/pipecat/tree/main/examples/) โ€” complete applications that you can use as starting points for development -## A simple voice agent running locally +## ๐Ÿ› ๏ธ Hacking on the framework itself -Here is a very basic Pipecat bot that greets a user when they join a real-time session. We'll use [Daily](https://daily.co) for real-time media transport, and [Cartesia](https://cartesia.ai/) for text-to-speech. +1. Set up a virtual environment before following these instructions. From the root of the repo: -```python -import asyncio + ```shell + python3 -m venv venv + source venv/bin/activate + ``` -from pipecat.frames.frames import TextFrame -from pipecat.pipeline.pipeline import Pipeline -from pipecat.pipeline.task import PipelineTask -from pipecat.pipeline.runner import PipelineRunner -from pipecat.services.cartesia import CartesiaTTSService -from pipecat.transports.services.daily import DailyParams, DailyTransport +2. Install the development dependencies: -async def main(): - # Use Daily as a real-time media transport (WebRTC) - transport = DailyTransport( - room_url=..., - token="", # leave empty. Note: token is _not_ your api key - bot_name="Bot Name", - params=DailyParams(audio_out_enabled=True)) + ```shell + pip install -r dev-requirements.txt + ``` - # Use Cartesia for Text-to-Speech - tts = CartesiaTTSService( - api_key=..., - voice_id=... - ) +3. Install the git pre-commit hooks (these help ensure your code follows project rules): - # Simple pipeline that will process text to speech and output the result - pipeline = Pipeline([tts, transport.output()]) + ```shell + pre-commit install + ``` - # Create Pipecat processor that can run one or more pipelines tasks - runner = PipelineRunner() +4. Install the `pipecat-ai` package locally in editable mode: - # Assign the task callable to run the pipeline - task = PipelineTask(pipeline) + ```shell + pip install -e . + ``` - # Register an event handler to play audio when a - # participant joins the transport WebRTC session - @transport.event_handler("on_first_participant_joined") - async def on_first_participant_joined(transport, participant): - participant_name = participant.get("info", {}).get("userName", "") - # Queue a TextFrame that will get spoken by the TTS service (Cartesia) - await task.queue_frame(TextFrame(f"Hello there, {participant_name}!")) + > The `-e` or `--editable` option allows you to modify the code without reinstalling. - # Register an event handler to exit the application when the user leaves. - @transport.event_handler("on_participant_left") - async def on_participant_left(transport, participant, reason): - await task.cancel() +5. Include optional dependencies as needed. For example: - # Run the pipeline task - await runner.run(task) + ```shell + pip install -e ".[daily,deepgram,cartesia,openai,silero]" # Updated for the services you're using + ``` -if __name__ == "__main__": - asyncio.run(main()) -``` +6. (Optional) If you want to use this package from another directory: -Run it with: - -```shell -python app.py -``` - -Daily provides a prebuilt WebRTC user interface. While the app is running, you can visit at `https://.daily.co/` and listen to the bot say hello! - -## WebRTC for production use - -WebSockets are fine for server-to-server communication or for initial development. But for production use, youโ€™ll need client-server audio to use a protocol designed for real-time media transport. (For an explanation of the difference between WebSockets and WebRTC, see [this post.](https://www.daily.co/blog/how-to-talk-to-an-llm-with-your-voice/#webrtc)) - -One way to get up and running quickly with WebRTC is to sign up for a Daily developer account. Daily gives you SDKs and global infrastructure for audio (and video) routing. Every account gets 10,000 audio/video/transcription minutes free each month. - -Sign up [here](https://dashboard.daily.co/u/signup) and [create a room](https://docs.daily.co/reference/rest-api/rooms) in the developer Dashboard. - -## Hacking on the framework itself - -_Note: You may need to set up a virtual environment before following these instructions. From the root of the repo:_ - -```shell -python3 -m venv venv -source venv/bin/activate -``` - -Install the development dependencies: - -```shell -pip install -r dev-requirements.txt -``` - -Install the git pre-commit hooks (these help ensure your code follows project rules): - -```shell -pre-commit install -``` - -Install the `pipecat-ai` package locally in editable mode: - -```shell -pip install -e . -``` - -The `-e` or `--editable` option allows you to modify the code without reinstalling. - -To include optional dependencies, add them to the install command. For example: - -```shell -pip install -e ".[daily,deepgram,cartesia,openai,silero]" # Updated for the services you're using -``` - -If you want to use this package from another directory: - -```shell -pip install "path_to_this_repo[option,...]" -``` + ```shell + pip install "path_to_this_repo[option,...]" + ``` ### Running tests @@ -197,11 +136,11 @@ From the root directory, run: pytest ``` -## Setting up your editor +### Setting up your editor This project uses strict [PEP 8](https://peps.python.org/pep-0008/) formatting via [Ruff](https://github.com/astral-sh/ruff). -### Emacs +#### Emacs You can use [use-package](https://github.com/jwiegley/use-package) to install [emacs-lazy-ruff](https://github.com/christophermadsen/emacs-lazy-ruff) package and configure `ruff` arguments: @@ -223,7 +162,7 @@ You can use [use-package](https://github.com/jwiegley/use-package) to install [e :hook ((python-mode . pyvenv-auto-run))) ``` -### Visual Studio Code +#### Visual Studio Code Install the [Ruff](https://marketplace.visualstudio.com/items?itemName=charliermarsh.ruff) extension. Then edit the user settings (_Ctrl-Shift-P_ `Open User Settings (JSON)`) and set it as the default Python formatter, and enable formatting on save: @@ -235,7 +174,7 @@ Install the } ``` -### PyCharm +#### PyCharm `ruff` was installed in the `venv` environment described before, now to enable autoformatting on save, go to `File` -> `Settings` -> `Tools` -> `File Watchers` and add a new watcher with the following settings: @@ -245,7 +184,7 @@ Install the 4. **Arguments**: `format $FilePath$` 5. **Program**: `$PyInterpreterDirectory$/ruff` -## Contributing +## ๐Ÿค Contributing We welcome contributions from the community! Whether you're fixing bugs, improving documentation, or adding new features, here's how you can help: @@ -258,7 +197,7 @@ Before submitting a pull request, please check existing issues and PRs to avoid We aim to review all contributions promptly and provide constructive feedback to help get your changes merged. -## Getting help +## ๐Ÿ›Ÿ Getting help โžก๏ธ [Join our Discord](https://discord.gg/pipecat)