wx44wx/pipecat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'pipecat-ai:main' into anthropic-client-bug-fixes

Browse Source
This commit is contained in:
Prem Adithya
2025-04-04 16:41:04 +11:00
committed by GitHub
parent e8783f6a33 ec00edc893
commit c510870736
118 changed files with 6455 additions and 5842 deletions
Download Patch File Download Diff File Expand all files Collapse all files

76
CHANGELOG.md
View File

@@ -9,6 +9,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- Added new processors `ProducerProcessor` and `ConsumerProcessor`. The producer
processor processes frames from the pipeline and decides whether the consumers
should consume it or not. If so, the same frame that is received by the
producer is sent to the consumer. There can be multiple consumers per
producer. These processors can be useful to push frames from one part of a
pipeline to a different one (e.g. when using `ParallelPipeline`).
### Fixed
- Fixed an issue where `LLMAssistantContextAggregator` would prevent a
`BotStoppedSpeakingFrame` from moving through the pipeline.
## [0.0.62] - 2025-04-01 "An April Fools' release"
### Added
- Added `TransportParams.audio_out_10ms_chunks` parameter to allow controlling
the amount of audio being sent by the output transport. It defaults to 2, so
20ms audio chunks are sent.
- Added `QwenLLMService` for Qwen integration with an OpenAI-compatible
interface. Added foundational example `14q-function-calling-qwen.py`.
@@ -29,6 +49,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **voice-agent**: A minimal example of creating a voice agent with
`SmallWebRTCTransport`.
- `GladiaSTTService` now have comprehensive support for the latest API config
options, including model, language detection, preprocessing, custom
vocabulary, custom spelling, translation, and message filtering options.
- Added `SmallWebRTCTransport`, a new P2P WebRTC transport.
- Created two examples in `p2p-webrtc`:
@@ -49,16 +73,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
- `FunctionCallResultFrame`a are now system frames. This is to prevent function
call results to be discarded during interruptions.
- Pipecat services have been reorganized into packages. Each package can have
one or more of the following modules (in the future new module names might be
needed) depending on the services implemented:
- image: for image generation services
- llm: for LLM services
- memory: for memory services
- stt: for Speech-To-Text services
- tts: for Text-To-Speech services
- video: for video generation services
- vision: for video recognition services
- image: for image generation services
- llm: for LLM services
- memory: for memory services
- stt: for Speech-To-Text services
- tts: for Text-To-Speech services
- video: for video generation services
- vision: for video recognition services
- Base classes for AI services have been reorganized into modules. They can now
be found in
`pipecat.services.[ai_service,image_service,llm_service,stt_service,vision_service]`.
- `GladiaSTTService` now uses the `solaria-1` model by default. Other params
use Gladia's default values. Added support for more language codes.
### Deprecated
@@ -67,14 +102,41 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`pipecat.services.[service].[image,llm,memory,stt,tts,video,vision]`. For
example, `from pipecat.services.openai.llm import OpenAILLMService`.
- Import for AI services base classes from `pipecat.services.ai_services` is now
deprecated, use one of
`pipecat.services.[ai_service,image_service,llm_service,stt_service,vision_service]`.
- Deprecated the `language` parameter in `GladiaSTTService.InputParams` in
favor of `language_config`, which better aligns with Gladia's API.
- Deprecated using `GladiaSTTService.InputParams` directly. Use the new
`GladiaInputParams` class instead.
### Fixed
- Fixed a `FastAPIWebsocketTransport` and `WebsocketClientTransport` issue that
would cause the transport to be closed prematurely, preventing the internally
queued audio to be sent. The same issue could also cause an infinite loop
while using an output mixer and when sending an `EndFrame`, preventing the bot
to finish.
- Fixed an issue that could cause the `TranscriptionUpdateFrame` being pushed
because of an interruption to be discarded.
- Fixed an issue that would cause `SegmentedSTTService` based services
(e.g. `OpenAISTTService`) to try to transcribe non-spoken audio, causing
invalid transcriptions.
- Fixed an issue where `GoogleTTSService` was emitting two `TTSStoppedFrames`.
### Performance
- Output transports now send 40ms audio chunks instead of 20ms. This should
improve performance.
- `BotSpeakingFrame`s are now sent every 200ms. If the output transport audio chunks
are higher than 200ms then they will be sent at every audio chunk.
### Other
- Added foundational example `37-mem0.py` demonstrating how to use the

67
CONTRIBUTING.md
View File

@@ -26,11 +26,52 @@ git commit -m "Description of your changes"
git push origin your-branch-name
```
9. **Submit a Pull Request (PR)**: Open a PR from your forked repository to the main branch of this repo.
> Important: Describe the changes you've made clearly!
8. **Submit a Pull Request (PR)**: Open a PR from your forked repository to the main branch of this repo.
> Important: Describe the changes you've made clearly!
Our maintainers will review your PR, and once everything is good, your contributions will be merged!
## Code Style and Documentation
### Python Code Style
We use Ruff for code linting and formatting. Please ensure your code passes all linting checks before submitting a PR.
### Docstring Conventions
We follow Google-style docstrings with these specific conventions:
- Class docstrings should fully document all parameters used in `__init__`
- We don't require separate docstrings for `__init__` methods when parameters are documented in the class docstring
- Property methods should have docstrings explaining their purpose and return value
Example of correctly documented class:
```python
class MyClass:
"""Class description.
Additional details about the class.
Args:
param1: Description of first parameter.
param2: Description of second parameter.
"""
def __init__(self, param1, param2):
# No docstring required here as parameters are documented above
self.param1 = param1
self.param2 = param2
@property
def some_property(self) -> str:
"""Get the formatted property value.
Returns:
A string representation of the property.
"""
return f"Property: {self.param1}"
```
# Contributor Covenant Code of Conduct
@@ -51,23 +92,23 @@ diverse, inclusive, and healthy community.
Examples of behavior that contributes to a positive environment for our
community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
- Demonstrating empathy and kindness toward other people
- Being respectful of differing opinions, viewpoints, and experiences
- Giving and gracefully accepting constructive feedback
- Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the overall
- Focusing on what is best not just for us as individuals, but for the overall
community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or advances of
- The use of sexualized language or imagery, and sexual attention or advances of
any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email address,
- Trolling, insulting or derogatory comments, and personal or political attacks
- Public or private harassment
- Publishing others' private information, such as a physical or email address,
without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
- Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement Responsibilities
@@ -162,4 +203,4 @@ For answers to common questions about this code of conduct, see the FAQ at
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
[Mozilla CoC]: https://github.com/mozilla/diversity
[FAQ]: https://www.contributor-covenant.org/faq
[translations]: https://www.contributor-covenant.org/translations
[translations]: https://www.contributor-covenant.org/translations

4
docs/api/index.rst
View File

@@ -45,8 +45,10 @@ Transport & Serialization
Utilities
~~~~~~~~~
* :mod:`Adapters <pipecat.adapters>`
* :mod:`Clocks <pipecat.clocks>`
* :mod:`Metrics <pipecat.metrics>`
* :mod:`Observers <pipecat.observers>`
* :mod:`Sync <pipecat.sync>`
* :mod:`Transcriptions <pipecat.transcriptions>`
* :mod:`Utils <pipecat.utils>`
@@ -56,10 +58,12 @@ Utilities
:caption: API Reference
:hidden:
Adapters <api/pipecat.adapters>
Audio <api/pipecat.audio>
Clocks <api/pipecat.clocks>
Frames <api/pipecat.frames>
Metrics <api/pipecat.metrics>
Observers <api/pipecat.observers>
Pipeline <api/pipecat.pipeline>
Processors <api/pipecat.processors>
Serializers <api/pipecat.serializers>

2
examples/foundational/07g-interruptible-openai.py
View File

@@ -59,7 +59,7 @@ async def main():
prompt="Expect words related to dogs, such as breed names.",
)
tts = OpenAITTSService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o-mini-tts-latest")
tts = OpenAITTSService(api_key=os.getenv("OPENAI_API_KEY"), voice="ballad")
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")

13
examples/foundational/07j-interruptible-gladia.py
View File

@@ -18,9 +18,11 @@ from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.services.cartesia import CartesiaTTSService
from pipecat.services.gladia import GladiaSTTService
from pipecat.services.openai import OpenAILLMService
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.services.gladia.config import GladiaInputParams, LanguageConfig
from pipecat.services.gladia.stt import GladiaSTTService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.transcriptions.language import Language
from pipecat.transports.services.daily import DailyParams, DailyTransport
load_dotenv(override=True)
@@ -47,6 +49,11 @@ async def main():
stt = GladiaSTTService(
api_key=os.getenv("GLADIA_API_KEY"),
params=GladiaInputParams(
language_config=LanguageConfig(
languages=[Language.EN],
)
),
)
tts = CartesiaTTSService(

2
examples/foundational/07u-interruptible-ultravox.py
View File

@@ -34,7 +34,7 @@ logger.add(sys.stderr, level="DEBUG")
# Want to initialize the ultravox processor since it takes time to load the model and dont
# want to load it every time the pipeline is run
ultravox_processor = UltravoxSTTService(
model_size="fixie-ai/ultravox-v0_4_1-llama-3_1-8b",
model_name="fixie-ai/ultravox-v0_5-llama-3_1-8b",
hf_token=os.getenv("HF_TOKEN"),
)

2
examples/foundational/19a-azure-realtime-beta.py
View File

@@ -147,8 +147,8 @@ Remember, your responses should be short. Just one or two sentences, usually."""
transport.input(), # Transport user input
context_aggregator.user(),
llm, # LLM
context_aggregator.assistant(),
transport.output(), # Transport bot output
context_aggregator.assistant(),
]
)

2
examples/foundational/20a-persistent-context-openai.py
View File

@@ -205,8 +205,8 @@ async def main():
context_aggregator.user(),
llm, # LLM
tts,
context_aggregator.assistant(),
transport.output(), # Transport bot output
context_aggregator.assistant(),
]
)

2
examples/foundational/20b-persistent-context-openai-realtime.py
View File

@@ -230,8 +230,8 @@ Remember, your responses should be short. Just one or two sentences, usually."""
transport.input(), # Transport user input
context_aggregator.user(),
llm, # LLM
context_aggregator.assistant(),
transport.output(), # Transport bot output
context_aggregator.assistant(),
]
)

2
examples/foundational/20c-persistent-context-anthropic.py
View File

@@ -202,8 +202,8 @@ async def main():
context_aggregator.user(),
llm, # LLM
tts,
context_aggregator.assistant(),
transport.output(), # Transport bot output
context_aggregator.assistant(),
]
)

2
examples/foundational/20d-persistent-context-gemini.py
View File

@@ -261,8 +261,8 @@ async def main():
context_aggregator.user(),
llm, # LLM
tts,
context_aggregator.assistant(),
transport.output(), # Transport bot output
context_aggregator.assistant(),
]
)

2
examples/foundational/26b-gemini-multimodal-live-function-calling.py
View File

@@ -110,8 +110,8 @@ async def main():
transport.input(),
context_aggregator.user(),
llm,
context_aggregator.assistant(),
transport.output(),
context_aggregator.assistant(),
]
)

872
examples/instant-voice/client/javascript/package-lock.json generated
View File

@@ -9,8 +9,8 @@
"version": "1.0.0",
"license": "ISC",
"dependencies": {
"@pipecat-ai/client-js": "^0.3.2",
"@pipecat-ai/daily-transport": "^0.3.5"
"@pipecat-ai/client-js": "^0.3.5",
"@pipecat-ai/daily-transport": "^0.3.8"
},
"devDependencies": {
"@types/node": "^22.13.1",
@@ -20,9 +20,9 @@
}
},
"node_modules/@babel/runtime": {
"version": "7.26.0",
"resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.26.0.tgz",
"integrity": "sha512-FDSOghenHTiToteC/QRlv2q3DhPZ/oOXTBoirfWNx1Cx3TMVcGWQtMMmQcSvb/JjpNeGzx8Pq/b4fKEJuWm1sw==",
"version": "7.27.0",
"resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.27.0.tgz",
"integrity": "sha512-VtPOkrdPHZsKc/clNqyi9WUA8TINkZ4cGk63UUE3u4pmB2k+ZMQRDuIOagv8UVd6j7k0T3+RRIb7beKTebNbcw==",
"license": "MIT",
"dependencies": {
"regenerator-runtime": "^0.14.0"
@@ -32,9 +32,9 @@
}
},
"node_modules/@daily-co/daily-js": {
"version": "0.73.0",
"resolved": "https://registry.npmjs.org/@daily-co/daily-js/-/daily-js-0.73.0.tgz",
"integrity": "sha512-Wz8c60hgmkx8fcEeDAi4L4J0rbafiihWKyXFyhYoFYPsw2OdChHpA4RYwIB+1enRws5IK+/HdmzFDYLQsB4A6w==",
"version": "0.77.0",
"resolved": "https://registry.npmjs.org/@daily-co/daily-js/-/daily-js-0.77.0.tgz",
"integrity": "sha512-icNXKieKAkRR/C5dcPjrCkL1jQGFp5C5WtLHy5uHAdTztm+mo9wlPJuehbWaGOM3TV24mgWHZ/+8jOys1G0I4w==",
"license": "BSD-2-Clause",
"dependencies": {
"@babel/runtime": "^7.12.5",
@@ -47,74 +47,6 @@
"node": ">=10.0.0"
}
},
"node_modules/@esbuild/aix-ppc64": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.24.0.tgz",
"integrity": "sha512-WtKdFM7ls47zkKHFVzMz8opM7LkcsIp9amDUBIAWirg70RM71WRSjdILPsY5Uv1D42ZpUfaPILDlfactHgsRkw==",
"cpu": [
"ppc64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"aix"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/android-arm": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.24.0.tgz",
"integrity": "sha512-arAtTPo76fJ/ICkXWetLCc9EwEHKaeya4vMrReVlEIUCAUncH7M4bhMQ+M9Vf+FFOZJdTNMXNBrWwW+OXWpSew==",
"cpu": [
"arm"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"android"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/android-arm64": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.24.0.tgz",
"integrity": "sha512-Vsm497xFM7tTIPYK9bNTYJyF/lsP590Qc1WxJdlB6ljCbdZKU9SY8i7+Iin4kyhV/KV5J2rOKsBQbB77Ab7L/w==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"android"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/android-x64": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.24.0.tgz",
"integrity": "sha512-t8GrvnFkiIY7pa7mMgJd7p8p8qqYIz1NYiAoKc75Zyv73L3DZW++oYMSHPRarcotTKuSs6m3hTOa5CKHaS02TQ==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"android"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/darwin-arm64": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.24.0.tgz",
@@ -132,333 +64,10 @@
"node": ">=18"
}
},
"node_modules/@esbuild/darwin-x64": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.24.0.tgz",
"integrity": "sha512-rgtz6flkVkh58od4PwTRqxbKH9cOjaXCMZgWD905JOzjFKW+7EiUObfd/Kav+A6Gyud6WZk9w+xu6QLytdi2OA==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"darwin"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/freebsd-arm64": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.24.0.tgz",
"integrity": "sha512-6Mtdq5nHggwfDNLAHkPlyLBpE5L6hwsuXZX8XNmHno9JuL2+bg2BX5tRkwjyfn6sKbxZTq68suOjgWqCicvPXA==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"freebsd"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/freebsd-x64": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.24.0.tgz",
"integrity": "sha512-D3H+xh3/zphoX8ck4S2RxKR6gHlHDXXzOf6f/9dbFt/NRBDIE33+cVa49Kil4WUjxMGW0ZIYBYtaGCa2+OsQwQ==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"freebsd"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/linux-arm": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.24.0.tgz",
"integrity": "sha512-gJKIi2IjRo5G6Glxb8d3DzYXlxdEj2NlkixPsqePSZMhLudqPhtZ4BUrpIuTjJYXxvF9njql+vRjB2oaC9XpBw==",
"cpu": [
"arm"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/linux-arm64": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.24.0.tgz",
"integrity": "sha512-TDijPXTOeE3eaMkRYpcy3LarIg13dS9wWHRdwYRnzlwlA370rNdZqbcp0WTyyV/k2zSxfko52+C7jU5F9Tfj1g==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/linux-ia32": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.24.0.tgz",
"integrity": "sha512-K40ip1LAcA0byL05TbCQ4yJ4swvnbzHscRmUilrmP9Am7//0UjPreh4lpYzvThT2Quw66MhjG//20mrufm40mA==",
"cpu": [
"ia32"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/linux-loong64": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.24.0.tgz",
"integrity": "sha512-0mswrYP/9ai+CU0BzBfPMZ8RVm3RGAN/lmOMgW4aFUSOQBjA31UP8Mr6DDhWSuMwj7jaWOT0p0WoZ6jeHhrD7g==",
"cpu": [
"loong64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/linux-mips64el": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.24.0.tgz",
"integrity": "sha512-hIKvXm0/3w/5+RDtCJeXqMZGkI2s4oMUGj3/jM0QzhgIASWrGO5/RlzAzm5nNh/awHE0A19h/CvHQe6FaBNrRA==",
"cpu": [
"mips64el"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/linux-ppc64": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.24.0.tgz",
"integrity": "sha512-HcZh5BNq0aC52UoocJxaKORfFODWXZxtBaaZNuN3PUX3MoDsChsZqopzi5UupRhPHSEHotoiptqikjN/B77mYQ==",
"cpu": [
"ppc64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/linux-riscv64": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.24.0.tgz",
"integrity": "sha512-bEh7dMn/h3QxeR2KTy1DUszQjUrIHPZKyO6aN1X4BCnhfYhuQqedHaa5MxSQA/06j3GpiIlFGSsy1c7Gf9padw==",
"cpu": [
"riscv64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/linux-s390x": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.24.0.tgz",
"integrity": "sha512-ZcQ6+qRkw1UcZGPyrCiHHkmBaj9SiCD8Oqd556HldP+QlpUIe2Wgn3ehQGVoPOvZvtHm8HPx+bH20c9pvbkX3g==",
"cpu": [
"s390x"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/linux-x64": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.24.0.tgz",
"integrity": "sha512-vbutsFqQ+foy3wSSbmjBXXIJ6PL3scghJoM8zCL142cGaZKAdCZHyf+Bpu/MmX9zT9Q0zFBVKb36Ma5Fzfa8xA==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/netbsd-x64": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.24.0.tgz",
"integrity": "sha512-hjQ0R/ulkO8fCYFsG0FZoH+pWgTTDreqpqY7UnQntnaKv95uP5iW3+dChxnx7C3trQQU40S+OgWhUVwCjVFLvg==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"netbsd"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/openbsd-arm64": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.24.0.tgz",
"integrity": "sha512-MD9uzzkPQbYehwcN583yx3Tu5M8EIoTD+tUgKF982WYL9Pf5rKy9ltgD0eUgs8pvKnmizxjXZyLt0z6DC3rRXg==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"openbsd"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/openbsd-x64": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.24.0.tgz",
"integrity": "sha512-4ir0aY1NGUhIC1hdoCzr1+5b43mw99uNwVzhIq1OY3QcEwPDO3B7WNXBzaKY5Nsf1+N11i1eOfFcq+D/gOS15Q==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"openbsd"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/sunos-x64": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.24.0.tgz",
"integrity": "sha512-jVzdzsbM5xrotH+W5f1s+JtUy1UWgjU0Cf4wMvffTB8m6wP5/kx0KiaLHlbJO+dMgtxKV8RQ/JvtlFcdZ1zCPA==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"sunos"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/win32-arm64": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.24.0.tgz",
"integrity": "sha512-iKc8GAslzRpBytO2/aN3d2yb2z8XTVfNV0PjGlCxKo5SgWmNXx82I/Q3aG1tFfS+A2igVCY97TJ8tnYwpUWLCA==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"win32"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/win32-ia32": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.24.0.tgz",
"integrity": "sha512-vQW36KZolfIudCcTnaTpmLQ24Ha1RjygBo39/aLkM2kmjkWmZGEJ5Gn9l5/7tzXA42QGIoWbICfg6KLLkIw6yw==",
"cpu": [
"ia32"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"win32"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@esbuild/win32-x64": {
"version": "0.24.0",
"resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.24.0.tgz",
"integrity": "sha512-7IAFPrjSQIJrGsK6flwg7NFmwBoSTyF3rl7If0hNUFQU4ilTsEPL6GuMuU9BfIWVVGuRnuIidkSMC+c0Otu8IA==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"win32"
],
"engines": {
"node": ">=18"
}
},
"node_modules/@pipecat-ai/client-js": {
"version": "0.3.2",
"resolved": "https://registry.npmjs.org/@pipecat-ai/client-js/-/client-js-0.3.2.tgz",
"integrity": "sha512-psunOVrJjPka2SWlq53vxVWCA0Vt8pSXsXtn8pOLC0YTKFsUx+b7Z6quYUJcDZjCe1aAg9cKETek3Xal3Co8Tg==",
"version": "0.3.5",
"resolved": "https://registry.npmjs.org/@pipecat-ai/client-js/-/client-js-0.3.5.tgz",
"integrity": "sha512-qmhnDjwY2XUtLjww35ShsYf5TF9BCuAk0tIj0oHjpTe6v6QOlgKQt8JVCAdc32p5ycouzSZOeDFtBd2aNWuq1g==",
"license": "BSD-2-Clause",
"dependencies": {
"@types/events": "^3.0.3",
@@ -469,45 +78,17 @@
}
},
"node_modules/@pipecat-ai/daily-transport": {
"version": "0.3.5",
"resolved": "https://registry.npmjs.org/@pipecat-ai/daily-transport/-/daily-transport-0.3.5.tgz",
"integrity": "sha512-nJ0TvWPCqXPmU81U8cXOqk5mUEEvEuI06Mis+N0jN8KZUrNy1pP08iWbs07ObmIXdnQcoL+kQmHOerT4q/bF0w==",
"version": "0.3.8",
"resolved": "https://registry.npmjs.org/@pipecat-ai/daily-transport/-/daily-transport-0.3.8.tgz",
"integrity": "sha512-AcRP51LGOsEA7DH0yPaZTqX/pozfTpkJbKC0itgWLv6uCM8dAnNtBj/m1CdFKRsE7QObhEOa+cRp5PUAyF4wCA==",
"license": "BSD-2-Clause",
"dependencies": {
"@daily-co/daily-js": "^0.73.0"
"@daily-co/daily-js": "^0.77.0"
},
"peerDependencies": {
"@pipecat-ai/client-js": "~0.3.2"
"@pipecat-ai/client-js": "~0.3.5"
}
},
"node_modules/@rollup/rollup-android-arm-eabi": {
"version": "4.28.0",
"resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm-eabi/-/rollup-android-arm-eabi-4.28.0.tgz",
"integrity": "sha512-wLJuPLT6grGZsy34g4N1yRfYeouklTgPhH1gWXCYspenKYD0s3cR99ZevOGw5BexMNywkbV3UkjADisozBmpPQ==",
"cpu": [
"arm"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"android"
]
},
"node_modules/@rollup/rollup-android-arm64": {
"version": "4.28.0",
"resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm64/-/rollup-android-arm64-4.28.0.tgz",
"integrity": "sha512-eiNkznlo0dLmVG/6wf+Ifi/v78G4d4QxRhuUl+s8EWZpDewgk7PX3ZyECUXU0Zq/Ca+8nU8cQpNC4Xgn2gFNDA==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"android"
]
},
"node_modules/@rollup/rollup-darwin-arm64": {
"version": "4.28.0",
"resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-arm64/-/rollup-darwin-arm64-4.28.0.tgz",
@@ -522,286 +103,76 @@
"darwin"
]
},
"node_modules/@rollup/rollup-darwin-x64": {
"version": "4.28.0",
"resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-x64/-/rollup-darwin-x64-4.28.0.tgz",
"integrity": "sha512-8hxgfReVs7k9Js1uAIhS6zq3I+wKQETInnWQtgzt8JfGx51R1N6DRVy3F4o0lQwumbErRz52YqwjfvuwRxGv1w==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"darwin"
]
},
"node_modules/@rollup/rollup-freebsd-arm64": {
"version": "4.28.0",
"resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-arm64/-/rollup-freebsd-arm64-4.28.0.tgz",
"integrity": "sha512-lA1zZB3bFx5oxu9fYud4+g1mt+lYXCoch0M0V/xhqLoGatbzVse0wlSQ1UYOWKpuSu3gyN4qEc0Dxf/DII1bhQ==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"freebsd"
]
},
"node_modules/@rollup/rollup-freebsd-x64": {
"version": "4.28.0",
"resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-x64/-/rollup-freebsd-x64-4.28.0.tgz",
"integrity": "sha512-aI2plavbUDjCQB/sRbeUZWX9qp12GfYkYSJOrdYTL/C5D53bsE2/nBPuoiJKoWp5SN78v2Vr8ZPnB+/VbQ2pFA==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"freebsd"
]
},
"node_modules/@rollup/rollup-linux-arm-gnueabihf": {
"version": "4.28.0",
"resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-gnueabihf/-/rollup-linux-arm-gnueabihf-4.28.0.tgz",
"integrity": "sha512-WXveUPKtfqtaNvpf0iOb0M6xC64GzUX/OowbqfiCSXTdi/jLlOmH0Ba94/OkiY2yTGTwteo4/dsHRfh5bDCZ+w==",
"cpu": [
"arm"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@rollup/rollup-linux-arm-musleabihf": {
"version": "4.28.0",
"resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-musleabihf/-/rollup-linux-arm-musleabihf-4.28.0.tgz",
"integrity": "sha512-yLc3O2NtOQR67lI79zsSc7lk31xjwcaocvdD1twL64PK1yNaIqCeWI9L5B4MFPAVGEVjH5k1oWSGuYX1Wutxpg==",
"cpu": [
"arm"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@rollup/rollup-linux-arm64-gnu": {
"version": "4.28.0",
"resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-gnu/-/rollup-linux-arm64-gnu-4.28.0.tgz",
"integrity": "sha512-+P9G9hjEpHucHRXqesY+3X9hD2wh0iNnJXX/QhS/J5vTdG6VhNYMxJ2rJkQOxRUd17u5mbMLHM7yWGZdAASfcg==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@rollup/rollup-linux-arm64-musl": {
"version": "4.28.0",
"resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-musl/-/rollup-linux-arm64-musl-4.28.0.tgz",
"integrity": "sha512-1xsm2rCKSTpKzi5/ypT5wfc+4bOGa/9yI/eaOLW0oMs7qpC542APWhl4A37AENGZ6St6GBMWhCCMM6tXgTIplw==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@rollup/rollup-linux-powerpc64le-gnu": {
"version": "4.28.0",
"resolved": "https://registry.npmjs.org/@rollup/rollup-linux-powerpc64le-gnu/-/rollup-linux-powerpc64le-gnu-4.28.0.tgz",
"integrity": "sha512-zgWxMq8neVQeXL+ouSf6S7DoNeo6EPgi1eeqHXVKQxqPy1B2NvTbaOUWPn/7CfMKL7xvhV0/+fq/Z/J69g1WAQ==",
"cpu": [
"ppc64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@rollup/rollup-linux-riscv64-gnu": {
"version": "4.28.0",
"resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-gnu/-/rollup-linux-riscv64-gnu-4.28.0.tgz",
"integrity": "sha512-VEdVYacLniRxbRJLNtzwGt5vwS0ycYshofI7cWAfj7Vg5asqj+pt+Q6x4n+AONSZW/kVm+5nklde0qs2EUwU2g==",
"cpu": [
"riscv64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@rollup/rollup-linux-s390x-gnu": {
"version": "4.28.0",
"resolved": "https://registry.npmjs.org/@rollup/rollup-linux-s390x-gnu/-/rollup-linux-s390x-gnu-4.28.0.tgz",
"integrity": "sha512-LQlP5t2hcDJh8HV8RELD9/xlYtEzJkm/aWGsauvdO2ulfl3QYRjqrKW+mGAIWP5kdNCBheqqqYIGElSRCaXfpw==",
"cpu": [
"s390x"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@rollup/rollup-linux-x64-gnu": {
"version": "4.28.0",
"resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-gnu/-/rollup-linux-x64-gnu-4.28.0.tgz",
"integrity": "sha512-Nl4KIzteVEKE9BdAvYoTkW19pa7LR/RBrT6F1dJCV/3pbjwDcaOq+edkP0LXuJ9kflW/xOK414X78r+K84+msw==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@rollup/rollup-linux-x64-musl": {
"version": "4.28.0",
"resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-musl/-/rollup-linux-x64-musl-4.28.0.tgz",
"integrity": "sha512-eKpJr4vBDOi4goT75MvW+0dXcNUqisK4jvibY9vDdlgLx+yekxSm55StsHbxUsRxSTt3JEQvlr3cGDkzcSP8bw==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/@rollup/rollup-win32-arm64-msvc": {
"version": "4.28.0",
"resolved": "https://registry.npmjs.org/@rollup/rollup-win32-arm64-msvc/-/rollup-win32-arm64-msvc-4.28.0.tgz",
"integrity": "sha512-Vi+WR62xWGsE/Oj+mD0FNAPY2MEox3cfyG0zLpotZdehPFXwz6lypkGs5y38Jd/NVSbOD02aVad6q6QYF7i8Bg==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"win32"
]
},
"node_modules/@rollup/rollup-win32-ia32-msvc": {
"version": "4.28.0",
"resolved": "https://registry.npmjs.org/@rollup/rollup-win32-ia32-msvc/-/rollup-win32-ia32-msvc-4.28.0.tgz",
"integrity": "sha512-kN/Vpip8emMLn/eOza+4JwqDZBL6MPNpkdaEsgUtW1NYN3DZvZqSQrbKzJcTL6hd8YNmFTn7XGWMwccOcJBL0A==",
"cpu": [
"ia32"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"win32"
]
},
"node_modules/@rollup/rollup-win32-x64-msvc": {
"version": "4.28.0",
"resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-msvc/-/rollup-win32-x64-msvc-4.28.0.tgz",
"integrity": "sha512-Bvno2/aZT6usSa7lRDL2+hMjVAGjuqaymF1ApZm31JXzniR/hvr14jpU+/z4X6Gt5BPlzosscyJZGUvguXIqeQ==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"win32"
]
},
"node_modules/@sentry-internal/browser-utils": {
"version": "8.49.0",
"resolved": "https://registry.npmjs.org/@sentry-internal/browser-utils/-/browser-utils-8.49.0.tgz",
"integrity": "sha512-XkPHHdFqsN7EPaB+QGUOEmpFqXiqP67t2rRZ1HG1UwJoe0PhJEKNy7b4+WRwmT7ODSt+PvFk1gNBlJBpThwH7Q==",
"version": "8.55.0",
"resolved": "https://registry.npmjs.org/@sentry-internal/browser-utils/-/browser-utils-8.55.0.tgz",
"integrity": "sha512-ROgqtQfpH/82AQIpESPqPQe0UyWywKJsmVIqi3c5Fh+zkds5LUxnssTj3yNd1x+kxaPDVB023jAP+3ibNgeNDw==",
"license": "MIT",
"dependencies": {
"@sentry/core": "8.49.0"
"@sentry/core": "8.55.0"
},
"engines": {
"node": ">=14.18"
}
},
"node_modules/@sentry-internal/feedback": {
"version": "8.49.0",
"resolved": "https://registry.npmjs.org/@sentry-internal/feedback/-/feedback-8.49.0.tgz",
"integrity": "sha512-v/wf7WvPxEvZUB7xrCnecI3fhevVo84hw8WlxgZIz6mLUHXEIX8xYWc9H8Yet/KKJ2uEB8GQ8aDsY6S1hVEIUA==",
"version": "8.55.0",
"resolved": "https://registry.npmjs.org/@sentry-internal/feedback/-/feedback-8.55.0.tgz",
"integrity": "sha512-cP3BD/Q6pquVQ+YL+rwCnorKuTXiS9KXW8HNKu4nmmBAyf7urjs+F6Hr1k9MXP5yQ8W3yK7jRWd09Yu6DHWOiw==",
"license": "MIT",
"dependencies": {
"@sentry/core": "8.49.0"
"@sentry/core": "8.55.0"
},
"engines": {
"node": ">=14.18"
}
},
"node_modules/@sentry-internal/replay": {
"version": "8.49.0",
"resolved": "https://registry.npmjs.org/@sentry-internal/replay/-/replay-8.49.0.tgz",
"integrity": "sha512-BDiiCBxskkktTd6FNplBc9V8l14R4T/AwRIZj2itX4xnuHewTTDjVbeyvGol4roA4r+V0Mzoi31hLEGI6yFQ5Q==",
"version": "8.55.0",
"resolved": "https://registry.npmjs.org/@sentry-internal/replay/-/replay-8.55.0.tgz",
"integrity": "sha512-roCDEGkORwolxBn8xAKedybY+Jlefq3xYmgN2fr3BTnsXjSYOPC7D1/mYqINBat99nDtvgFvNfRcZPiwwZ1hSw==",
"license": "MIT",
"dependencies": {
"@sentry-internal/browser-utils": "8.49.0",
"@sentry/core": "8.49.0"
"@sentry-internal/browser-utils": "8.55.0",
"@sentry/core": "8.55.0"
},
"engines": {
"node": ">=14.18"
}
},
"node_modules/@sentry-internal/replay-canvas": {
"version": "8.49.0",
"resolved": "https://registry.npmjs.org/@sentry-internal/replay-canvas/-/replay-canvas-8.49.0.tgz",
"integrity": "sha512-/yXxI7f+Wu24FIYoRE7A0AidNxORuhAyPzb5ey1wFqMXP72nG8dXhOpcl0w+bi554FkqkLjdeUDhSOBWYZXH9g==",
"version": "8.55.0",
"resolved": "https://registry.npmjs.org/@sentry-internal/replay-canvas/-/replay-canvas-8.55.0.tgz",
"integrity": "sha512-nIkfgRWk1091zHdu4NbocQsxZF1rv1f7bbp3tTIlZYbrH62XVZosx5iHAuZG0Zc48AETLE7K4AX9VGjvQj8i9w==",
"license": "MIT",
"dependencies": {
"@sentry-internal/replay": "8.49.0",
"@sentry/core": "8.49.0"
"@sentry-internal/replay": "8.55.0",
"@sentry/core": "8.55.0"
},
"engines": {
"node": ">=14.18"
}
},
"node_modules/@sentry/browser": {
"version": "8.49.0",
"resolved": "https://registry.npmjs.org/@sentry/browser/-/browser-8.49.0.tgz",
"integrity": "sha512-dS4Sw2h8EixHeXOIR++XEVMTen6xCGcIQ/XhJbsjqvddXeIijW0WkxSeTfPkfs0dsqFHSisWmlmo0xhHbXvEsQ==",
"version": "8.55.0",
"resolved": "https://registry.npmjs.org/@sentry/browser/-/browser-8.55.0.tgz",
"integrity": "sha512-1A31mCEWCjaMxJt6qGUK+aDnLDcK6AwLAZnqpSchNysGni1pSn1RWSmk9TBF8qyTds5FH8B31H480uxMPUJ7Cw==",
"license": "MIT",
"dependencies": {
"@sentry-internal/browser-utils": "8.49.0",
"@sentry-internal/feedback": "8.49.0",
"@sentry-internal/replay": "8.49.0",
"@sentry-internal/replay-canvas": "8.49.0",
"@sentry/core": "8.49.0"
"@sentry-internal/browser-utils": "8.55.0",
"@sentry-internal/feedback": "8.55.0",
"@sentry-internal/replay": "8.55.0",
"@sentry-internal/replay-canvas": "8.55.0",
"@sentry/core": "8.55.0"
},
"engines": {
"node": ">=14.18"
}
},
"node_modules/@sentry/core": {
"version": "8.49.0",
"resolved": "https://registry.npmjs.org/@sentry/core/-/core-8.49.0.tgz",
"integrity": "sha512-/OAm6LdHhh8TvfDAucWfSJV7M03IOHrJm5LVjrrKr4gwQ1HKd4CDbARsBbPwHIzSRAle0IgG3sbJxEvv52JUIw==",
"version": "8.55.0",
"resolved": "https://registry.npmjs.org/@sentry/core/-/core-8.55.0.tgz",
"integrity": "sha512-6g7jpbefjHYs821Z+EBJ8r4Z7LT5h80YSWRJaylGS4nW5W5Z2KXzpdnyFarv37O7QjauzVC2E+PABmpkw5/JGA==",
"license": "MIT",
"engines": {
"node": ">=14.18"
@@ -863,159 +234,6 @@
"node": ">=10"
}
},
"node_modules/@swc/core-darwin-x64": {
"version": "1.10.14",
"resolved": "https://registry.npmjs.org/@swc/core-darwin-x64/-/core-darwin-x64-1.10.14.tgz",
"integrity": "sha512-KpzotL/I0O12RE3tF8NmQErINv0cQe/0mnN/Q50ESFzB5kU6bLgp2HMnnwDTm/XEZZRJCNe0oc9WJ5rKbAJFRQ==",
"cpu": [
"x64"
],
"dev": true,
"license": "Apache-2.0 AND MIT",
"optional": true,
"os": [
"darwin"
],
"engines": {
"node": ">=10"
}
},
"node_modules/@swc/core-linux-arm-gnueabihf": {
"version": "1.10.14",
"resolved": "https://registry.npmjs.org/@swc/core-linux-arm-gnueabihf/-/core-linux-arm-gnueabihf-1.10.14.tgz",
"integrity": "sha512-20yRXZjMJVz1wp1TcscKiGTVXistG+saIaxOmxSNQia1Qun3hSWLL+u6+5kXbfYGr7R2N6kqSwtZbIfJI25r9Q==",
"cpu": [
"arm"
],
"dev": true,
"license": "Apache-2.0",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">=10"
}
},
"node_modules/@swc/core-linux-arm64-gnu": {
"version": "1.10.14",
"resolved": "https://registry.npmjs.org/@swc/core-linux-arm64-gnu/-/core-linux-arm64-gnu-1.10.14.tgz",
"integrity": "sha512-Gy7cGrNkiMfPxQyLGxdgXPwyWzNzbHuWycJFcoKBihxZKZIW8hkPBttkGivuLC+0qOgsV2/U+S7tlvAju7FtmQ==",
"cpu": [
"arm64"
],
"dev": true,
"license": "Apache-2.0 AND MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">=10"
}
},
"node_modules/@swc/core-linux-arm64-musl": {
"version": "1.10.14",
"resolved": "https://registry.npmjs.org/@swc/core-linux-arm64-musl/-/core-linux-arm64-musl-1.10.14.tgz",
"integrity": "sha512-+oYVqJvFw62InZ8PIy1rBACJPC2WTe4vbVb9kM1jJj2D7dKLm9acnnYIVIDsM5Wo7Uab8RvPHXVbs19IBurzuw==",
"cpu": [
"arm64"
],
"dev": true,
"license": "Apache-2.0 AND MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">=10"
}
},
"node_modules/@swc/core-linux-x64-gnu": {
"version": "1.10.14",
"resolved": "https://registry.npmjs.org/@swc/core-linux-x64-gnu/-/core-linux-x64-gnu-1.10.14.tgz",
"integrity": "sha512-OmEbVEKQFLQVHwo4EJl9osmlulURy46k232Opfpn/1ji0t2KcNCci3POsnfMuoZjLkGJv8vGNJdPQxX+CP+wSA==",
"cpu": [
"x64"
],
"dev": true,
"license": "Apache-2.0 AND MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">=10"
}
},
"node_modules/@swc/core-linux-x64-musl": {
"version": "1.10.14",
"resolved": "https://registry.npmjs.org/@swc/core-linux-x64-musl/-/core-linux-x64-musl-1.10.14.tgz",
"integrity": "sha512-OZW+Icm8DMPqHbhdxplkuG8qrNnPk5i7xJOZWYi1y5bTjgGFI4nEzrsmmeHKMdQTaWwsFrm3uK1rlyQ48MmXmg==",
"cpu": [
"x64"
],
"dev": true,
"license": "Apache-2.0 AND MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">=10"
}
},
"node_modules/@swc/core-win32-arm64-msvc": {
"version": "1.10.14",
"resolved": "https://registry.npmjs.org/@swc/core-win32-arm64-msvc/-/core-win32-arm64-msvc-1.10.14.tgz",
"integrity": "sha512-sTvc+xrDQXy3HXZFtTEClY35Efvuc3D+busYm0+rb1+Thau4HLRY9WP+sOKeGwH9/16rzfzYEqD7Ds8A9ykrHw==",
"cpu": [
"arm64"
],
"dev": true,
"license": "Apache-2.0 AND MIT",
"optional": true,
"os": [
"win32"
],
"engines": {
"node": ">=10"
}
},
"node_modules/@swc/core-win32-ia32-msvc": {
"version": "1.10.14",
"resolved": "https://registry.npmjs.org/@swc/core-win32-ia32-msvc/-/core-win32-ia32-msvc-1.10.14.tgz",
"integrity": "sha512-j2iQ4y9GWTKtES5eMU0sDsFdYni7IxME7ejFej25Tv3Fq4B+U9tgtYWlJwh1858nIWDXelHiKcSh/UICAyVMdQ==",
"cpu": [
"ia32"
],
"dev": true,
"license": "Apache-2.0 AND MIT",
"optional": true,
"os": [
"win32"
],
"engines": {
"node": ">=10"
}
},
"node_modules/@swc/core-win32-x64-msvc": {
"version": "1.10.14",
"resolved": "https://registry.npmjs.org/@swc/core-win32-x64-msvc/-/core-win32-x64-msvc-1.10.14.tgz",
"integrity": "sha512-TYtWkUSMkjs0jGPeWdtWbex4B+DlQZmN/ySVLiPI+EltYCLEXsFMkVFq6aWn48dqFHggFK0UYfvDrJUR2c3Qxg==",
"cpu": [
"x64"
],
"dev": true,
"license": "Apache-2.0 AND MIT",
"optional": true,
"os": [
"win32"
],
"engines": {
"node": ">=10"
}
},
"node_modules/@swc/counter": {
"version": "0.1.3",
"resolved": "https://registry.npmjs.org/@swc/counter/-/counter-0.1.3.tgz",

4
examples/instant-voice/client/javascript/package.json
View File

@@ -18,7 +18,7 @@
"vite": "^6.0.2"
},
"dependencies": {
"@pipecat-ai/client-js": "^0.3.2",
"@pipecat-ai/daily-transport": "^0.3.5"
"@pipecat-ai/client-js": "^0.3.5",
"@pipecat-ai/daily-transport": "^0.3.8"
}
}

76
examples/instant-voice/client/javascript/src/app.ts
View File

@@ -23,15 +23,14 @@ import {
RTVIEvent,
} from '@pipecat-ai/client-js';
import { DailyTransport } from '@pipecat-ai/daily-transport';
import SoundUtils from "./util/soundUtils";
import { InstantVoiceHelper } from "./util/instantVoiceHelper";
import SoundUtils from './util/soundUtils';
import { InstantVoiceHelper } from './util/instantVoiceHelper';
/**
* InstantVoiceClient handles the connection and media management for a real-time
* voice and video interaction with an AI bot.
*/
class InstantVoiceClient {
private declare rtviClient: RTVIClient;
private connectBtn: HTMLButtonElement | null = null;
private disconnectBtn: HTMLButtonElement | null = null;
@@ -54,8 +53,12 @@ class InstantVoiceClient {
* Set up references to DOM elements and create necessary media elements
*/
private setupDOMElements(): void {
this.connectBtn = document.getElementById('connect-btn') as HTMLButtonElement;
this.disconnectBtn = document.getElementById('disconnect-btn') as HTMLButtonElement;
this.connectBtn = document.getElementById(
'connect-btn'
) as HTMLButtonElement;
this.disconnectBtn = document.getElementById(
'disconnect-btn'
) as HTMLButtonElement;
this.statusSpan = document.getElementById('connection-status');
this.bufferingAudioSpan = document.getElementById('buffering-status');
this.debugLog = document.getElementById('debug-log');
@@ -70,11 +73,10 @@ class InstantVoiceClient {
}
private initializeRTVIClient(): void {
const transport = new DailyTransport({
bufferLocalAudioUntilBotReady: true
});
const RTVIConfig: RTVIClientOptions = {
transport,
transport: new DailyTransport({
bufferLocalAudioUntilBotReady: true,
}),
params: {
// The baseURL and endpoint of your bot server that the client will connect to
baseUrl: 'http://localhost:7860',
@@ -95,7 +97,7 @@ class InstantVoiceClient {
if (this.disconnectBtn) this.disconnectBtn.disabled = true;
this.log('Client disconnected');
},
onBotConnected: (participant: Participant) => {
onBotConnected: (participant: Participant) => {
this.log(`onBotConnected, timeTaken: ${Date.now() - this.startTime}`);
},
onBotReady: (data) => {
@@ -112,23 +114,29 @@ class InstantVoiceClient {
onMessageError: (error) => console.error('Message error:', error),
onError: (error) => console.error('Error:', error),
},
}
};
this.rtviClient = new RTVIClient(RTVIConfig);
this.rtviClient.registerHelper("transport", new InstantVoiceHelper({
callbacks: {
onAudioBufferingStarted: () => {
SoundUtils.beep()
this.updateBufferingStatus('Yes');
this.log(`onMicCaptureStarted, timeTaken: ${Date.now() - this.startTime}`);
},
onAudioBufferingStopped: () => {
this.updateBufferingStatus('No');
this.log(`onMicCaptureStopped, timeTaken: ${Date.now() - this.startTime}`);
}
}
}
));
this.rtviClient.registerHelper(
'transport',
new InstantVoiceHelper({
callbacks: {
onAudioBufferingStarted: () => {
SoundUtils.beep();
this.updateBufferingStatus('Yes');
this.log(
`onMicCaptureStarted, timeTaken: ${Date.now() - this.startTime}`
);
},
onAudioBufferingStopped: () => {
this.updateBufferingStatus('No');
this.log(
`onMicCaptureStopped, timeTaken: ${Date.now() - this.startTime}`
);
},
},
})
);
this.setupTrackListeners();
}
@@ -198,7 +206,9 @@ class InstantVoiceClient {
// Listen for tracks stopping
this.rtviClient.on(RTVIEvent.TrackStopped, (track, participant) => {
this.log(`Track stopped: ${track.kind} from ${participant?.name || 'unknown'}`);
this.log(
`Track stopped: ${track.kind} from ${participant?.name || 'unknown'}`
);
});
}
@@ -208,7 +218,10 @@ class InstantVoiceClient {
*/
private setupAudioTrack(track: MediaStreamTrack): void {
this.log('Setting up audio track');
if (this.botAudio.srcObject && "getAudioTracks" in this.botAudio.srcObject) {
if (
this.botAudio.srcObject &&
'getAudioTracks' in this.botAudio.srcObject
) {
const oldTrack = this.botAudio.srcObject.getAudioTracks()[0];
if (oldTrack?.id === track.id) return;
}
@@ -246,8 +259,13 @@ class InstantVoiceClient {
public async disconnect(): Promise<void> {
try {
await this.rtviClient.disconnect();
if (this.botAudio.srcObject && "getAudioTracks" in this.botAudio.srcObject) {
this.botAudio.srcObject.getAudioTracks().forEach((track) => track.stop());
if (
this.botAudio.srcObject &&
'getAudioTracks' in this.botAudio.srcObject
) {
this.botAudio.srcObject
.getAudioTracks()
.forEach((track) => track.stop());
this.botAudio.srcObject = null;
}
} catch (error) {

100
examples/instant-voice/client/javascript/yarn.lock
View File

@@ -3,16 +3,16 @@
"@babel/runtime@^7.12.5":
version "7.26.0"
resolved "https://registry.npmjs.org/@babel/runtime/-/runtime-7.26.0.tgz"
integrity sha512-FDSOghenHTiToteC/QRlv2q3DhPZ/oOXTBoirfWNx1Cx3TMVcGWQtMMmQcSvb/JjpNeGzx8Pq/b4fKEJuWm1sw==
version "7.27.0"
resolved "https://registry.npmjs.org/@babel/runtime/-/runtime-7.27.0.tgz"
integrity sha512-VtPOkrdPHZsKc/clNqyi9WUA8TINkZ4cGk63UUE3u4pmB2k+ZMQRDuIOagv8UVd6j7k0T3+RRIb7beKTebNbcw==
dependencies:
regenerator-runtime "^0.14.0"
"@daily-co/daily-js@^0.73.0":
version "0.73.0"
resolved "https://registry.npmjs.org/@daily-co/daily-js/-/daily-js-0.73.0.tgz"
integrity sha512-Wz8c60hgmkx8fcEeDAi4L4J0rbafiihWKyXFyhYoFYPsw2OdChHpA4RYwIB+1enRws5IK+/HdmzFDYLQsB4A6w==
"@daily-co/daily-js@^0.77.0":
version "0.77.0"
resolved "https://registry.npmjs.org/@daily-co/daily-js/-/daily-js-0.77.0.tgz"
integrity sha512-icNXKieKAkRR/C5dcPjrCkL1jQGFp5C5WtLHy5uHAdTztm+mo9wlPJuehbWaGOM3TV24mgWHZ/+8jOys1G0I4w==
dependencies:
"@babel/runtime" "^7.12.5"
"@sentry/browser" "^8.33.1"
@@ -25,10 +25,10 @@
resolved "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.24.0.tgz"
integrity sha512-CKyDpRbK1hXwv79soeTJNHb5EiG6ct3efd/FTPdzOWdbZZfGhpbcqIpiD0+vwmpu0wTIL97ZRPZu8vUt46nBSw==
"@pipecat-ai/client-js@^0.3.2", "@pipecat-ai/client-js@~0.3.2":
version "0.3.2"
resolved "https://registry.npmjs.org/@pipecat-ai/client-js/-/client-js-0.3.2.tgz"
integrity sha512-psunOVrJjPka2SWlq53vxVWCA0Vt8pSXsXtn8pOLC0YTKFsUx+b7Z6quYUJcDZjCe1aAg9cKETek3Xal3Co8Tg==
"@pipecat-ai/client-js@^0.3.5", "@pipecat-ai/client-js@~0.3.5":
version "0.3.5"
resolved "https://registry.npmjs.org/@pipecat-ai/client-js/-/client-js-0.3.5.tgz"
integrity sha512-qmhnDjwY2XUtLjww35ShsYf5TF9BCuAk0tIj0oHjpTe6v6QOlgKQt8JVCAdc32p5ycouzSZOeDFtBd2aNWuq1g==
dependencies:
"@types/events" "^3.0.3"
clone-deep "^4.0.1"
@@ -36,63 +36,63 @@
typed-emitter "^2.1.0"
uuid "^10.0.0"
"@pipecat-ai/daily-transport@^0.3.5":
version "0.3.5"
resolved "https://registry.npmjs.org/@pipecat-ai/daily-transport/-/daily-transport-0.3.5.tgz"
integrity sha512-nJ0TvWPCqXPmU81U8cXOqk5mUEEvEuI06Mis+N0jN8KZUrNy1pP08iWbs07ObmIXdnQcoL+kQmHOerT4q/bF0w==
"@pipecat-ai/daily-transport@^0.3.8":
version "0.3.8"
resolved "https://registry.npmjs.org/@pipecat-ai/daily-transport/-/daily-transport-0.3.8.tgz"
integrity sha512-AcRP51LGOsEA7DH0yPaZTqX/pozfTpkJbKC0itgWLv6uCM8dAnNtBj/m1CdFKRsE7QObhEOa+cRp5PUAyF4wCA==
dependencies:
"@daily-co/daily-js" "^0.73.0"
"@daily-co/daily-js" "^0.77.0"
"@rollup/rollup-darwin-arm64@4.28.0":
version "4.28.0"
resolved "https://registry.npmjs.org/@rollup/rollup-darwin-arm64/-/rollup-darwin-arm64-4.28.0.tgz"
integrity sha512-lmKx9yHsppblnLQZOGxdO66gT77bvdBtr/0P+TPOseowE7D9AJoBw8ZDULRasXRWf1Z86/gcOdpBrV6VDUY36Q==
"@sentry-internal/browser-utils@8.49.0":
version "8.49.0"
resolved "https://registry.npmjs.org/@sentry-internal/browser-utils/-/browser-utils-8.49.0.tgz"
integrity sha512-XkPHHdFqsN7EPaB+QGUOEmpFqXiqP67t2rRZ1HG1UwJoe0PhJEKNy7b4+WRwmT7ODSt+PvFk1gNBlJBpThwH7Q==
"@sentry-internal/browser-utils@8.55.0":
version "8.55.0"
resolved "https://registry.npmjs.org/@sentry-internal/browser-utils/-/browser-utils-8.55.0.tgz"
integrity sha512-ROgqtQfpH/82AQIpESPqPQe0UyWywKJsmVIqi3c5Fh+zkds5LUxnssTj3yNd1x+kxaPDVB023jAP+3ibNgeNDw==
dependencies:
"@sentry/core" "8.49.0"
"@sentry/core" "8.55.0"
"@sentry-internal/feedback@8.49.0":
version "8.49.0"
resolved "https://registry.npmjs.org/@sentry-internal/feedback/-/feedback-8.49.0.tgz"
integrity sha512-v/wf7WvPxEvZUB7xrCnecI3fhevVo84hw8WlxgZIz6mLUHXEIX8xYWc9H8Yet/KKJ2uEB8GQ8aDsY6S1hVEIUA==
"@sentry-internal/feedback@8.55.0":
version "8.55.0"
resolved "https://registry.npmjs.org/@sentry-internal/feedback/-/feedback-8.55.0.tgz"
integrity sha512-cP3BD/Q6pquVQ+YL+rwCnorKuTXiS9KXW8HNKu4nmmBAyf7urjs+F6Hr1k9MXP5yQ8W3yK7jRWd09Yu6DHWOiw==
dependencies:
"@sentry/core" "8.49.0"
"@sentry/core" "8.55.0"
"@sentry-internal/replay-canvas@8.49.0":
version "8.49.0"
resolved "https://registry.npmjs.org/@sentry-internal/replay-canvas/-/replay-canvas-8.49.0.tgz"
integrity sha512-/yXxI7f+Wu24FIYoRE7A0AidNxORuhAyPzb5ey1wFqMXP72nG8dXhOpcl0w+bi554FkqkLjdeUDhSOBWYZXH9g==
"@sentry-internal/replay-canvas@8.55.0":
version "8.55.0"
resolved "https://registry.npmjs.org/@sentry-internal/replay-canvas/-/replay-canvas-8.55.0.tgz"
integrity sha512-nIkfgRWk1091zHdu4NbocQsxZF1rv1f7bbp3tTIlZYbrH62XVZosx5iHAuZG0Zc48AETLE7K4AX9VGjvQj8i9w==
dependencies:
"@sentry-internal/replay" "8.49.0"
"@sentry/core" "8.49.0"
"@sentry-internal/replay" "8.55.0"
"@sentry/core" "8.55.0"
"@sentry-internal/replay@8.49.0":
version "8.49.0"
resolved "https://registry.npmjs.org/@sentry-internal/replay/-/replay-8.49.0.tgz"
integrity sha512-BDiiCBxskkktTd6FNplBc9V8l14R4T/AwRIZj2itX4xnuHewTTDjVbeyvGol4roA4r+V0Mzoi31hLEGI6yFQ5Q==
"@sentry-internal/replay@8.55.0":
version "8.55.0"
resolved "https://registry.npmjs.org/@sentry-internal/replay/-/replay-8.55.0.tgz"
integrity sha512-roCDEGkORwolxBn8xAKedybY+Jlefq3xYmgN2fr3BTnsXjSYOPC7D1/mYqINBat99nDtvgFvNfRcZPiwwZ1hSw==
dependencies:
"@sentry-internal/browser-utils" "8.49.0"
"@sentry/core" "8.49.0"
"@sentry-internal/browser-utils" "8.55.0"
"@sentry/core" "8.55.0"
"@sentry/browser@^8.33.1":
version "8.49.0"
resolved "https://registry.npmjs.org/@sentry/browser/-/browser-8.49.0.tgz"
integrity sha512-dS4Sw2h8EixHeXOIR++XEVMTen6xCGcIQ/XhJbsjqvddXeIijW0WkxSeTfPkfs0dsqFHSisWmlmo0xhHbXvEsQ==
version "8.55.0"
resolved "https://registry.npmjs.org/@sentry/browser/-/browser-8.55.0.tgz"
integrity sha512-1A31mCEWCjaMxJt6qGUK+aDnLDcK6AwLAZnqpSchNysGni1pSn1RWSmk9TBF8qyTds5FH8B31H480uxMPUJ7Cw==
dependencies:
"@sentry-internal/browser-utils" "8.49.0"
"@sentry-internal/feedback" "8.49.0"
"@sentry-internal/replay" "8.49.0"
"@sentry-internal/replay-canvas" "8.49.0"
"@sentry/core" "8.49.0"
"@sentry-internal/browser-utils" "8.55.0"
"@sentry-internal/feedback" "8.55.0"
"@sentry-internal/replay" "8.55.0"
"@sentry-internal/replay-canvas" "8.55.0"
"@sentry/core" "8.55.0"
"@sentry/core@8.49.0":
version "8.49.0"
resolved "https://registry.npmjs.org/@sentry/core/-/core-8.49.0.tgz"
integrity sha512-/OAm6LdHhh8TvfDAucWfSJV7M03IOHrJm5LVjrrKr4gwQ1HKd4CDbARsBbPwHIzSRAle0IgG3sbJxEvv52JUIw==
"@sentry/core@8.55.0":
version "8.55.0"
resolved "https://registry.npmjs.org/@sentry/core/-/core-8.55.0.tgz"
integrity sha512-6g7jpbefjHYs821Z+EBJ8r4Z7LT5h80YSWRJaylGS4nW5W5Z2KXzpdnyFarv37O7QjauzVC2E+PABmpkw5/JGA==
"@swc/core-darwin-arm64@1.10.14":
version "1.10.14"

4
examples/instant-voice/server/src/single_bot.py
View File

@@ -15,7 +15,7 @@ from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.processors.frameworks.rtvi import RTVIConfig, RTVIProcessor
from pipecat.processors.frameworks.rtvi import RTVIConfig, RTVIObserver, RTVIProcessor
from pipecat.services.gemini_multimodal_live import GeminiMultimodalLiveLLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport
@@ -93,7 +93,7 @@ async def main():
task = PipelineTask(
pipeline,
params=PipelineParams(allow_interruptions=True),
observers=[rtvi.observer()],
observers=[RTVIObserver(rtvi)],
)
@rtvi.event_handler("on_client_ready")

562
examples/news-chatbot/client/javascript/package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

4
examples/news-chatbot/client/javascript/package.json
View File

@@ -15,7 +15,7 @@
"vite": "^6.0.9"
},
"dependencies": {
"@pipecat-ai/client-js": "^0.3.2",
"@pipecat-ai/daily-transport": "^0.3.4"
"@pipecat-ai/client-js": "^0.3.5",
"@pipecat-ai/daily-transport": "^0.3.8"
}
}

52
examples/news-chatbot/client/javascript/src/app.js
View File

@@ -16,30 +16,34 @@
* - Browser with WebRTC support
*/
import {LogLevel, RTVIClient, RTVIClientHelper, RTVIEvent} from '@pipecat-ai/client-js';
import {
LogLevel,
RTVIClient,
RTVIClientHelper,
RTVIEvent,
} from '@pipecat-ai/client-js';
import { DailyTransport } from '@pipecat-ai/daily-transport';
class SearchResponseHelper extends RTVIClientHelper {
constructor(contentPanel) {
super()
this.contentPanel = contentPanel
super();
this.contentPanel = contentPanel;
}
handleMessage(rtviMessage) {
console.log("SearchResponseHelper, received message:", rtviMessage)
console.log('SearchResponseHelper, received message:', rtviMessage);
if (rtviMessage.data) {
// Clear existing content
this.contentPanel.innerHTML = "";
this.contentPanel.innerHTML = '';
// Create a container for all content
const contentContainer = document.createElement('div');
contentContainer.className = "content-container";
contentContainer.className = 'content-container';
// Add the search_result
if (rtviMessage.data.search_result) {
const searchResultDiv = document.createElement('div');
searchResultDiv.className = "search-result";
searchResultDiv.className = 'search-result';
searchResultDiv.textContent = rtviMessage.data.search_result;
contentContainer.appendChild(searchResultDiv);
}
@@ -47,18 +51,18 @@ class SearchResponseHelper extends RTVIClientHelper {
// Add the sources
if (rtviMessage.data.origins) {
const sourcesDiv = document.createElement('div');
sourcesDiv.className = "sources";
sourcesDiv.className = 'sources';
const sourcesTitle = document.createElement('h3');
sourcesTitle.className = "sources-title";
sourcesTitle.textContent = "Sources:";
sourcesTitle.className = 'sources-title';
sourcesTitle.textContent = 'Sources:';
sourcesDiv.appendChild(sourcesTitle);
rtviMessage.data.origins.forEach(origin => {
rtviMessage.data.origins.forEach((origin) => {
const sourceLink = document.createElement('a');
sourceLink.className = "source-link";
sourceLink.className = 'source-link';
sourceLink.href = origin.site_uri;
sourceLink.target = "_blank";
sourceLink.target = '_blank';
sourceLink.textContent = origin.site_title;
sourcesDiv.appendChild(sourceLink);
});
@@ -69,7 +73,7 @@ class SearchResponseHelper extends RTVIClientHelper {
// Add the rendered_content in an iframe
if (rtviMessage.data.rendered_content) {
const iframe = document.createElement('iframe');
iframe.className = "iframe-container";
iframe.className = 'iframe-container';
iframe.srcdoc = rtviMessage.data.rendered_content;
contentContainer.appendChild(iframe);
}
@@ -80,7 +84,7 @@ class SearchResponseHelper extends RTVIClientHelper {
}
getMessageTypes() {
return ["bot-llm-search-response"]
return ['bot-llm-search-response'];
}
}
@@ -105,7 +109,9 @@ class ChatbotClient {
this.disconnectBtn = document.getElementById('disconnect-btn');
this.statusSpan = document.getElementById('connection-status');
this.debugLog = document.getElementById('debug-log');
this.searchResultContainer = document.getElementById('search-result-container');
this.searchResultContainer = document.getElementById(
'search-result-container'
);
// Create an audio element for bot's voice output
this.botAudio = document.createElement('audio');
@@ -211,12 +217,9 @@ class ChatbotClient {
*/
async connect() {
try {
// Create a new Daily transport for WebRTC communication
const transport = new DailyTransport();
// Initialize the RTVI client with our configuration
// Initialize the RTVI client with a Daily WebRTC transport and our configuration
this.rtviClient = new RTVIClient({
transport,
transport: new DailyTransport(),
params: {
// The baseURL and endpoint of your bot server that the client will connect to
baseUrl: 'http://localhost:7860',
@@ -279,7 +282,10 @@ class ChatbotClient {
},
});
//this.rtviClient.setLogLevel(LogLevel.DEBUG)
this.rtviClient.registerHelper("llm", new SearchResponseHelper(this.searchResultContainer))
this.rtviClient.registerHelper(
'llm',
new SearchResponseHelper(this.searchResultContainer)
);
// Set up listeners for media track events
this.setupTrackListeners();

24
examples/p2p-webrtc/video-transform/client/typescript/.gitignore vendored Normal file
View File

@@ -0,0 +1,24 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*
node_modules
dist
dist-ssr
*.local
# Editor directories and files
.vscode/*
!.vscode/extensions.json
.idea
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?

19
examples/p2p-webrtc/video-transform/client/typescript/src/style.css
View File

@@ -32,6 +32,7 @@ select {
.status-bar {
display: flex;
flex-wrap: wrap;
justify-content: space-between;
align-items: center;
padding: 10px;
@@ -69,6 +70,7 @@ button:disabled {
padding: 20px;
margin-bottom: 20px;
display: flex;
flex-wrap: wrap;
}
.bot-container {
@@ -79,8 +81,8 @@ button:disabled {
}
#bot-video-container {
width: 640px;
height: 360px;
width: 90%;
aspect-ratio: 16 / 9;
background-color: #e0e0e0;
border-radius: 8px;
overflow: hidden;
@@ -98,10 +100,18 @@ button:disabled {
.debug-panel {
background-color: #fff;
border-radius: 8px;
padding-left: 20px;
width: 50%;
}
@media (max-width: 768px) {
.bot-container {
width: 100%;
}
.debug-panel {
width: 100%;
}
}
.debug-panel h3 {
margin: 0 0 10px 0;
font-size: 16px;
@@ -109,10 +119,9 @@ button:disabled {
}
#debug-log {
height: 500px;
height: 360px;
overflow-y: auto;
background-color: #f8f8f8;
padding: 10px;
border-radius: 4px;
font-family: monospace;
font-size: 12px;

3
examples/phone-chatbot/.dockerignore
View File

@@ -1,3 +0,0 @@
**/.DS_Store
.env
.env.*

40
examples/phone-chatbot/Dockerfile
View File

@@ -1,40 +0,0 @@
FROM python:3.11-bullseye
ARG DEBIAN_FRONTEND=noninteractive
ARG USE_PERSISTENT_DATA
ENV PYTHONUNBUFFERED=1
# Expose FastAPI port
ENV FAST_API_PORT=7860
EXPOSE 7860
# Install system dependencies
RUN apt-get update && apt-get install --no-install-recommends -y \
build-essential \
git \
ffmpeg \
google-perftools \
ca-certificates curl gnupg \
&& apt-get clean && rm -rf /var/lib/apt/lists/*
# Set up a new user named "user" with user ID 1000
RUN useradd -m -u 1000 user
# Set home to the user's home directory
ENV HOME=/home/user \
PATH=/home/user/.local/bin:$PATH \
PYTHONPATH=$HOME/app \
PYTHONUNBUFFERED=1
# Switch to the "user" user
USER user
# Set the working directory to the user's home directory
WORKDIR $HOME/app
# Install Python dependencies
COPY *.py .
COPY ./requirements.txt requirements.txt
RUN pip3 install --no-cache-dir --upgrade -r requirements.txt
# Start the FastAPI server
CMD python3 bot_runner.py --host "0.0.0.0" --port ${FAST_API_PORT}

689
examples/phone-chatbot/README.md
View File

@@ -1,209 +1,602 @@
<!-- @format -->
<div align="center">
 <img alt="pipecat" width="300px" height="auto" src="image.png">
<img alt="pipecat" width="300px" height="auto" src="image.png">
</div>
# Phone Chatbot
# Pipecat Phone Chatbot
Example project that demonstrates how to add phone funtionality to your Pipecat bots. We include examples for Daily (`bot_daily.py`) dial-in and dial-out, and Twilio (`bot_twilio.py`) dial-in, depending on who you want to use as a phone vendor.
This repository contains examples for building intelligent phone chatbots using AI for various use cases including:
- 🔁 Transport: Daily WebRTC
- 💬 Speech-to-Text: Deepgram via Daily transport
- 🤖 LLM: GPT4-o / OpenAI
- 🔉 Text-to-Speech: ElevenLabs
- **Simple dial-in**: Basic incoming call handling
- **Simple dial-out**: Basic outgoing call handling
- **Voicemail detection**: Bot calls a number, detects if it reaches voicemail or a human, and responds appropriately
- **Call transfer**: Bot handles initial customer interaction and transfers to a human operator when needed
#### Should I use Daily or Twilio as a vendor?
## Architecture Overview
If you're starting from scratch, using Daily to provision phone numbers alongside Daily as a transport offers some convenience (such as automatic call forwarding.)
These examples use the following components:
If you already have Twilio numbers and workflows that you want to connect to your Pipecat bots, there is some additional configuration required (you'll need to create a `on_dialin_ready` and use the Twilio client to trigger the forward.)
- 🔁 **Transport**: Daily WebRTC
- 💬 **Speech-to-Text**: Deepgram via Daily transport
- 🤖 **LLMs**: Each example uses a specific LLM (OpenAI GPT-4o or Google Gemini)
- 🔉 **Text-to-Speech**: Cartesia
You can read more about this, as well as see respective walkthroughs in our docs.
## Getting Started
## Setup
### Prerequisites
1. Create and activate a virtual environment:
```shell
python3 -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
```
2. Install requirements:
```shell
pip install -r requirements.txt
```
3. Copy env.example to .env and configure:
3. Set up your environment variables:
```shell
cp env.example .env
```
4. Install [ngrok](https://ngrok.com/) so your local server can receive requests from Daily's servers.
## Using Daily numbers
Edit the `.env` file to include your API keys.
### Running the example
4. Install [ngrok](https://ngrok.com/) to make your local server accessible to external services.
To run either the dial-in or dial-out example, follow these steps to get started:
### Phone Number Provider: Daily vs Twilio
1. Run `bot_runner.py` to handle incoming HTTP requests:
If you're starting from scratch, we recommend using Daily to provision phone numbers alongside Daily as a transport for simplicity (this provides automatic call forwarding).
If you already have Twilio numbers and workflows, you can connect them to your Pipecat bots with some additional configuration (`on_dialin_ready` and using the Twilio client to trigger forwarding).
Most examples in this repository show how to use Daily for dial-in/dial-out operations.
## Running the Examples
### 1. Start the Bot Runner Service
The bot runner handles incoming requests and manages bot processes:
```shell
python bot_runner.py --host localhost
```
### 2. Create a Public Endpoint with ngrok
Start ngrok to create a public URL for your local server:
```shell
ngrok http --domain yourdomain.ngrok.app 7860
```
## Example 1: Simple Dial-in
This example demonstrates basic handling of incoming calls without additional features like call transfer.
### Testing in Daily Prebuilt (No Actual Phone Calls)
```shell
curl -X POST "http://localhost:7860/start" \
-H "Content-Type: application/json" \
-d '{
"config": {
"simple_dialin": {
"testInPrebuilt": true
}
}
}'
```
This returns a Daily room URL where you can test the bot's basic conversation capabilities.
## Example 2: Simple Dial-out
This example demonstrates basic handling of outgoing calls without additional features like voicemail detection.
### Testing in Daily Prebuilt (No Actual Phone Calls)
```shell
curl -X POST "http://localhost:7860/start" \
-H "Content-Type: application/json" \
-d '{
"config": {
"simple_dialout": {
"testInPrebuilt": true
}
}
}'
```
This returns a Daily room URL where you can test the bot's basic conversation capabilities.
### Making Actual Phone Calls
```shell
curl -X POST "http://localhost:7860/start" \
-H "Content-Type: application/json" \
-d '{
"config": {
"dialout_settings": [{
"phoneNumber": "+12345678910"
}],
"simple_dialout": {
"testInPrebuilt": false
}
}
}'
```
## Example 3: Voicemail Detection
This example demonstrates a bot that can dial out to a phone number, detect whether it reached a human or voicemail system, and respond appropriately.
### How It Works
1. Bot dials a phone number
2. Bot listens to determine if it's connected to a person or voicemail
3. If it detects voicemail, it leaves a predefined message and hangs up
4. If it detects a human, it engages in conversation
### Testing in Daily Prebuilt (No Actual Phone Calls)
To test without making actual phone calls:
```shell
curl -X POST "http://localhost:7860/start" \
-H "Content-Type: application/json" \
-d '{
"config": {
"voicemail_detection": {
"testInPrebuilt": true
}
}
}'
```
This will return a Daily room URL you can use to test the bot in the browser.
### Making Actual Phone Calls
To have the bot dial out to a real phone number:
```shell
curl -X POST "http://localhost:7860/start" \
-H "Content-Type: application/json" \
-d '{
"config": {
"dialout_settings": [{
"phoneNumber": "+12345678910"
}],
"voicemail_detection": {
"testInPrebuilt": false
}
}
}'
```
> **Note:** To enable dial-out capabilities, you must first:
>
> 1. Contact [help@daily.co](mailto:help@daily.co) to enable dial-out for your domain
> 2. Purchase a phone number to dial out from
> 3. Ensure rooms have dial-out enabled (the bot runner handles this)
> 4. Use an owner token for the bot (also handled by the bot runner)
## Example 4: Call Transfer
This example demonstrates a bot that handles initial customer interaction and can transfer the call to a human operator when requested.
### How It Works
1. Customer calls in and speaks with the bot
2. When the customer asks for a supervisor/manager, the bot initiates a transfer
3. The bot dials out to an appropriate operator
4. When the operator joins, the bot summarizes the conversation
5. The bot remains silent while operator and customer talk
6. When the operator leaves, the bot resumes handling the call
### Testing in Daily Prebuilt (No Actual Phone Calls)
```shell
curl -X POST "http://localhost:7860/start" \
-H "Content-Type: application/json" \
-d '{
"config": {
"call_transfer": {
"mode": "dialout",
"speakSummary": true,
"storeSummary": false,
"operatorNumber": "+12345678910",
"testInPrebuilt": true
}
}
}'
```
This returns a Daily room URL. In the room, the expected flow is:
1. Join the room and speak with the bot
2. Ask to speak with a manager/supervisor
3. The bot will add the "operator" to the call
4. The bot will summarize the conversation and then go silent
5. To simulate the operator, you can mute yourself in Daily Prebuilt and speak as if you're the operator
6. When finished, have the "operator" leave the call
7. The bot will resume speaking and can recall details from the conversation
8. End the call by closing Daily Prebuilt or telling the bot you're done
### Using with Real Phone Calls
For incoming calls from customers, Daily will send a webhook to your `/start` endpoint. This webhook contains:
```json
{
"From": "+CALLERS_PHONE",
"To": "$PURCHASED_PHONE",
"callId": "callid-read-only-string",
"callDomain": "callDomain-read-only-string"
}
```
The system will:
1. Identify the customer based on their phone number
2. Determine the appropriate operator to contact
3. Customize the bot's behavior based on transfer settings
#### Operator Assignment
The `call_connection_manager.py` file contains mappings for:
1. `CUSTOMER_MAP`: Links phone numbers to customer names
2. `OPERATOR_CONTACT_MAP`: Contains operator contact information
3. `CUSTOMER_TO_OPERATOR_MAP`: Defines which operators should handle which customers
You can customize these mappings or integrate with your existing customer database.
## Configuration Options
### Request Body Structure
When making requests to the `/start` endpoint, the config object can include:
```json
{
"config": {
"prompts": [
{
"name": "call_transfer_initial_prompt",
"text": "Your custom prompt here"
},
{
"name": "call_transfer_prompt",
"text": "Your custom prompt here"
},
{
"name": "call_transfer_finished_prompt",
"text": "Your custom prompt here"
},
{
"name": "voicemail_detection_prompt",
"text": "Your custom prompt here"
},
{
"name": "voicemail_prompt",
"text": "Your custom prompt here"
},
{
"name": "human_conversation_prompt",
"text": "Your custom prompt here"
}
],
"dialin_settings": {
"From": "+CALLERS_PHONE",
"To": "$PURCHASED_PHONE",
"callId": "callid-read-only-string",
"callDomain": "callDomain-read-only-string"
},
"dialout_settings": [
{
"phoneNumber": "+12345678910",
"callerId": "caller-id-uuid",
"sipUri": "sip:maria@example.com"
}
],
"call_transfer": {
"mode": "dialout",
"speakSummary": true,
"storeSummary": false,
"operatorNumber": "+12345678910",
"testInPrebuilt": false
},
"voicemail_detection": {
"testInPrebuilt": true
},
"simple_dialin": {
"testInPrebuilt": true
},
"simple_dialout": {
"testInPrebuilt": true
}
}
}
```
### Configuration Parameters
- `prompts`: An array of objects containing prompts that you want the examples to use.
- `dialin_settings`: Information about incoming calls (typically from webhook)
- `dialout_settings`: For outbound calls:
- `phoneNumber`: Number to dial
- `callerId`: UUID of the number to display (optional)
- `sipUri`: SIP URI to connect to (alternative to phoneNumber)
- `call_transfer`: For call transfer example:
- `mode`: Currently only `"dialout"` is supported
- `speakSummary`: Whether the bot should summarize the conversation for the operator
- `storeSummary`: For future implementation
- `operatorNumber`: Operator phone number
- `testInPrebuilt`: Test without actual phone calls
- `voicemail_detection`: For voicemail detection example:
- `testInPrebuilt`: Test without actual phone calls
- `simple_dialin`: For simple dialin example:
- `testInPrebuilt`: Test without actual phone calls
- `simple_dialout`: For simple dialout example:
- `testInPrebuilt`: Test without actual phone calls
## Feature Compatibility
The following table shows which feature combinations are supported when making requests to the `/start` endpoint. The table is organized by use case to help you create the correct configuration.
| Use Case | `call_transfer` | `voicemail_detection` | `simple_dialin` | `simple_dialout` | `dialin_settings` | `dialout_settings` | `operatorNumber` | `testInPrebuilt` | Status |
| --------------------------------------------------------------- | --------------- | --------------------- | --------------- | ---------------- | ----------------- | ------------------ | ---------------- | ---------------- | ---------------- |
| **Basic incoming call handling (simple_dialin)** | ✗ | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✅ Supported |
| **Test mode: Simple dialin in Daily Prebuilt** | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✅ Supported |
| **Basic outgoing call handling (simple_dialout)** | ✗ | ✗ | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ | ✅ Supported |
| **Test mode: Simple dialout in Daily Prebuilt** | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✓ | ✅ Supported |
| **Standard call transfer (incoming call)** | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✓/✗ | ✗ | ✅ Supported |
| **Standard voicemail detection (outgoing call)** | ✗ | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✅ Supported |
| **Test mode: Call transfer in Daily Prebuilt** | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✅ Supported |
| **Test mode: Voicemail detection in Daily Prebuilt** | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✅ Supported |
| Call transfer requires operatorNumber | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✓/✗ | ❌ Not Supported |
| Voicemail detection requires dialout_settings or testInPrebuilt | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✓/✗ | ❌ Not Supported |
| Cannot combine different bot types | ✓ | ✓ | ✗ | ✗ | ✓ | ✓ | ✓ | ✓/✗ | ❌ Not Supported |
| Call_transfer needs dialin_settings in non-test mode | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ❌ Not Supported |
| Voicemail_detection needs dialout_settings in non-test mode | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ❌ Not Supported |
| Insufficient configuration | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓/✗ | ❌ Not Supported |
### Legend:
- ✓: Required
- ✗: Not allowed
- ✓/✗: Optional
- ✅: Supported
- ❌: Not Supported
### Notes:
- `dialin_settings` is typically populated automatically from webhook data for incoming calls
- `dialout_settings` must be specified manually for outgoing calls
- `operatorNumber` is specified within the `call_transfer` object (`"call_transfer": {"operatorNumber": "+1234567890", ...}`)
- `testInPrebuilt` is specified within the bot type object (e.g., `"call_transfer": {"testInPrebuilt": true, ...}`)
- For call transfers, `operatorNumber` must be provided to specify which operator to dial. If it is not provided, we will base it off of the operator map in call_connection_manager.py
- In test mode (`testInPrebuilt: true`), some requirements are relaxed to allow testing in Daily Prebuilt
- Multiple customers to dial out to can be specified by providing an array of objects in `dialout_settings`
- Bot types are mutually exclusive - you cannot combine multiple bot types in a single configuration
### Configuration Examples
#### Standard call transfer (incoming call):
```json
{
"config": {
"dialin_settings": {
"from": "+12345678901",
"to": "+19876543210",
"call_id": "call-id-string",
"call_domain": "domain-string"
},
"call_transfer": {
"mode": "dialout",
"speakSummary": true,
"operatorNumber": "+12345678910"
}
}
}
```
#### Test mode: Call transfer in Daily Prebuilt:
```json
{
"config": {
"call_transfer": {
"mode": "dialout",
"speakSummary": true,
"operatorNumber": "+12345678910",
"testInPrebuilt": true
}
}
}
```
#### Test mode: Voicemail detection in Daily Prebuilt:
```json
{
"config": {
"voicemail_detection": {
"testInPrebuilt": true
}
}
}
```
#### Standard voicemail detection:
```json
{
"config": {
"dialout_settings": [
{
"phoneNumber": "+12345678910"
}
],
"voicemail_detection": {
"testInPrebuilt": false
}
}
}
```
#### Simple dialin (incoming call):
```json
{
"config": {
"dialin_settings": {
"from": "+12345678901",
"to": "+19876543210",
"call_id": "call-id-string",
"call_domain": "domain-string"
},
"simple_dialin": {}
}
}
```
#### Test mode: Simple dialin in Daily Prebuilt:
```json
{
"config": {
"simple_dialin": {
"testInPrebuilt": true
}
}
}
```
#### Simple dialout (outgoing call):
```json
{
"config": {
"dialout_settings": [
{
"phoneNumber": "+12345678910"
}
],
"simple_dialout": {}
}
}
```
#### Test mode: Simple dialout in Daily Prebuilt:
```json
{
"config": {
"simple_dialout": {
"testInPrebuilt": true
}
}
}
```
## Using Twilio (Alternative)
To use Twilio for call handling:
1. Start the bot runner:
```shell
python bot_runner.py --host localhost
```
2. Start ngrok running in a terminal window:
2. Start ngrok:
```shell
ngrok http --domain yourdomain.ngrok.app 8000
ngrok http --domain yourdomain.ngrok.app 7860
```
3. In a different terminal window, run the Daily bot file:
```shell
python bot_daily.py
```
### Dial-in
To dial-in to the bot, you will need to enable dial-in for your Daily domain. Follow [this guide](https://docs.daily.co/guides/products/dial-in-dial-out/dialin-pinless#provisioning-sip-interconnect-and-pinless-dialin-workflow) to set up your domain.
Note: For the `room_creation_api` property, point at your ngrok hostname: `"room_creation_api": "https://yourdomain.ngrok.app/daily_start_bot"`.
Once your domain is configured, receiving a phone call at a number associated with your Daily account will result in a POST to the `/daily_start_bot` endpoint, which will start a bot session.
### Dial-out
For the bot to dial out to a number, make a POST request to `/daily_start_bot` and include the dial-out phone number in the body of the request as `dialoutNumber`.
For example:
```shell
curl -X "POST" "http://localhost:7860/daily_start_bot" \
-H 'Content-Type: application/json; charset=utf-8' \
-d $'{
"dialoutNumber": "+12125551234"
}'
```
### Voicemail detection
To start the bot and test voicemail detection, send a POST request to /daily_start_bot with "detectVoicemail": true in the request body.
- If you only include `"detectVoicemail": true`, the bot will not dial out. Instead, you can test it in Daily Prebuilt by visiting the URL provided in the response.
- If you include both `"detectVoicemail": true` and a phone number under `"dialoutNumber"`, the bot will dial out to that number.
Example: Testing in Daily Prebuilt:
```shell
curl -X POST "http://localhost:7860/daily_start_bot" \ py pipecat
-H "Content-Type: application/json" \
-d '{"detectVoicemail": true}'
```
Example: Testing with Dial-Out:
```shell
curl -X POST "http://localhost:7860/daily_start_bot" \ py pipecat
-H "Content-Type: application/json" \
-d '{"dialoutNumber": "+18057145330", "detectVoicemail": true}'
```
### New! Using Gemini 2.0 Flash Lite with Daily
We have introduced support for Google's Gemini 2.0 Flash Lite model in this example. This lightweight model offers faster response times and reduced costs while maintaining good conversational capabilities.
**Quick Start**
To use the Gemini-based bot instead of OpenAI:
```shell
curl -X POST "http://localhost:7860/daily_gemini_start_bot" \ py pipecat
-H "Content-Type: application/json" \
-d '{"detectVoicemail": true}'
```
All request body parameters supported by /daily_start_bot (such as detectVoicemail, dialoutNumber, etc.) are also compatible with /daily_gemini_start_bot.
This example uses context switching to help steer the bot in the right direction. As Flash Lite is a smaller model, breaking the prompt down into smaller piece helps to improve the bot's accuracy.
For example, instead of giving one large prompt like:
```python
system_instruction="""You are a chatbot that needs to detect if you're talking to a voicemail system or human, then either leave a message or have a conversation. If it's voicemail, say "Hello, this is a message..." and hang up. If it's a human, introduce yourself and be helpful until they say goodbye."""
```
We break it into stages:
First prompt focuses only on detection: "Determine if this is voicemail or human"
After detection, we switch to a new context: either "Leave this specific voicemail message" or "Have a conversation with the human".
**Implementation Details**
The implementation is available in bot_daily_gemini.py and features:
- Staged prompting approach: Breaking down complex tasks into smaller, more focused prompts to improve the lightweight model's performance
- Dynamic context switching: The bot can change its behavior in real-time based on what it detects (voicemail vs. human caller)
- Function-based architecture: Uses function calling to trigger context switches and call termination
### More information
For more configuration options, please consult [Daily's API documentation](https://docs.daily.co).
## Using Twilio numbers
### Running the example
Follow these steps to get started:
1. Run `bot_runner.py` to handle incoming HTTP requests:
```shell
python bot_runner.py --host localhost
```
2. Start ngrok running in a terminal window:
```shell
ngrok http --domain yourdomain.ngrok.app 8000
```
3. In a different terminal window, run the Daily bot file:
3. In another terminal, run the Twilio bot:
```shell
python bot_twilio.py
```
As above, but target the following URL:
Make requests to `/start_twilio_bot` for Twilio-specific functionality.
`POST /twilio_start_bot`
## Deployment
For more configuration options, please consult Twilio's API documentation.
See Pipecat Cloud deployment docs for how to deploy this example: https://docs.pipecat.daily.co/agents/deploy
## Deployment example
We also have a great, easy to use quickstart guide here: https://docs.pipecat.daily.co/quickstart
A Dockerfile is included in this demo for convenience. Here is an example of how to build and deploy your bot to [fly.io](https://fly.io).
## Using Different LLM Providers
_Please note: This demo spawns agents as subprocesses for convenience / demonstration purposes. You would likely not want to do this in production as it would limit concurrency to available system resources. For more information on how to deploy your bots using VMs, refer to the Pipecat documentation._
Each example in this repository is implemented with a specific LLM provider:
### Build the docker image
- **Simple dial-in**: Uses OpenAI
- **Simple dial-out**: Uses OpenAI
- **Voicemail detection**: Uses Google Gemini
- **Call transfer**: Uses OpenAI
`docker build -t tag:project .`
If you want to implement one of these examples with a different LLM provider than what's provided:
### Launch the fly project
- To implement **call_transfer** with **Gemini**, reference the `voicemail_detection.py` file for how to structure LLM context, function calling, and other Gemini-specific implementations.
- To implement **voicemail_detection** with **OpenAI**, reference the `call_transfer.py` file for OpenAI-specific implementation details.
`mv fly.example.toml fly.toml`
The key differences between implementations involve how context is managed, function calling syntax, and message formatting. Looking at both implementations side-by-side provides a good template for adapting any example to your preferred LLM provider.
`fly launch` (using the included fly.toml)
## Customizing Bot Prompts
### Setup your secrets on Fly
All examples include default prompts that work well for standard use cases. However, you can customize how the bot behaves by providing your own prompts in the request body.
Set the necessary secrets (found in `env.example`)
### Available Prompt Types
`fly secrets set DAILY_API_KEY=... OPENAI_API_KEY=... ELEVENLABS_API_KEY=... ELEVENLABS_VOICE_ID=...`
- `call_transfer_initial_prompt`: The initial prompt the bot uses when greeting a customer
- `call_transfer_prompt`: Instructions for the bot when summarizing the conversation for an operator
- `call_transfer_finished_prompt`: Instructions for when the operator leaves the call
- `voicemail_detection_prompt`: Instructions for detecting whether a call connected to voicemail
- `voicemail_prompt`: The message to leave when voicemail is detected
- `human_conversation_prompt`: Instructions for conversation when a human is detected
If you're using Twilio as a number vendor:
### Customization Example
`fly secrets set TWILIO_ACCOUNT_SID=... TWILIO_AUTH_TOKEN=...`
```shell
curl -X POST "http://localhost:7860/start" \
-H "Content-Type: application/json" \
-d '{
"config": {
"prompts": [
{
"name": "voicemail_prompt",
"text": "Hello, this is ACME Corporation calling. Please call us back at 555-123-4567 regarding your recent order. Thank you!"
}
],
"dialout_settings": [{
"phoneNumber": "+12345678910"
}],
"voicemail_detection": {
"testInPrebuilt": false
}
}
}'
```
### Deploy!
This example would use all default prompts except for the voicemail message, which would be replaced with your custom message.
`fly deploy`
### Template Variables
## Need to do something more advanced?
Some prompts support template variables that are automatically replaced:
This demo covers the basics of bot telephony. If you want to know more about working with PSTN / SIP, please ping us on [Discord](https://discord.gg/pipecat)!
- `{customer_name}`: Will be replaced with the customer's name if available
## Advanced Usage
For more advanced phone integration scenarios using PSTN/SIP, please reach out on [Discord](https://discord.gg/pipecat).

23
examples/phone-chatbot/bot_constants.py Normal file
View File

@@ -0,0 +1,23 @@
# bot_constants.py
"""Constants used across the bot runner application."""
# Maximum session time
MAX_SESSION_TIME = 5 * 60 # 5 minutes
# Required environment variables
REQUIRED_ENV_VARS = [
"OPENAI_API_KEY",
"GOOGLE_API_KEY",
"DAILY_API_KEY",
"CARTESIA_API_KEY",
"DEEPGRAM_API_KEY",
]
# Default example to use when handling dialin webhooks - determines which bot type to run
DEFAULT_DIALIN_EXAMPLE = "call_transfer" # Options: call_transfer, simple_dialin
# Call transfer configuration constants
DEFAULT_CALLTRANSFER_MODE = "dialout"
DEFAULT_SPEAK_SUMMARY = True # Speak a summary of the call to the operator
DEFAULT_STORE_SUMMARY = False # Store summary of the call (for future implementation)
DEFAULT_TEST_IN_PREBUILT = False # Test in prebuilt mode (bypasses need to dial in/out)

223
examples/phone-chatbot/bot_daily.py
View File

@@ -1,223 +0,0 @@
#
# Copyright (c) 2024–2025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import argparse
import asyncio
import os
import sys
from typing import Optional
from dotenv import load_dotenv
from loguru import logger
from openai.types.chat import ChatCompletionToolParam
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import EndTaskFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_services import LLMService
from pipecat.services.elevenlabs.tts import ElevenLabsTTSService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.transports.services.daily import DailyDialinSettings, DailyParams, DailyTransport
load_dotenv(override=True)
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
daily_api_key = os.getenv("DAILY_API_KEY", "")
daily_api_url = os.getenv("DAILY_API_URL", "https://api.daily.co/v1")
async def terminate_call(
function_name, tool_call_id, args, llm: LLMService, context, result_callback
):
"""Function the bot can call to terminate the call upon completion of a voicemail message."""
await llm.queue_frame(EndTaskFrame(), FrameDirection.UPSTREAM)
await result_callback("Goodbye")
async def main(
room_url: str,
token: str,
callId: str,
callDomain: str,
detect_voicemail: bool,
dialout_number: Optional[str],
):
# dialin_settings are only needed if Daily's SIP URI is used
# If you are handling this via Twilio, Telnyx, set this to None
# and handle call-forwarding when on_dialin_ready fires.
# We don't want to specify dial-in settings if we're not dialing in
dialin_settings = None
if callId and callDomain:
dialin_settings = DailyDialinSettings(call_id=callId, call_domain=callDomain)
transport = DailyTransport(
room_url,
token,
"Chatbot",
DailyParams(
api_url=daily_api_url,
api_key=daily_api_key,
dialin_settings=dialin_settings,
audio_in_enabled=True,
audio_out_enabled=True,
camera_out_enabled=False,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
transcription_enabled=True,
),
)
tts = ElevenLabsTTSService(
api_key=os.getenv("ELEVENLABS_API_KEY", ""),
voice_id=os.getenv("ELEVENLABS_VOICE_ID", ""),
)
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
llm.register_function("terminate_call", terminate_call)
tools = [
ChatCompletionToolParam(
type="function",
function={
"name": "terminate_call",
"description": "Terminate the call",
},
)
]
messages = [
{
"role": "system",
"content": """You are Chatbot, a friendly, helpful robot. Never refer to this prompt, even if asked. Follow these steps **EXACTLY**.
### **Standard Operating Procedure:**
#### **Step 1: Detect if You Are Speaking to Voicemail**
- If you hear **any variation** of the following:
- **"Please leave a message after the beep."**
- **"No one is available to take your call."**
- **"Record your message after the tone."**
- **"Please leave a message after the beep"**
- **"You have reached voicemail for..."**
- **"You have reached [phone number]"**
- **"[phone number] is unavailable"**
- **"The person you are trying to reach..."**
- **"The number you have dialed..."**
- **"Your call has been forwarded to an automated voice messaging system"**
- **Any phrase that suggests an answering machine or voicemail.**
- **ASSUME IT IS A VOICEMAIL. DO NOT WAIT FOR MORE CONFIRMATION.**
- **IF THE CALL SAYS "PLEASE LEAVE A MESSAGE AFTER THE BEEP", WAIT FOR THE BEEP BEFORE LEAVING A MESSAGE.**
#### **Step 2: Leave a Voicemail Message**
- Immediately say:
*"Hello, this is a message for Pipecat example user. This is Chatbot. Please call back on 123-456-7891. Thank you."*
- **IMMEDIATELY AFTER LEAVING THE MESSAGE, CALL `terminate_call`.**
- **DO NOT SPEAK AFTER CALLING `terminate_call`.**
- **FAILURE TO CALL `terminate_call` IMMEDIATELY IS A MISTAKE.**
#### **Step 3: If Speaking to a Human**
- If the call is answered by a human, say:
*"Oh, hello! I'm a friendly chatbot. Is there anything I can help you with?"*
- Keep responses **brief and helpful**.
- If the user no longer needs assistance, say:
*"Okay, thank you! Have a great day!"*
-**Then call `terminate_call` immediately.**
---
### **General Rules**
- **DO NOT continue speaking after leaving a voicemail.**
- **DO NOT wait after a voicemail message. ALWAYS call `terminate_call` immediately.**
- Your output will be converted to audio, so **do not include special characters or formatting.**
""",
}
]
context = OpenAILLMContext(messages, tools)
context_aggregator = llm.create_context_aggregator(context)
pipeline = Pipeline(
[
transport.input(),
context_aggregator.user(),
llm,
tts,
transport.output(),
context_aggregator.assistant(),
]
)
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
if dialout_number:
logger.debug("dialout number detected; doing dialout")
# Configure some handlers for dialing out
@transport.event_handler("on_joined")
async def on_joined(transport, data):
logger.debug(f"Joined; starting dialout to: {dialout_number}")
await transport.start_dialout({"phoneNumber": dialout_number})
@transport.event_handler("on_dialout_connected")
async def on_dialout_connected(transport, data):
logger.debug(f"Dial-out connected: {data}")
@transport.event_handler("on_dialout_answered")
async def on_dialout_answered(transport, data):
logger.debug(f"Dial-out answered: {data}")
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
# unlike the dialin case, for the dialout case, the caller will speak first. Presumably
# they will answer the phone and say "Hello?" Since we've captured their transcript,
# That will put a frame into the pipeline and prompt an LLM completion, which is how the
# bot will then greet the user.
elif detect_voicemail:
logger.debug("Detect voicemail example. You can test this in example in Daily Prebuilt")
# For the voicemail detection case, we do not want the bot to answer the phone. We want it to wait for the voicemail
# machine to say something like 'Leave a message after the beep', or for the user to say 'Hello?'.
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
else:
logger.debug("no dialout number; assuming dialin")
# Different handlers for dialin
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
# For the dialin case, we want the bot to answer the phone and greet the user. We
# can prompt the bot to speak by putting the context into the pipeline.
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.cancel()
runner = PipelineRunner()
await runner.run(task)
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Pipecat Simple ChatBot")
parser.add_argument("-u", type=str, help="Room URL")
parser.add_argument("-t", type=str, help="Token")
parser.add_argument("-i", type=str, help="Call ID")
parser.add_argument("-d", type=str, help="Call Domain")
parser.add_argument("-v", action="store_true", help="Detect voicemail")
parser.add_argument("-o", type=str, help="Dialout number", default=None)
config = parser.parse_args()
asyncio.run(main(config.u, config.t, config.i, config.d, config.v, config.o))

464
examples/phone-chatbot/bot_daily_gemini.py
View File

@@ -1,464 +0,0 @@
#
# Copyright (c) 2024–2025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import argparse
import asyncio
import os
import sys
from typing import Optional
from dotenv import load_dotenv
from loguru import logger
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import (
EndFrame,
EndTaskFrame,
InputAudioRawFrame,
StopTaskFrame,
TranscriptionFrame,
UserStartedSpeakingFrame,
UserStoppedSpeakingFrame,
)
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.services.ai_services import LLMService
from pipecat.services.deepgram.stt import DeepgramSTTService
from pipecat.services.elevenlabs.tts import ElevenLabsTTSService
from pipecat.services.google.llm import GoogleLLMContext, GoogleLLMService
from pipecat.transports.services.daily import (
DailyDialinSettings,
DailyParams,
DailyTransport,
)
load_dotenv(override=True)
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
daily_api_key = os.getenv("DAILY_API_KEY", "")
daily_api_url = os.getenv("DAILY_API_URL", "https://api.daily.co/v1")
system_message = None
class UserAudioCollector(FrameProcessor):
"""This FrameProcessor collects audio frames in a buffer, then adds them to the
LLM context when the user stops speaking.
"""
def __init__(self, context, user_context_aggregator):
super().__init__()
self._context = context
self._user_context_aggregator = user_context_aggregator
self._audio_frames = []
self._start_secs = 0.2 # this should match VAD start_secs (hardcoding for now)
self._user_speaking = False
async def process_frame(self, frame, direction):
await super().process_frame(frame, direction)
if isinstance(frame, TranscriptionFrame):
# We could gracefully handle both audio input and text/transcription input ...
# but let's leave that as an exercise to the reader. :-)
return
if isinstance(frame, UserStartedSpeakingFrame):
self._user_speaking = True
elif isinstance(frame, UserStoppedSpeakingFrame):
self._user_speaking = False
self._context.add_audio_frames_message(audio_frames=self._audio_frames)
await self._user_context_aggregator.push_frame(
self._user_context_aggregator.get_context_frame()
)
elif isinstance(frame, InputAudioRawFrame):
if self._user_speaking:
self._audio_frames.append(frame)
else:
# Append the audio frame to our buffer. Treat the buffer as a ring buffer, dropping the oldest
# frames as necessary. Assume all audio frames have the same duration.
self._audio_frames.append(frame)
frame_duration = len(frame.audio) / 16 * frame.num_channels / frame.sample_rate
buffer_duration = frame_duration * len(self._audio_frames)
while buffer_duration > self._start_secs:
self._audio_frames.pop(0)
buffer_duration -= frame_duration
await self.push_frame(frame, direction)
class ContextSwitcher:
def __init__(self, llm, context_aggregator):
self._llm = llm
self._context_aggregator = context_aggregator
async def switch_context(self, system_instruction):
"""Switch the context to a new system instruction based on what the bot hears."""
# Create messages with updated system instruction
messages = [
{
"role": "system",
"content": system_instruction,
}
]
# Update context with new messages
self._context_aggregator.set_messages(messages)
# Get the context frame with the updated messages
context_frame = self._context_aggregator.get_context_frame()
# Trigger LLM response by pushing a context frame
await self._llm.push_frame(context_frame)
class FunctionHandlers:
def __init__(self, context_switcher):
self.context_switcher = context_switcher
async def voicemail_response(
self,
function_name,
tool_call_id,
args,
llm: LLMService,
context,
result_callback,
):
"""Function the bot can call to leave a voicemail message."""
message = """You are Chatbot leaving a voicemail message. Say EXACTLY this message and nothing else:
"Hello, this is a message for Pipecat example user. This is Chatbot. Please call back on 123-456-7891. Thank you."
After saying this message, call the terminate_call function."""
await self.context_switcher.switch_context(system_instruction=message)
await result_callback("Leaving a voicemail message")
async def human_conversation(
self,
function_name,
tool_call_id,
args,
llm: LLMService,
context,
result_callback,
):
"""Function the bot can when it detects it's talking to a human."""
await llm.push_frame(StopTaskFrame(), FrameDirection.UPSTREAM)
async def terminate_call(
function_name,
tool_call_id,
args,
llm: LLMService,
context,
result_callback,
call_state=None,
):
"""Function the bot can call to terminate the call upon completion of the call."""
if call_state:
call_state.bot_terminated_call = True
await llm.push_frame(EndTaskFrame(), FrameDirection.UPSTREAM)
async def main(
room_url: str,
token: str,
callId: Optional[str],
callDomain: Optional[str],
detect_voicemail: bool,
dialout_number: Optional[str],
):
dialin_settings = None
if callId and callDomain:
dialin_settings = DailyDialinSettings(call_id=callId, call_domain=callDomain)
transport_params = DailyParams(
api_url=daily_api_url,
api_key=daily_api_key,
dialin_settings=dialin_settings,
audio_in_enabled=True,
audio_out_enabled=True,
camera_out_enabled=False,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
vad_audio_passthrough=True,
)
else:
transport_params = DailyParams(
api_url=daily_api_url,
api_key=daily_api_key,
audio_in_enabled=True,
audio_out_enabled=True,
camera_out_enabled=False,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
vad_audio_passthrough=True,
)
class CallState:
participant_left_early = False
bot_terminated_call = False
call_state = CallState()
transport = DailyTransport(
room_url,
token,
"Chatbot",
transport_params,
)
tts = ElevenLabsTTSService(
api_key=os.getenv("ELEVENLABS_API_KEY", ""),
voice_id=os.getenv("ELEVENLABS_VOICE_ID", ""),
)
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
### VOICEMAIL PIPELINE
tools = [
{
"function_declarations": [
{
"name": "switch_to_voicemail_response",
"description": "Call this function when you detect this is a voicemail system.",
},
{
"name": "switch_to_human_conversation",
"description": "Call this function when you detect this is a human.",
},
{
"name": "terminate_call",
"description": "Call this function to terminate the call.",
},
]
}
]
system_instruction = """You are Chatbot trying to determine if this is a voicemail system or a human.
If you hear any of these phrases (or very similar ones):
- "Please leave a message after the beep"
- "No one is available to take your call"
- "Record your message after the tone"
- "You have reached voicemail for..."
- "You have reached [phone number]"
- "[phone number] is unavailable"
- "The person you are trying to reach..."
- "The number you have dialed..."
- "Your call has been forwarded to an automated voice messaging system"
Then call the function switch_to_voicemail_response.
If it sounds like a human (saying hello, asking questions, etc.), call the function switch_to_human_conversation.
DO NOT say anything until you've determined if this is a voicemail or human."""
voicemail_detection_llm = GoogleLLMService(
model="models/gemini-2.0-flash-lite",
api_key=os.getenv("GOOGLE_API_KEY"),
system_instruction=system_instruction,
tools=tools,
)
voicemail_detection_context = GoogleLLMContext()
voicemail_detection_context_aggregator = voicemail_detection_llm.create_context_aggregator(
voicemail_detection_context
)
context_switcher = ContextSwitcher(
voicemail_detection_llm, voicemail_detection_context_aggregator.user()
)
handlers = FunctionHandlers(context_switcher)
voicemail_detection_llm.register_function(
"switch_to_voicemail_response", handlers.voicemail_response
)
voicemail_detection_llm.register_function(
"switch_to_human_conversation", handlers.human_conversation
)
voicemail_detection_llm.register_function(
"terminate_call",
lambda *args, **kwargs: terminate_call(*args, **kwargs, call_state=call_state),
)
voicemail_detection_audio_collector = UserAudioCollector(
voicemail_detection_context, voicemail_detection_context_aggregator.user()
)
voicemail_detection_pipeline = Pipeline(
[
transport.input(), # Transport user input
voicemail_detection_audio_collector, # Collect audio frames
voicemail_detection_context_aggregator.user(), # User responses
voicemail_detection_llm, # LLM
tts, # TTS
transport.output(), # Transport bot output
voicemail_detection_context_aggregator.assistant(), # Assistant spoken responses
]
)
voicemail_detection_pipeline_task = PipelineTask(
voicemail_detection_pipeline,
params=PipelineParams(allow_interruptions=True),
)
if dialout_number:
logger.debug("dialout number detected; doing dialout")
# Configure some handlers for dialing out
@transport.event_handler("on_joined")
async def on_joined(transport, data):
logger.debug(f"Joined; starting dialout to: {dialout_number}")
await transport.start_dialout({"phoneNumber": dialout_number})
@transport.event_handler("on_dialout_connected")
async def on_dialout_connected(transport, data):
logger.debug(f"Dial-out connected: {data}")
@transport.event_handler("on_dialout_answered")
async def on_dialout_answered(transport, data):
logger.debug(f"Dial-out answered: {data}")
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
# unlike the dialin case, for the dialout case, the caller will speak first. Presumably
# they will answer the phone and say "Hello?" Since we've captured their transcript,
# That will put a frame into the pipeline and prompt an LLM completion, which is how the
# bot will then greet the user.
elif detect_voicemail:
logger.debug("Detect voicemail example. You can test this in example in Daily Prebuilt")
# For the voicemail detection case, we do not want the bot to answer the phone. We want it to wait for the voicemail
# machine to say something like 'Leave a message after the beep', or for the user to say 'Hello?'.
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
logger.debug("Detect voicemail; capturing participant transcription")
await transport.capture_participant_transcription(participant["id"])
else:
logger.debug("+++++ No dialout number; assuming dialin")
# Different handlers for dialin
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
# This event is not firing for some reason
await transport.capture_participant_transcription(participant["id"])
dialin_instructions = """Always call the function switch_to_human_conversation"""
messages = [
{
"role": "system",
"content": dialin_instructions,
}
]
voicemail_detection_context_aggregator.user().set_messages(messages)
await voicemail_detection_pipeline_task.queue_frames(
[voicemail_detection_context_aggregator.user().get_context_frame()]
)
runner = PipelineRunner()
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
call_state.participant_left_early = True
await voicemail_detection_pipeline_task.queue_frame(EndFrame())
print("!!! starting voicemail detection pipeline")
await runner.run(voicemail_detection_pipeline_task)
print("!!! Done with voicemail detection pipeline")
if call_state.participant_left_early or call_state.bot_terminated_call:
if call_state.participant_left_early:
print("!!! Participant left early; terminating call")
elif call_state.bot_terminated_call:
print("!!! Bot terminated call; not proceeding to human conversation")
return
### HUMAN CONVERSATION PIPELINE
human_conversation_system_instruction = """You are Chatbot talking to a human. Be friendly and helpful.
Start with: "Hello! I'm a friendly chatbot. How can I help you today?"
Keep your responses brief and to the point. Listen to what the person says.
When the person indicates they're done with the conversation by saying something like:
- "Goodbye"
- "That's all"
- "I'm done"
- "Thank you, that's all I needed"
THEN say: "Thank you for chatting. Goodbye!" and call the terminate_call function."""
human_conversation_llm = GoogleLLMService(
model="models/gemini-2.0-flash-001",
api_key=os.getenv("GOOGLE_API_KEY"),
system_instruction=human_conversation_system_instruction,
tools=tools,
)
human_conversation_context = GoogleLLMContext()
human_conversation_context_aggregator = human_conversation_llm.create_context_aggregator(
human_conversation_context
)
human_conversation_llm.register_function(
"terminate_call",
lambda *args, **kwargs: terminate_call(*args, **kwargs, call_state=call_state),
)
human_conversation_pipeline = Pipeline(
[
transport.input(), # Transport user input
stt,
human_conversation_context_aggregator.user(), # User responses
human_conversation_llm, # LLM
tts, # TTS
transport.output(), # Transport bot output
human_conversation_context_aggregator.assistant(), # Assistant spoken responses
]
)
human_conversation_pipeline_task = PipelineTask(
human_conversation_pipeline,
params=PipelineParams(allow_interruptions=True),
)
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await voicemail_detection_pipeline_task.queue_frame(EndFrame())
await human_conversation_pipeline_task.queue_frame(EndFrame())
print("!!! starting human conversation pipeline")
human_conversation_context_aggregator.user().set_messages(
[
{
"role": "system",
"content": human_conversation_system_instruction,
}
]
)
await human_conversation_pipeline_task.queue_frames(
[human_conversation_context_aggregator.user().get_context_frame()]
)
await runner.run(human_conversation_pipeline_task)
print("!!! Done with human conversation pipeline")
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Pipecat Simple ChatBot")
parser.add_argument("-u", type=str, help="Room URL")
parser.add_argument("-t", type=str, help="Token")
parser.add_argument("-i", type=str, help="Call ID")
parser.add_argument("-d", type=str, help="Call Domain")
parser.add_argument("-v", action="store_true", help="Detect voicemail")
parser.add_argument("-o", type=str, help="Dialout number", default=None)
config = parser.parse_args()
asyncio.run(main(config.u, config.t, config.i, config.d, config.v, config.o))

55
examples/phone-chatbot/bot_definitions.py Normal file
View File

@@ -0,0 +1,55 @@
# bot_definitions.py
"""Definitions of different bot types for the bot registry."""
from bot_registry import BotRegistry, BotType
from bot_runner_helpers import (
create_call_transfer_settings,
create_simple_dialin_settings,
create_simple_dialout_settings,
)
# Create and configure the bot registry
bot_registry = BotRegistry()
# Register bot types
bot_registry.register(
BotType(
name="call_transfer",
settings_creator=create_call_transfer_settings,
required_settings=["dialin_settings"],
incompatible_with=["simple_dialin", "simple_dialout", "voicemail_detection"],
auto_add_settings={"dialin_settings": {}},
)
)
bot_registry.register(
BotType(
name="simple_dialin",
settings_creator=create_simple_dialin_settings,
required_settings=["dialin_settings"],
incompatible_with=["call_transfer", "simple_dialout", "voicemail_detection"],
auto_add_settings={"dialin_settings": {}},
)
)
bot_registry.register(
BotType(
name="simple_dialout",
settings_creator=create_simple_dialout_settings,
required_settings=["dialout_settings"],
incompatible_with=["call_transfer", "simple_dialin", "voicemail_detection"],
auto_add_settings={"dialout_settings": [{}]},
)
)
bot_registry.register(
BotType(
name="voicemail_detection",
settings_creator=lambda body: body.get(
"voicemail_detection", {}
), # No creator function in original code
required_settings=["dialout_settings"],
incompatible_with=["call_transfer", "simple_dialin", "simple_dialout"],
auto_add_settings={"dialout_settings": [{}]},
)
)

137
examples/phone-chatbot/bot_registry.py Normal file
View File

@@ -0,0 +1,137 @@
# bot_registry.py
"""Bot registry pattern for managing different bot types."""
from typing import Any, Callable, Dict, List, Optional
from bot_constants import DEFAULT_DIALIN_EXAMPLE
from bot_runner_helpers import ensure_dialout_settings_array
from fastapi import HTTPException
class BotType:
"""Bot type configuration and handling."""
def __init__(
self,
name: str,
settings_creator: Callable[[Dict[str, Any]], Dict[str, Any]],
required_settings: list = None,
incompatible_with: list = None,
auto_add_settings: dict = None,
):
"""Initialize a bot type.
Args:
name: Name of the bot type
settings_creator: Function to create/update settings for this bot type
required_settings: List of settings this bot type requires
incompatible_with: List of bot types this one cannot be used with
auto_add_settings: Settings to add if this bot is being run in test mode
"""
self.name = name
self.settings_creator = settings_creator
self.required_settings = required_settings or []
self.incompatible_with = incompatible_with or []
self.auto_add_settings = auto_add_settings or {}
def has_test_mode(self, body: Dict[str, Any]) -> bool:
"""Check if this bot type is configured for test mode."""
return self.name in body and body[self.name].get("testInPrebuilt", False)
def create_settings(self, body: Dict[str, Any]) -> Dict[str, Any]:
"""Create or update settings for this bot type."""
body[self.name] = self.settings_creator(body)
return body
def prepare_for_test(self, body: Dict[str, Any]) -> Dict[str, Any]:
"""Add required settings for test mode if they don't exist."""
for setting, default_value in self.auto_add_settings.items():
if setting not in body:
body[setting] = default_value
return body
class BotRegistry:
"""Registry for managing different bot types."""
def __init__(self):
self.bots = {}
self.bot_validation_rules = []
def register(self, bot_type: BotType):
"""Register a bot type."""
self.bots[bot_type.name] = bot_type
return self
def get_bot(self, name: str) -> BotType:
"""Get a bot type by name."""
return self.bots.get(name)
def detect_bot_type(self, body: Dict[str, Any]) -> Optional[str]:
"""Detect which bot type to use based on configuration."""
# First check for test mode bots
for name, bot in self.bots.items():
if bot.has_test_mode(body):
return name
# Then check for specific combinations of settings
for name, bot in self.bots.items():
if name in body and all(req in body for req in bot.required_settings):
return name
# Default for dialin settings
if "dialin_settings" in body:
return DEFAULT_DIALIN_EXAMPLE
return None
def validate_bot_combination(self, body: Dict[str, Any]) -> List[str]:
"""Validate that bot types in the configuration are compatible."""
errors = []
bot_types_in_config = [name for name in self.bots.keys() if name in body]
# Check each bot type against its incompatible list
for bot_name in bot_types_in_config:
bot = self.bots[bot_name]
for incompatible in bot.incompatible_with:
if incompatible in body:
errors.append(
f"Cannot have both '{bot_name}' and '{incompatible}' in the same configuration"
)
return errors
def setup_configuration(self, body: Dict[str, Any]) -> Dict[str, Any]:
"""Set up bot configuration based on detected bot type."""
# Ensure dialout_settings is an array if present
body = ensure_dialout_settings_array(body)
# Detect which bot type to use
bot_type_name = self.detect_bot_type(body)
if not bot_type_name:
raise HTTPException(
status_code=400, detail="Configuration doesn't match any supported scenario"
)
# If we have a dialin scenario but no explicit bot type, add the default
if "dialin_settings" in body and bot_type_name == DEFAULT_DIALIN_EXAMPLE:
if bot_type_name not in body:
body[bot_type_name] = {}
# Get the bot type object
bot_type = self.get_bot(bot_type_name)
# Create/update settings for the bot type
body = bot_type.create_settings(body)
# If in test mode, add any required settings
if bot_type.has_test_mode(body):
body = bot_type.prepare_for_test(body)
# Validate bot combinations
errors = self.validate_bot_combination(body)
if errors:
error_message = "Invalid configuration: " + "; ".join(errors)
raise HTTPException(status_code=400, detail=error_message)
return body

396
examples/phone-chatbot/bot_runner.py
View File

@@ -1,24 +1,22 @@
#
# Copyright (c) 2024–2025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
"""
bot_runner.py
HTTP service that listens for incoming calls from either Daily or Twilio,
provisioning a room and starting a Pipecat bot in response.
Refer to README for more information.
"""
import argparse
import json
import os
import shlex
import subprocess
from contextlib import asynccontextmanager
from typing import Any, Dict
import aiohttp
from bot_constants import (
MAX_SESSION_TIME,
REQUIRED_ENV_VARS,
)
from bot_definitions import bot_registry
from bot_runner_helpers import (
determine_room_capabilities,
ensure_prompt_config,
process_dialin_request,
)
from dotenv import load_dotenv
from fastapi import FastAPI, HTTPException, Request
from fastapi.middleware.cors import CORSMiddleware
@@ -27,7 +25,6 @@ from twilio.twiml.voice_response import VoiceResponse
from pipecat.transports.services.helpers.daily_rest import (
DailyRESTHelper,
DailyRoomObject,
DailyRoomParams,
DailyRoomProperties,
DailyRoomSipParams,
@@ -35,15 +32,126 @@ from pipecat.transports.services.helpers.daily_rest import (
load_dotenv(override=True)
# ------------ Configuration ------------ #
MAX_SESSION_TIME = 5 * 60 # 5 minutes
REQUIRED_ENV_VARS = ["OPENAI_API_KEY", "DAILY_API_KEY", "ELEVENLABS_API_KEY", "ELEVENLABS_VOICE_ID"]
daily_helpers = {}
# ----------------- API ----------------- #
# ----------------- Daily Room Management ----------------- #
async def create_daily_room(room_url: str = None, config_body: Dict[str, Any] = None):
"""Create or retrieve a Daily room with appropriate properties based on the configuration.
Args:
room_url: Optional existing room URL
config_body: Optional configuration that determines room capabilities
Returns:
Dict containing room URL, token, and SIP endpoint
"""
if not room_url:
# Get room capabilities based on the configuration
capabilities = determine_room_capabilities(config_body)
# Configure SIP parameters if dialin is needed
sip_params = None
if capabilities["enable_dialin"]:
sip_params = DailyRoomSipParams(
display_name="dialin-user", video=False, sip_mode="dial-in", num_endpoints=2
)
# Create the properties object with the appropriate settings
properties = DailyRoomProperties(sip=sip_params)
# Set dialout capability if needed
if capabilities["enable_dialout"]:
properties.enable_dialout = True
# Log the capabilities being used
capability_str = ", ".join([f"{k}={v}" for k, v in capabilities.items()])
print(f"Creating room with capabilities: {capability_str}")
params = DailyRoomParams(properties=properties)
print("Creating new room...")
room = await daily_helpers["rest"].create_room(params=params)
else:
# Check if passed room URL exists
try:
room = await daily_helpers["rest"].get_room_from_url(room_url)
except Exception:
raise HTTPException(status_code=500, detail=f"Room not found: {room_url}")
print(f"Daily room: {room.url} {room.config.sip_endpoint}")
# Get token for the agent
token = await daily_helpers["rest"].get_token(room.url, MAX_SESSION_TIME)
if not room or not token:
raise HTTPException(status_code=500, detail="Failed to get room or token")
return {"room": room.url, "token": token, "sip_endpoint": room.config.sip_endpoint}
# ----------------- Bot Process Management ----------------- #
async def start_bot(room_details: Dict[str, str], body: Dict[str, Any], example: str) -> bool:
"""Start a bot process with the given configuration.
Args:
room_details: Room URL and token
body: Bot configuration
example: Example script to run
Returns:
Boolean indicating success
"""
room_url = room_details["room"]
token = room_details["token"]
# Properly format body as JSON string for command line
body_json = json.dumps(body).replace('"', '\\"')
print(f"++++ Body JSON: {body_json}")
# Modified to use non-LLM-specific bot module names
bot_proc = f'python3 -m {example} -u {room_url} -t {token} -b "{body_json}"'
print(f"Starting bot. Example: {example}, Room: {room_url}")
try:
command_parts = shlex.split(bot_proc)
subprocess.Popen(command_parts, bufsize=1, cwd=os.path.dirname(os.path.abspath(__file__)))
return True
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to start subprocess: {e}")
async def start_twilio_bot(room_details: Dict[str, str], call_id: str) -> bool:
"""Start a Twilio bot process with the given configuration.
Args:
room_details: Room URL, token, and SIP endpoint
call_id: Twilio call ID (CallSid)
Returns:
Boolean indicating success
"""
room_url = room_details["room"]
token = room_details["token"]
sip_endpoint = room_details["sip_endpoint"]
# Format command for Twilio bot
bot_proc = f"python3 -m bot_twilio -u {room_url} -t {token} -i {call_id} -s {sip_endpoint}"
print(f"Starting Twilio bot. Room: {room_url}")
try:
command_parts = shlex.split(bot_proc)
subprocess.Popen(command_parts, bufsize=1, cwd=os.path.dirname(os.path.abspath(__file__)))
return True
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to start subprocess: {e}")
# ----------------- API Setup ----------------- #
@asynccontextmanager
@@ -68,111 +176,44 @@ app.add_middleware(
allow_headers=["*"],
)
"""
Create Daily room, tell the bot if the room is created for Twilio's SIP or Daily's SIP (vendor).
When the vendor is Daily, the bot handles the call forwarding automatically,
i.e, forwards the call from the "hold music state" to the Daily Room's SIP URI.
Alternatively, when the vendor is Twilio (not Daily), the bot is responsible for
updating the state on Twilio. So when `dialin-ready` fires, it takes appropriate
action using the Twilio Client library.
"""
async def _create_daily_room(
room_url, callId, callDomain=None, dialoutNumber=None, vendor="daily", detect_voicemail=False
):
if not room_url:
# Create base properties with SIP settings
properties = DailyRoomProperties(
sip=DailyRoomSipParams(
display_name="dialin-user", video=False, sip_mode="dial-in", num_endpoints=1
)
)
# Only enable dialout if dialoutNumber is provided
if dialoutNumber:
properties.enable_dialout = True
params = DailyRoomParams(properties=properties)
print(f"Creating new room...")
room: DailyRoomObject = await daily_helpers["rest"].create_room(params=params)
else:
# Check passed room URL exist (we assume that it already has a sip set up!)
try:
room: DailyRoomObject = await daily_helpers["rest"].get_room_from_url(room_url)
except Exception:
raise HTTPException(status_code=500, detail=f"Room not found: {room_url}")
print(f"Daily room: {room.url} {room.config.sip_endpoint}")
# Give the agent a token to join the session
token = await daily_helpers["rest"].get_token(room.url, MAX_SESSION_TIME)
if not room or not token:
raise HTTPException(status_code=500, detail=f"Failed to get room or token token")
# Spawn a new agent, and join the user session
# Note: this is mostly for demonstration purposes (refer to 'deployment' in docs)
print(f"Vendor: {vendor}")
if vendor == "daily":
bot_proc = f"python3 -m bot_daily -u {room.url} -t {token} -i {callId} -d {callDomain}{' -v' if detect_voicemail else ''}"
if dialoutNumber:
bot_proc += f" -o {dialoutNumber}"
elif vendor == "daily-gemini":
bot_proc = f"python3 -m bot_daily_gemini -u {room.url} -t {token} -i {callId} -d {callDomain}{' -v' if detect_voicemail else ''}"
if dialoutNumber:
bot_proc += f" -o {dialoutNumber}"
else:
bot_proc = f"python3 -m bot_twilio -u {room.url} -t {token} -i {callId} -s {room.config.sip_endpoint}"
try:
subprocess.Popen(
[bot_proc], shell=True, bufsize=1, cwd=os.path.dirname(os.path.abspath(__file__))
)
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to start subprocess: {e}")
return room
# ----------------- API Endpoints ----------------- #
@app.post("/twilio_start_bot", response_class=PlainTextResponse)
async def twilio_start_bot(request: Request):
print(f"POST /twilio_voice_bot")
"""Handle incoming Twilio webhook calls and start a Twilio bot.
# twilio_start_bot is invoked directly by Twilio (as a web hook).
# On Twilio, under Active Numbers, pick the phone number
# Click Configure and under Voice Configuration,
# "a call comes in" choose webhook and point the URL to
# where this code is hosted.
data = {}
This endpoint is called directly by Twilio as a webhook when a call is received.
It puts the call on hold with music and starts a bot that will handle the call.
"""
print("POST /twilio_start_bot")
# Get form data from Twilio webhook
try:
# shouldnt have received json, twilio sends form data
form_data = await request.form()
data = dict(form_data)
except Exception:
pass
except Exception as e:
raise HTTPException(status_code=400, detail=f"Failed to parse Twilio form data: {str(e)}")
# Get default room URL from environment
room_url = os.getenv("DAILY_SAMPLE_ROOM_URL", None)
callId = data.get("CallSid")
if not callId:
raise HTTPException(status_code=500, detail="Missing 'CallSid' in request")
# Extract call ID from Twilio data
call_id = data.get("CallSid")
if not call_id:
raise HTTPException(status_code=400, detail="Missing 'CallSid' in request")
print("CallId: %s" % callId)
print(f"CallId: {call_id}")
# create room and tell the bot to join the created room
# note: Twilio does not require a callDomain
room: DailyRoomObject = await _create_daily_room(room_url, callId, None, "twilio")
# Create Daily room for the Twilio call
room_details = await create_daily_room(room_url, None) # No special config for Twilio rooms
print(f"Put Twilio on hold...")
# We have the room and the SIP URI,
# but we do not know if the Daily SIP Worker and the Bot have joined the call
# put the call on hold until the 'on_dialin_ready' fires.
# Then, the bot will update the called sid with the sip uri.
# http://com.twilio.music.classical.s3.amazonaws.com/BusyStrings.mp3
# Start the Twilio bot
await start_twilio_bot(room_details, call_id)
# Put the call on hold until the bot is ready to handle it
# The bot will update the call with the SIP URI when it's ready
resp = VoiceResponse()
resp.play(
url="http://com.twilio.sounds.music.s3.amazonaws.com/MARKOVICHAMP-Borghestral.mp3", loop=10
@@ -180,73 +221,98 @@ async def twilio_start_bot(request: Request):
return str(resp)
@app.post("/daily_start_bot")
async def daily_start_bot(request: Request) -> JSONResponse:
# The /daily_start_bot is invoked when a call is received on Daily's SIP URI
# daily_start_bot will create the room, put the call on hold until
# the bot and sip worker are ready. Daily will automatically
# forward the call to the SIP URi when dialin_ready fires.
# Use specified room URL, or create a new one if not specified
@app.post("/start")
async def handle_start_request(request: Request) -> JSONResponse:
"""Unified endpoint to handle bot configuration for different scenarios."""
# Get default room URL from environment
room_url = os.getenv("DAILY_SAMPLE_ROOM_URL", None)
# Get the dial-in properties from the request
try:
data = await request.json()
# Check if this is form data (from Twilio) or JSON
content_type = request.headers.get("content-type", "").lower()
if "application/x-www-form-urlencoded" in content_type:
# Handle form data from Twilio
form_data = await request.form()
data = dict(form_data)
# Check for CallSid which indicates this is a Twilio webhook
if "CallSid" in data:
# Redirect to Twilio handler for backward compatibility
return await twilio_start_bot(request)
else:
# Parse JSON request data
data = await request.json()
# Handle webhook test
if "test" in data:
# Pass through any webhook checks
return JSONResponse({"test": True})
detect_voicemail = data.get("detectVoicemail", False)
callId = data.get("callId", None)
callDomain = data.get("callDomain", None)
dialoutNumber = data.get("dialoutNumber", None)
except Exception:
raise HTTPException(
status_code=500, detail="Missing properties 'callId', 'callDomain', or 'dialoutNumber'"
)
room: DailyRoomObject = await _create_daily_room(
room_url, callId, callDomain, dialoutNumber, "daily", detect_voicemail
)
# Handle direct dialin webhook from Daily
if all(key in data for key in ["From", "To", "callId", "callDomain"]):
body = await process_dialin_request(data)
# Handle body-based request
elif "config" in data:
# Use the registry to set up the bot configuration
body = bot_registry.setup_configuration(data["config"])
else:
raise HTTPException(status_code=400, detail="Invalid request format")
# Grab a token for the user to join with
return JSONResponse({"room_url": room.url, "sipUri": room.config.sip_endpoint})
# Ensure prompt configuration
body = ensure_prompt_config(body)
# Detect which bot type to use
bot_type_name = bot_registry.detect_bot_type(body)
if not bot_type_name:
raise HTTPException(
status_code=400, detail="Configuration doesn't match any supported scenario"
)
@app.post("/daily_gemini_start_bot")
async def daily_gemini_start_bot(request: Request) -> JSONResponse:
# The /daily_start_bot is invoked when a call is received on Daily's SIP URI
# daily_start_bot will create the room, put the call on hold until
# the bot and sip worker are ready. Daily will automatically
# forward the call to the SIP URi when dialin_ready fires.
# Create the Daily room
room_details = await create_daily_room(room_url, body)
# Use specified room URL, or create a new one if not specified
room_url = os.getenv("DAILY_SAMPLE_ROOM_URL", None)
# Get the dial-in properties from the request
try:
data = await request.json()
if "test" in data:
# Pass through any webhook checks
return JSONResponse({"test": True})
detect_voicemail = data.get("detectVoicemail", False)
callId = data.get("callId", None)
callDomain = data.get("callDomain", None)
dialoutNumber = data.get("dialoutNumber", None)
except Exception:
raise HTTPException(
status_code=500, detail="Missing properties 'callId', 'callDomain', or 'dialoutNumber'"
)
# Start the bot
await start_bot(room_details, body, bot_type_name)
room: DailyRoomObject = await _create_daily_room(
room_url, callId, callDomain, dialoutNumber, "daily-gemini", detect_voicemail
)
# Get the bot type
bot_type = bot_registry.get_bot(bot_type_name)
# Grab a token for the user to join with
return JSONResponse({"room_url": room.url, "sipUri": room.config.sip_endpoint})
# Build the response
response = {"status": "Bot started", "bot_type": bot_type_name}
# Add room URL for test mode
if bot_type.has_test_mode(body):
response["room_url"] = room_details["room"]
# Remove llm_model from response as it's no longer relevant
if "llm" in body:
response["llm_provider"] = body["llm"] # Optionally keep track of provider
# Add dialout info for dialout scenarios
if "dialout_settings" in body and len(body["dialout_settings"]) > 0:
first_setting = body["dialout_settings"][0]
if "phoneNumber" in first_setting:
response["dialing_to"] = f"phone:{first_setting['phoneNumber']}"
elif "sipUri" in first_setting:
response["dialing_to"] = f"sip:{first_setting['sipUri']}"
return JSONResponse(response)
except json.JSONDecodeError:
# Check if this might be form data from Twilio
try:
content_type = request.headers.get("content-type", "").lower()
if "application/x-www-form-urlencoded" in content_type:
return await twilio_start_bot(request)
except Exception:
pass
raise HTTPException(status_code=400, detail="Invalid JSON in request body")
except Exception as e:
raise HTTPException(status_code=400, detail=f"Request processing error: {str(e)}")
# ----------------- Main ----------------- #
if __name__ == "__main__":
# Check environment variables
for env_var in REQUIRED_ENV_VARS:

211
examples/phone-chatbot/bot_runner_helpers.py Normal file
View File

@@ -0,0 +1,211 @@
# bot_runner_helpers.py
from typing import Any, Dict, Optional
from bot_constants import (
DEFAULT_CALLTRANSFER_MODE,
DEFAULT_DIALIN_EXAMPLE,
DEFAULT_SPEAK_SUMMARY,
DEFAULT_STORE_SUMMARY,
DEFAULT_TEST_IN_PREBUILT,
)
from call_connection_manager import CallConfigManager
# ----------------- Configuration Helpers ----------------- #
def determine_room_capabilities(config_body: Optional[Dict[str, Any]] = None) -> Dict[str, bool]:
"""Determine room capabilities based on the configuration.
This function examines the configuration to determine which capabilities
the Daily room should have enabled.
Args:
config_body: Configuration dictionary that determines room capabilities
Returns:
Dictionary of capability flags
"""
capabilities = {
"enable_dialin": False,
"enable_dialout": False,
# Add more capabilities here in the future as needed
}
if not config_body:
return capabilities
# Check for dialin capability
capabilities["enable_dialin"] = "dialin_settings" in config_body
# Check for dialout capability - needed for outbound calls or transfers
has_dialout_settings = "dialout_settings" in config_body
# Check if there's a transfer to an operator configured
has_call_transfer = "call_transfer" in config_body
# Enable dialout if any condition requires it
capabilities["enable_dialout"] = has_dialout_settings or has_call_transfer
return capabilities
def ensure_dialout_settings_array(body: Dict[str, Any]) -> Dict[str, Any]:
"""Ensures dialout_settings is an array of objects.
Args:
body: The configuration dictionary
Returns:
Updated configuration with dialout_settings as an array
"""
if "dialout_settings" in body:
# Convert to array if it's not already one
if not isinstance(body["dialout_settings"], list):
body["dialout_settings"] = [body["dialout_settings"]]
return body
def ensure_prompt_config(body: Dict[str, Any]) -> Dict[str, Any]:
"""Ensures the body has appropriate prompts settings, but doesn't add defaults.
Only makes sure the prompt section exists, allowing the bot script to handle defaults.
Args:
body: The configuration dictionary
Returns:
Updated configuration with prompt settings section
"""
if "prompts" not in body:
body["prompts"] = []
return body
def create_call_transfer_settings(body: Dict[str, Any]) -> Dict[str, Any]:
"""Create call transfer settings based on configuration and customer mapping.
Args:
body: The configuration dictionary
Returns:
Call transfer settings dictionary
"""
# Default transfer settings
transfer_settings = {
"mode": DEFAULT_CALLTRANSFER_MODE,
"speakSummary": DEFAULT_SPEAK_SUMMARY,
"storeSummary": DEFAULT_STORE_SUMMARY,
"testInPrebuilt": DEFAULT_TEST_IN_PREBUILT,
}
# If call_transfer already exists, merge the defaults with the existing settings
# This ensures all required fields exist while preserving user-specified values
if "call_transfer" in body:
existing_settings = body["call_transfer"]
# Update defaults with existing settings (existing values will override defaults)
for key, value in existing_settings.items():
transfer_settings[key] = value
else:
# No existing call_transfer - check if we have dialin settings for customer lookup
if "dialin_settings" in body:
# Create a temporary routing manager just for customer lookup
call_config_manager = CallConfigManager(body)
# Get caller info
caller_info = call_config_manager.get_caller_info()
from_number = caller_info.get("caller_number")
if from_number:
# Get customer name from phone number
customer_name = call_config_manager.get_customer_name(from_number)
# If we know the customer name, add it to the config for the bot to use
if customer_name:
transfer_settings["customerName"] = customer_name
return transfer_settings
def create_simple_dialin_settings(body: Dict[str, Any]) -> Dict[str, Any]:
"""Create simple dialin settings based on configuration.
Args:
body: The configuration dictionary
Returns:
Simple dialin settings dictionary
"""
# Default simple dialin settings
simple_dialin_settings = {
"testInPrebuilt": DEFAULT_TEST_IN_PREBUILT,
}
# If simple_dialin already exists, merge the defaults with the existing settings
if "simple_dialin" in body:
existing_settings = body["simple_dialin"]
# Update defaults with existing settings (existing values will override defaults)
for key, value in existing_settings.items():
simple_dialin_settings[key] = value
return simple_dialin_settings
def create_simple_dialout_settings(body: Dict[str, Any]) -> Dict[str, Any]:
"""Create simple dialout settings based on configuration.
Args:
body: The configuration dictionary
Returns:
Simple dialout settings dictionary
"""
# Default simple dialout settings
simple_dialout_settings = {
"testInPrebuilt": DEFAULT_TEST_IN_PREBUILT,
}
# If simple_dialout already exists, merge the defaults with the existing settings
if "simple_dialout" in body:
existing_settings = body["simple_dialout"]
# Update defaults with existing settings (existing values will override defaults)
for key, value in existing_settings.items():
simple_dialout_settings[key] = value
return simple_dialout_settings
async def process_dialin_request(data: Dict[str, Any]) -> Dict[str, Any]:
"""Process incoming dial-in request data to create a properly formatted body.
Converts camelCase fields received from webhook to snake_case format
for internal consistency across the codebase.
Args:
data: Raw dialin data from webhook
Returns:
Properly formatted configuration with snake_case keys
"""
# Create base body with dialin settings
body = {
"dialin_settings": {
"to": data.get("To", ""),
"from": data.get("From", ""),
"call_id": data.get("callId", data.get("CallSid", "")), # Convert to snake_case
"call_domain": data.get("callDomain", ""), # Convert to snake_case
}
}
# Use the global default to determine which example to run for dialin webhooks
example = DEFAULT_DIALIN_EXAMPLE
# Configure the bot based on the example
if example == "call_transfer":
# Create call transfer settings
body["call_transfer"] = create_call_transfer_settings(body)
elif example == "simple_dialin":
# Create simple dialin settings
body["simple_dialin"] = create_simple_dialin_settings(body)
return body

608
examples/phone-chatbot/call_connection_manager.py Normal file
View File

@@ -0,0 +1,608 @@
#
# Copyright (c) 2024–2025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
"""call_connection_manager.py.
Manages customer/operator relationships and call routing for voice bots.
Provides mapping between customers and operators, and functions for retrieving
contact information. Also includes call state management.
"""
import json
import os
from typing import Any, Dict, List, Optional
from loguru import logger
class CallFlowState:
"""State for tracking call flow operations and state transitions."""
def __init__(self):
# Operator-related state
self.dialed_operator = False
self.operator_connected = False
self.current_operator_index = 0
self.operator_dialout_settings = []
self.summary_finished = False
# Voicemail detection state
self.voicemail_detected = False
self.human_detected = False
self.voicemail_message_left = False
# Call termination state
self.call_terminated = False
self.participant_left_early = False
# Operator-related methods
def set_operator_dialed(self):
"""Mark that an operator has been dialed."""
self.dialed_operator = True
def set_operator_connected(self):
"""Mark that an operator has connected to the call."""
self.operator_connected = True
# Summary is not finished when operator first connects
self.summary_finished = False
def set_operator_disconnected(self):
"""Handle operator disconnection."""
self.operator_connected = False
self.summary_finished = False
def set_summary_finished(self):
"""Mark the summary as finished."""
self.summary_finished = True
def set_operator_dialout_settings(self, settings):
"""Set the list of operator dialout settings to try."""
self.operator_dialout_settings = settings
self.current_operator_index = 0
def get_current_dialout_setting(self):
"""Get the current operator dialout setting to try."""
if not self.operator_dialout_settings or self.current_operator_index >= len(
self.operator_dialout_settings
):
return None
return self.operator_dialout_settings[self.current_operator_index]
def move_to_next_operator(self):
"""Move to the next operator in the list."""
self.current_operator_index += 1
return self.get_current_dialout_setting()
# Voicemail detection methods
def set_voicemail_detected(self):
"""Mark that a voicemail system has been detected."""
self.voicemail_detected = True
self.human_detected = False
def set_human_detected(self):
"""Mark that a human has been detected (not voicemail)."""
self.human_detected = True
self.voicemail_detected = False
def set_voicemail_message_left(self):
"""Mark that a voicemail message has been left."""
self.voicemail_message_left = True
# Call termination methods
def set_call_terminated(self):
"""Mark that the call has been terminated by the bot."""
self.call_terminated = True
def set_participant_left_early(self):
"""Mark that a participant left the call early."""
self.participant_left_early = True
class SessionManager:
"""Centralized management of session IDs and state for all call participants."""
def __init__(self):
# Track session IDs of different participant types
self.session_ids = {
"operator": None,
"customer": None,
"bot": None,
# Add other participant types as needed
}
# References for easy access in processors that need mutable containers
self.session_id_refs = {
"operator": [None],
"customer": [None],
"bot": [None],
# Add other participant types as needed
}
# State object for call flow
self.call_flow_state = CallFlowState()
def set_session_id(self, participant_type, session_id):
"""Set the session ID for a specific participant type.
Args:
participant_type: Type of participant (e.g., "operator", "customer", "bot")
session_id: The session ID to set
"""
if participant_type in self.session_ids:
self.session_ids[participant_type] = session_id
# Also update the corresponding reference if it exists
if participant_type in self.session_id_refs:
self.session_id_refs[participant_type][0] = session_id
def get_session_id(self, participant_type):
"""Get the session ID for a specific participant type.
Args:
participant_type: Type of participant (e.g., "operator", "customer", "bot")
Returns:
The session ID or None if not set
"""
return self.session_ids.get(participant_type)
def get_session_id_ref(self, participant_type):
"""Get the mutable reference for a specific participant type.
Args:
participant_type: Type of participant (e.g., "operator", "customer", "bot")
Returns:
A mutable list container holding the session ID or None if not available
"""
return self.session_id_refs.get(participant_type)
def is_participant_type(self, session_id, participant_type):
"""Check if a session ID belongs to a specific participant type.
Args:
session_id: The session ID to check
participant_type: Type of participant (e.g., "operator", "customer", "bot")
Returns:
True if the session ID matches the participant type, False otherwise
"""
return self.session_ids.get(participant_type) == session_id
def reset_participant(self, participant_type):
"""Reset the state for a specific participant type.
Args:
participant_type: Type of participant (e.g., "operator", "customer", "bot")
"""
if participant_type in self.session_ids:
self.session_ids[participant_type] = None
if participant_type in self.session_id_refs:
self.session_id_refs[participant_type][0] = None
# Additional reset actions for specific participant types
if participant_type == "operator":
self.call_flow_state.set_operator_disconnected()
class CallConfigManager:
"""Manages customer/operator relationships and call routing."""
def __init__(self, body_data: Dict[str, Any] = None):
"""Initialize with optional body data.
Args:
body_data: Optional dictionary containing request body data
"""
self.body = body_data or {}
# Get environment variables with fallbacks
self.dial_in_from_number = os.getenv("DIAL_IN_FROM_NUMBER", "+10000000001")
self.dial_out_to_number = os.getenv("DIAL_OUT_TO_NUMBER", "+10000000002")
self.operator_number = os.getenv("OPERATOR_NUMBER", "+10000000003")
# Initialize maps with dynamic values
self._initialize_maps()
self._build_reverse_lookup_maps()
def _initialize_maps(self):
"""Initialize the customer and operator maps with environment variables."""
# Maps customer names to their contact information
self.CUSTOMER_MAP = {
"Dominic": {
"phoneNumber": self.dial_in_from_number, # I have two phone numbers, one for dialing in and one for dialing out. I give myself a separate name for each.
},
"Stewart": {
"phoneNumber": self.dial_out_to_number,
},
"James": {
"phoneNumber": "+10000000000",
"callerId": "james-caller-id-uuid",
"sipUri": "sip:james@example.com",
},
"Sarah": {
"sipUri": "sip:sarah@example.com",
},
"Michael": {
"phoneNumber": "+16505557890",
"callerId": "michael-caller-id-uuid",
},
}
# Maps customer names to their assigned operator names
self.CUSTOMER_TO_OPERATOR_MAP = {
"Dominic": ["Yunyoung", "Maria"], # Try Yunyoung first, then Maria
"Stewart": "Yunyoung",
"James": "Yunyoung",
"Sarah": "Jennifer",
"Michael": "Paul",
# Default mapping to ensure all customers have an operator
"Default": "Yunyoung",
}
# Maps operator names to their contact details
self.OPERATOR_CONTACT_MAP = {
"Paul": {
"phoneNumber": "+12345678904",
"callerId": "paul-caller-id-uuid",
},
"Yunyoung": {
"phoneNumber": self.operator_number, # Dials out to my other phone number.
},
"Maria": {
"sipUri": "sip:maria@example.com",
},
"Jennifer": {"phoneNumber": "+14155559876", "callerId": "jennifer-caller-id-uuid"},
"Default": {
"phoneNumber": self.operator_number, # Use the operator number as default
},
}
def _build_reverse_lookup_maps(self):
"""Build reverse lookup maps for phone numbers and SIP URIs to customer names."""
self._PHONE_TO_CUSTOMER_MAP = {}
self._SIP_TO_CUSTOMER_MAP = {}
for customer_name, contact_info in self.CUSTOMER_MAP.items():
if "phoneNumber" in contact_info:
self._PHONE_TO_CUSTOMER_MAP[contact_info["phoneNumber"]] = customer_name
if "sipUri" in contact_info:
self._SIP_TO_CUSTOMER_MAP[contact_info["sipUri"]] = customer_name
@classmethod
def from_json_string(cls, json_string: str):
"""Create a CallRoutingManager from a JSON string.
Args:
json_string: JSON string containing body data
Returns:
CallRoutingManager instance with parsed data
Raises:
json.JSONDecodeError: If JSON string is invalid
"""
body_data = json.loads(json_string)
return cls(body_data)
def find_customer_by_contact(self, contact_info: str) -> Optional[str]:
"""Find customer name from a contact identifier (phone number or SIP URI).
Args:
contact_info: The contact identifier (phone number or SIP URI)
Returns:
The customer name or None if not found
"""
# Check if it's a phone number
if contact_info in self._PHONE_TO_CUSTOMER_MAP:
return self._PHONE_TO_CUSTOMER_MAP[contact_info]
# Check if it's a SIP URI
if contact_info in self._SIP_TO_CUSTOMER_MAP:
return self._SIP_TO_CUSTOMER_MAP[contact_info]
return None
def get_customer_name(self, phone_number: str) -> Optional[str]:
"""Get customer name from their phone number.
Args:
phone_number: The customer's phone number
Returns:
The customer name or None if not found
"""
# Note: In production, this would likely query a database
return self.find_customer_by_contact(phone_number)
def get_operators_for_customer(self, customer_name: Optional[str]) -> List[str]:
"""Get the operator name(s) assigned to a customer.
Args:
customer_name: The customer's name
Returns:
List of operator names (single item or multiple)
"""
# Note: In production, this would likely query a database
if not customer_name or customer_name not in self.CUSTOMER_TO_OPERATOR_MAP:
return ["Default"]
operators = self.CUSTOMER_TO_OPERATOR_MAP[customer_name]
# Convert single string to list for consistency
if isinstance(operators, str):
return [operators]
return operators
def get_operator_dialout_settings(self, operator_name: str) -> Dict[str, str]:
"""Get an operator's dialout settings from their name.
Args:
operator_name: The operator's name
Returns:
Dictionary with dialout settings for the operator
"""
# Note: In production, this would likely query a database
return self.OPERATOR_CONTACT_MAP.get(operator_name, self.OPERATOR_CONTACT_MAP["Default"])
def get_dialout_settings_for_caller(
self, from_number: Optional[str] = None
) -> List[Dict[str, str]]:
"""Determine the appropriate operator dialout settings based on caller's number.
This method uses the caller's number to look up the customer name,
then finds the assigned operators for that customer, and returns
an array of operator dialout settings to try in sequence.
Args:
from_number: The caller's phone number (from dialin_settings)
Returns:
List of operator dialout settings to try
"""
if not from_number:
# If we don't have dialin settings, use the Default operator
return [self.get_operator_dialout_settings("Default")]
# Get customer name from phone number
customer_name = self.get_customer_name(from_number)
# Get operator names assigned to this customer
operator_names = self.get_operators_for_customer(customer_name)
# Get dialout settings for each operator
return [self.get_operator_dialout_settings(name) for name in operator_names]
def get_caller_info(self) -> Dict[str, Optional[str]]:
"""Get caller and dialed numbers from dialin settings in the body.
Returns:
Dictionary containing caller_number and dialed_number
"""
raw_dialin_settings = self.body.get("dialin_settings")
if not raw_dialin_settings:
return {"caller_number": None, "dialed_number": None}
# Handle different case variations
dialed_number = raw_dialin_settings.get("To") or raw_dialin_settings.get("to")
caller_number = raw_dialin_settings.get("From") or raw_dialin_settings.get("from")
return {"caller_number": caller_number, "dialed_number": dialed_number}
def get_caller_number(self) -> Optional[str]:
"""Get the caller's phone number from dialin settings in the body.
Returns:
The caller's phone number or None if not available
"""
return self.get_caller_info()["caller_number"]
async def start_dialout(self, transport, dialout_settings=None):
"""Helper function to start dialout using the provided settings or from body.
Args:
transport: The transport instance to use for dialout
dialout_settings: Optional override for dialout settings
Returns:
None
"""
# Use provided settings or get from body
settings = dialout_settings or self.get_dialout_settings()
if not settings:
logger.warning("No dialout settings available")
return
for setting in settings:
if "phoneNumber" in setting:
logger.info(f"Dialing number: {setting['phoneNumber']}")
if "callerId" in setting:
logger.info(f"with callerId: {setting['callerId']}")
await transport.start_dialout(
{"phoneNumber": setting["phoneNumber"], "callerId": setting["callerId"]}
)
else:
logger.info("with no callerId")
await transport.start_dialout({"phoneNumber": setting["phoneNumber"]})
elif "sipUri" in setting:
logger.info(f"Dialing sipUri: {setting['sipUri']}")
await transport.start_dialout({"sipUri": setting["sipUri"]})
else:
logger.warning(f"Unknown dialout setting format: {setting}")
def get_dialout_settings(self) -> Optional[List[Dict[str, Any]]]:
"""Extract dialout settings from the body.
Returns:
List of dialout setting objects or None if not present
"""
# Check if we have dialout settings
if "dialout_settings" in self.body:
dialout_settings = self.body["dialout_settings"]
# Convert to list if it's an object (for backward compatibility)
if isinstance(dialout_settings, dict):
return [dialout_settings]
elif isinstance(dialout_settings, list):
return dialout_settings
return None
def get_dialin_settings(self) -> Optional[Dict[str, Any]]:
"""Extract dialin settings from the body.
Handles both camelCase and snake_case variations of fields for backward compatibility,
but normalizes to snake_case for internal usage.
Returns:
Dictionary containing dialin settings or None if not present
"""
raw_dialin_settings = self.body.get("dialin_settings")
if not raw_dialin_settings:
return None
# Normalize dialin settings to handle different case variations
# Prioritize snake_case (call_id, call_domain) but fall back to camelCase (callId, callDomain)
dialin_settings = {
"call_id": raw_dialin_settings.get("call_id") or raw_dialin_settings.get("callId"),
"call_domain": raw_dialin_settings.get("call_domain")
or raw_dialin_settings.get("callDomain"),
"to": raw_dialin_settings.get("to") or raw_dialin_settings.get("To"),
"from": raw_dialin_settings.get("from") or raw_dialin_settings.get("From"),
}
return dialin_settings
# Bot prompt helper functions - no defaults provided, just return what's in the body
def get_prompt(self, prompt_name: str) -> Optional[str]:
"""Retrieve the prompt text for a given prompt name.
Args:
prompt_name: The name of the prompt to retrieve.
Returns:
The prompt string corresponding to the provided name, or None if not configured.
"""
prompts = self.body.get("prompts", [])
for prompt in prompts:
if prompt.get("name") == prompt_name:
return prompt.get("text")
return None
def get_transfer_mode(self) -> Optional[str]:
"""Get transfer mode from the body.
Returns:
Transfer mode string or None if not configured
"""
if "call_transfer" in self.body:
return self.body["call_transfer"].get("mode")
return None
def get_speak_summary(self) -> Optional[bool]:
"""Get speak summary from the body.
Returns:
Boolean indicating if summary should be spoken or None if not configured
"""
if "call_transfer" in self.body:
return self.body["call_transfer"].get("speakSummary")
return None
def get_store_summary(self) -> Optional[bool]:
"""Get store summary from the body.
Returns:
Boolean indicating if summary should be stored or None if not configured
"""
if "call_transfer" in self.body:
return self.body["call_transfer"].get("storeSummary")
return None
def is_test_mode(self) -> bool:
"""Check if running in test mode.
Returns:
Boolean indicating if test mode is enabled
"""
if "voicemail_detection" in self.body:
return bool(self.body["voicemail_detection"].get("testInPrebuilt"))
if "call_transfer" in self.body:
return bool(self.body["call_transfer"].get("testInPrebuilt"))
if "simple_dialin" in self.body:
return bool(self.body["simple_dialin"].get("testInPrebuilt"))
if "simple_dialout" in self.body:
return bool(self.body["simple_dialout"].get("testInPrebuilt"))
return False
def is_voicemail_detection_enabled(self) -> bool:
"""Check if voicemail detection is enabled in the body.
Returns:
Boolean indicating if voicemail detection is enabled
"""
return bool(self.body.get("voicemail_detection"))
def customize_prompt(self, prompt: str, customer_name: Optional[str] = None) -> str:
"""Insert customer name into prompt template if available.
Args:
prompt: The prompt template containing optional {customer_name} placeholders
customer_name: Optional customer name to insert
Returns:
Customized prompt with customer name inserted
"""
if customer_name and prompt:
return prompt.replace("{customer_name}", customer_name)
return prompt
def create_system_message(self, content: str) -> Dict[str, str]:
"""Create a properly formatted system message.
Args:
content: The message content
Returns:
Dictionary with role and content for the system message
"""
return {"role": "system", "content": content}
def create_user_message(self, content: str) -> Dict[str, str]:
"""Create a properly formatted user message.
Args:
content: The message content
Returns:
Dictionary with role and content for the user message
"""
return {"role": "user", "content": content}
def get_customer_info_suffix(
self, customer_name: Optional[str] = None, preposition: str = "for"
) -> str:
"""Create a consistent customer info suffix.
Args:
customer_name: Optional customer name
preposition: Preposition to use before the name (e.g., "for", "to", "")
Returns:
String with formatted customer info suffix
"""
if not customer_name:
return ""
# Add a space before the preposition if it's not empty
space_prefix = " " if preposition else ""
# For non-empty prepositions, add a space after it
space_suffix = " " if preposition else ""
return f"{space_prefix}{preposition}{space_suffix}{customer_name}"

481
examples/phone-chatbot/call_transfer.py Normal file
View File

@@ -0,0 +1,481 @@
#
# Copyright (c) 2024–2025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import argparse
import asyncio
import os
import sys
from call_connection_manager import CallConfigManager, SessionManager
from dotenv import load_dotenv
from loguru import logger
from pipecat.adapters.schemas.function_schema import FunctionSchema
from pipecat.adapters.schemas.tools_schema import ToolsSchema
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import (
BotStoppedSpeakingFrame,
EndTaskFrame,
Frame,
LLMMessagesFrame,
TranscriptionFrame,
)
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.processors.filters.function_filter import FunctionFilter
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.services.llm_service import LLMService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.transports.services.daily import DailyDialinSettings, DailyParams, DailyTransport
load_dotenv(override=True)
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
daily_api_key = os.getenv("DAILY_API_KEY", "")
daily_api_url = os.getenv("DAILY_API_URL", "https://api.daily.co/v1")
class TranscriptionModifierProcessor(FrameProcessor):
"""Processor that modifies transcription frames before they reach the context aggregator."""
def __init__(self, operator_session_id_ref):
"""Initialize with a reference to the operator_session_id variable.
Args:
operator_session_id_ref: A reference or container holding the operator's session ID
"""
super().__init__()
self.operator_session_id_ref = operator_session_id_ref
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
# Only process frames that are moving downstream
if direction == FrameDirection.DOWNSTREAM:
# Check if the frame is a transcription frame
if isinstance(frame, TranscriptionFrame):
# Check if this frame is from the operator
if (
self.operator_session_id_ref[0] is not None
and hasattr(frame, "user_id")
and frame.user_id == self.operator_session_id_ref[0]
):
# Modify the text to include operator prefix
frame.text = f"[OPERATOR]: {frame.text}"
logger.debug(f"++++ Modified Operator Transcription: {frame.text}")
# Push the (potentially modified) frame downstream
await self.push_frame(frame, direction)
class SummaryFinished(FrameProcessor):
"""Frame processor that monitors when summary has been finished."""
def __init__(self, dial_operator_state):
super().__init__()
# Store reference to the shared state object
self.dial_operator_state = dial_operator_state
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
# Check if operator is connected and this is the end of bot speaking
if self.dial_operator_state.operator_connected and isinstance(
frame, BotStoppedSpeakingFrame
):
logger.debug("Summary finished, bot will stop speaking")
self.dial_operator_state.set_summary_finished()
await self.push_frame(frame, direction)
async def main(
room_url: str,
token: str,
body: dict,
):
# ------------ CONFIGURATION AND SETUP ------------
# Create a routing manager using the provided body
call_config_manager = CallConfigManager.from_json_string(body) if body else CallConfigManager()
# Get caller information
caller_info = call_config_manager.get_caller_info()
caller_number = caller_info["caller_number"]
dialed_number = caller_info["dialed_number"]
# Get customer name based on caller number
customer_name = call_config_manager.get_customer_name(caller_number) if caller_number else None
# Get appropriate operator settings based on the caller
operator_dialout_settings = call_config_manager.get_dialout_settings_for_caller(caller_number)
logger.info(f"Caller number: {caller_number}")
logger.info(f"Dialed number: {dialed_number}")
logger.info(f"Customer name: {customer_name}")
logger.info(f"Operator dialout settings: {operator_dialout_settings}")
# Check if in test mode
test_mode = call_config_manager.is_test_mode()
# Get dialin settings if present
dialin_settings = call_config_manager.get_dialin_settings()
# ------------ TRANSPORT SETUP ------------
# Set up transport parameters
if test_mode:
logger.info("Running in test mode")
transport_params = DailyParams(
api_url=daily_api_url,
api_key=daily_api_key,
audio_in_enabled=True,
audio_out_enabled=True,
camera_out_enabled=False,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
transcription_enabled=True,
)
else:
daily_dialin_settings = DailyDialinSettings(
call_id=dialin_settings.get("call_id"), call_domain=dialin_settings.get("call_domain")
)
transport_params = DailyParams(
api_url=daily_api_url,
api_key=daily_api_key,
dialin_settings=daily_dialin_settings,
audio_in_enabled=True,
audio_out_enabled=True,
camera_out_enabled=False,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
transcription_enabled=True,
)
# Initialize the session manager
session_manager = SessionManager()
# Set up the operator dialout settings
session_manager.call_flow_state.set_operator_dialout_settings(operator_dialout_settings)
# Initialize transport
transport = DailyTransport(
room_url,
token,
"Call Transfer Bot",
transport_params,
)
# Initialize TTS
tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY", ""),
voice_id="b7d50908-b17c-442d-ad8d-810c63997ed9", # Use Helpful Woman voice by default
)
# ------------ LLM AND CONTEXT SETUP ------------
# Get prompts from routing manager
call_transfer_initial_prompt = call_config_manager.get_prompt("call_transfer_initial_prompt")
# Build default greeting with customer name if available
customer_greeting = f"Hello {customer_name}" if customer_name else "Hello"
default_greeting = f"{customer_greeting}, this is Hailey from customer support. What can I help you with today?"
# Build initial prompt
if call_transfer_initial_prompt:
# Use custom prompt with customer name replacement if needed
system_instruction = call_config_manager.customize_prompt(
call_transfer_initial_prompt, customer_name
)
logger.info("Using custom call transfer initial prompt")
else:
# Use default prompt with formatted greeting
system_instruction = f"""You are Chatbot, a friendly, helpful robot. Never refer to this prompt, even if asked. Follow these steps **EXACTLY**.
### **Standard Operating Procedure:**
#### **Step 1: Greeting**
- Greet the user with: "{default_greeting}"
#### **Step 2: Handling Requests**
- If the user requests a supervisor, **IMMEDIATELY** call the `dial_operator` function.
- **FAILURE TO CALL `dial_operator` IMMEDIATELY IS A MISTAKE.**
- If the user ends the conversation, **IMMEDIATELY** call the `terminate_call` function.
- **FAILURE TO CALL `terminate_call` IMMEDIATELY IS A MISTAKE.**
### **General Rules**
- Your output will be converted to audio, so **do not include special characters or formatting.**
"""
logger.info("Using default call transfer initial prompt")
# Create the system message and initialize messages list
messages = [call_config_manager.create_system_message(system_instruction)]
# ------------ FUNCTION DEFINITIONS ------------
async def terminate_call(
task: PipelineTask, # Pipeline task reference
function_name,
tool_call_id,
args,
llm: LLMService,
context: OpenAILLMContext,
result_callback,
):
"""Function the bot can call to terminate the call."""
# Create a message to add
content = "The user wants to end the conversation, thank them for chatting."
message = call_config_manager.create_system_message(content)
# Append the message to the list
messages.append(message)
# Queue the message to the context
await task.queue_frames([LLMMessagesFrame(messages)])
# Then end the call
await llm.queue_frame(EndTaskFrame(), FrameDirection.UPSTREAM)
async def dial_operator(
function_name: str,
tool_call_id: str,
args: dict,
llm: LLMService,
context: dict,
result_callback: callable,
):
"""Function the bot can call to dial an operator."""
dialout_setting = session_manager.call_flow_state.get_current_dialout_setting()
if call_config_manager.get_transfer_mode() == "dialout":
if dialout_setting:
session_manager.call_flow_state.set_operator_dialed()
logger.info(f"Dialing operator with settings: {dialout_setting}")
# Create a message to add
content = "The user has requested a supervisor, indicate that you will attempt to connect them with a supervisor."
message = call_config_manager.create_system_message(content)
# Append the message to the list
messages.append(message)
# Queue the message to the context
await task.queue_frames([LLMMessagesFrame(messages)])
# Start the dialout
await call_config_manager.start_dialout(transport, [dialout_setting])
else:
# Create a message to add
content = "Indicate that there are no operator dialout settings available."
message = call_config_manager.create_system_message(content)
# Append the message to the list
messages.append(message)
# Queue the message to the context
await task.queue_frames([LLMMessagesFrame(messages)])
logger.info("No operator dialout settings available")
else:
# Create a message to add
content = "Indicate that the current mode is not supported."
message = call_config_manager.create_system_message(content)
# Append the message to the list
messages.append(message)
# Queue the message to the context
await task.queue_frames([LLMMessagesFrame(messages)])
logger.info("Other mode not supported")
# Define function schemas for tools
terminate_call_function = FunctionSchema(
name="terminate_call",
description="Call this function to terminate the call.",
properties={},
required=[],
)
dial_operator_function = FunctionSchema(
name="dial_operator",
description="Call this function when the user asks to speak with a human",
properties={},
required=[],
)
# Create tools schema
tools = ToolsSchema(standard_tools=[terminate_call_function, dial_operator_function])
# Initialize LLM
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
# Register functions with the LLM
llm.register_function(
"terminate_call", lambda *args, **kwargs: terminate_call(task, *args, **kwargs)
)
llm.register_function("dial_operator", dial_operator)
# Initialize LLM context and aggregator
context = OpenAILLMContext(messages, tools)
context_aggregator = llm.create_context_aggregator(context)
# ------------ PIPELINE SETUP ------------
# Use the session manager's references
summary_finished = SummaryFinished(session_manager.call_flow_state)
transcription_modifier = TranscriptionModifierProcessor(
session_manager.get_session_id_ref("operator")
)
# Define function to determine if bot should speak
async def should_speak(self) -> bool:
result = (
not session_manager.call_flow_state.operator_connected
or not session_manager.call_flow_state.summary_finished
)
return result
# Build pipeline
pipeline = Pipeline(
[
transport.input(), # Transport user input
transcription_modifier, # Prepends operator transcription with [OPERATOR]
context_aggregator.user(), # User responses
FunctionFilter(should_speak),
llm,
tts,
summary_finished,
transport.output(), # Transport bot output
context_aggregator.assistant(), # Assistant spoken responses
]
)
# Create pipeline task
task = PipelineTask(
pipeline,
params=PipelineParams(allow_interruptions=True),
)
# ------------ EVENT HANDLERS ------------
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
# For the dialin case, we want the bot to answer the phone and greet the user
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_dialout_answered")
async def on_dialout_answered(transport, data):
logger.debug(f"++++ Dial-out answered: {data}")
await transport.capture_participant_transcription(data["sessionId"])
# Skip if operator already connected
if (
not session_manager.call_flow_state
or session_manager.call_flow_state.operator_connected
):
logger.debug(f"Operator already connected: {data}")
return
logger.debug(f"Operator connected with session ID: {data['sessionId']}")
# Set operator session ID in the session manager
session_manager.set_session_id("operator", data["sessionId"])
# Update state
session_manager.call_flow_state.set_operator_connected()
# Determine message content based on configuration
if call_config_manager.get_speak_summary():
logger.debug("Bot will speak summary")
call_transfer_prompt = call_config_manager.get_prompt("call_transfer_prompt")
if call_transfer_prompt:
# Use custom prompt
logger.info("Using custom call transfer prompt")
content = call_config_manager.customize_prompt(call_transfer_prompt, customer_name)
else:
# Use default summary prompt
logger.info("Using default call transfer prompt")
customer_info = call_config_manager.get_customer_info_suffix(customer_name)
content = f"""An operator is joining the call{customer_info}.
Give a brief summary of the customer's issues so far."""
else:
# Simple join notification without summary
logger.debug("Bot will not speak summary")
customer_info = call_config_manager.get_customer_info_suffix(customer_name)
content = f"""Indicate that an operator has joined the call{customer_info}."""
# Create and queue system message
message = call_config_manager.create_system_message(content)
messages.append(message)
await task.queue_frames([LLMMessagesFrame(messages)])
@transport.event_handler("on_dialout_stopped")
async def on_dialout_stopped(transport, data):
if session_manager.get_session_id("operator") and data[
"sessionId"
] == session_manager.get_session_id("operator"):
logger.debug("Dialout to operator stopped")
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
logger.debug(f"Participant left: {participant}, reason: {reason}")
# Check if the operator is the one who left
if not (
session_manager.get_session_id("operator")
and participant["id"] == session_manager.get_session_id("operator")
):
await task.cancel()
return
logger.debug("Operator left the call")
# Reset operator state
session_manager.reset_participant("operator")
# Determine message content
call_transfer_finished_prompt = call_config_manager.get_prompt(
"call_transfer_finished_prompt"
)
if call_transfer_finished_prompt:
# Use custom prompt for operator departure
logger.info("Using custom call transfer finished prompt")
content = call_config_manager.customize_prompt(
call_transfer_finished_prompt, customer_name
)
else:
# Use default prompt for operator departure
logger.info("Using default call transfer finished prompt")
customer_info = call_config_manager.get_customer_info_suffix(
customer_name, preposition=""
)
content = f"""The operator has left the call.
Resume your role as the primary support agent and use information from the operator's conversation to help the customer{customer_info}.
Let the customer know the operator has left and ask if they need further assistance."""
# Create and queue system message
message = call_config_manager.create_system_message(content)
messages.append(message)
await task.queue_frames([LLMMessagesFrame(messages)])
# ------------ RUN PIPELINE ------------
runner = PipelineRunner()
await runner.run(task)
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Pipecat Call Transfer Bot")
parser.add_argument("-u", "--url", type=str, help="Room URL")
parser.add_argument("-t", "--token", type=str, help="Room Token")
parser.add_argument("-b", "--body", type=str, help="JSON configuration string")
args = parser.parse_args()
# Log the arguments for debugging
logger.info(f"Room URL: {args.url}")
logger.info(f"Token: {args.token}")
logger.info(f"Body provided: {bool(args.body)}")
asyncio.run(main(args.url, args.token, args.body))

11
examples/phone-chatbot/env.example
View File

@@ -1,8 +1,11 @@
DAILY_SAMPLE_ROOM_URL=https://yourdomain.daily.co/yourroom # (optional: for joining the bot to the same room repeatedly for local dev)
DAILY_API_KEY=.
DAILY_API_KEY=
DAILY_API_URL=api.daily.co/v1
OPENAI_API_KEY=
ELEVENLABS_API_KEY=
ELEVENLABS_VOICE_ID=
GOOGLE_API_KEY
CARTESIA_API_KEY=
TWILIO_ACCOUNT_SID=
TWILIO_AUTH_TOKEN=
TWILIO_AUTH_TOKEN=
DIAL_IN_FROM_NUMBER=
DIAL_OUT_TO_NUMBER=
OPERATOR_NUMBER=

19
examples/phone-chatbot/fly.example.toml
View File

@@ -1,19 +0,0 @@
# fly.toml app configuration file generated for pipecat-dialin-demo on 2024-06-03T15:57:57+02:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#
app = 'pipecat-dialin-demo'
primary_region = 'sjc'
[build]
[http_service]
internal_port = 7860
force_https = true
auto_stop_machines = true
auto_start_machines = true
min_machines_running = 1
[[vm]]
size = 'performance-1x'

4
examples/phone-chatbot/requirements.txt
View File

@@ -1,5 +1,5 @@
pipecat-ai[daily,elevenlabs,openai,silero]
fastapi
pipecat-ai[daily,cartesia,openai,google,silero]
fastapi==3.11.12
uvicorn
python-dotenv
twilio

196
examples/phone-chatbot/simple_dialin.py Normal file
View File

@@ -0,0 +1,196 @@
#
# Copyright (c) 2024–2025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import argparse
import asyncio
import os
import sys
from call_connection_manager import CallConfigManager, SessionManager
from dotenv import load_dotenv
from loguru import logger
from pipecat.adapters.schemas.function_schema import FunctionSchema
from pipecat.adapters.schemas.tools_schema import ToolsSchema
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import EndTaskFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.services.llm_service import LLMService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.transports.services.daily import DailyDialinSettings, DailyParams, DailyTransport
load_dotenv(override=True)
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
daily_api_key = os.getenv("DAILY_API_KEY", "")
daily_api_url = os.getenv("DAILY_API_URL", "https://api.daily.co/v1")
async def main(
room_url: str,
token: str,
body: dict,
):
# ------------ CONFIGURATION AND SETUP ------------
# Create a config manager using the provided body
call_config_manager = CallConfigManager.from_json_string(body) if body else CallConfigManager()
# Get important configuration values
test_mode = call_config_manager.is_test_mode()
# Get dialin settings if present
dialin_settings = call_config_manager.get_dialin_settings()
# Initialize the session manager
session_manager = SessionManager()
# ------------ TRANSPORT SETUP ------------
# Set up transport parameters
if test_mode:
logger.info("Running in test mode")
transport_params = DailyParams(
api_url=daily_api_url,
api_key=daily_api_key,
audio_in_enabled=True,
audio_out_enabled=True,
camera_out_enabled=False,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
transcription_enabled=True,
)
else:
daily_dialin_settings = DailyDialinSettings(
call_id=dialin_settings.get("call_id"), call_domain=dialin_settings.get("call_domain")
)
transport_params = DailyParams(
api_url=daily_api_url,
api_key=daily_api_key,
dialin_settings=daily_dialin_settings,
audio_in_enabled=True,
audio_out_enabled=True,
camera_out_enabled=False,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
transcription_enabled=True,
)
# Initialize transport with Daily
transport = DailyTransport(
room_url,
token,
"Simple Dial-in Bot",
transport_params,
)
# Initialize TTS
tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY", ""),
voice_id="b7d50908-b17c-442d-ad8d-810c63997ed9", # Use Helpful Woman voice by default
)
# ------------ FUNCTION DEFINITIONS ------------
async def terminate_call(
function_name, tool_call_id, args, llm: LLMService, context, result_callback
):
"""Function the bot can call to terminate the call upon completion of a voicemail message."""
if session_manager:
# Mark that the call was terminated by the bot
session_manager.call_flow_state.set_call_terminated()
# Then end the call
await llm.queue_frame(EndTaskFrame(), FrameDirection.UPSTREAM)
# Define function schemas for tools
terminate_call_function = FunctionSchema(
name="terminate_call",
description="Call this function to terminate the call.",
properties={},
required=[],
)
# Create tools schema
tools = ToolsSchema(standard_tools=[terminate_call_function])
# ------------ LLM AND CONTEXT SETUP ------------
# Set up the system instruction for the LLM
system_instruction = """You are Chatbot, a friendly, helpful robot. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by introducing yourself. If the user ends the conversation, **IMMEDIATELY** call the `terminate_call` function. """
# Initialize LLM
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
# Register functions with the LLM
llm.register_function("terminate_call", terminate_call)
# Create system message and initialize messages list
messages = [call_config_manager.create_system_message(system_instruction)]
# Initialize LLM context and aggregator
context = OpenAILLMContext(messages, tools)
context_aggregator = llm.create_context_aggregator(context)
# ------------ PIPELINE SETUP ------------
# Build pipeline
pipeline = Pipeline(
[
transport.input(), # Transport user input
context_aggregator.user(), # User responses
llm, # LLM
tts, # TTS
transport.output(), # Transport bot output
context_aggregator.assistant(), # Assistant spoken responses
]
)
# Create pipeline task
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
# ------------ EVENT HANDLERS ------------
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
logger.debug(f"First participant joined: {participant['id']}")
await transport.capture_participant_transcription(participant["id"])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
logger.debug(f"Participant left: {participant}, reason: {reason}")
await task.cancel()
# ------------ RUN PIPELINE ------------
if test_mode:
logger.debug("Running in test mode (can be tested in Daily Prebuilt)")
runner = PipelineRunner()
await runner.run(task)
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Simple Dial-in Bot")
parser.add_argument("-u", "--url", type=str, help="Room URL")
parser.add_argument("-t", "--token", type=str, help="Room Token")
parser.add_argument("-b", "--body", type=str, help="JSON configuration string")
args = parser.parse_args()
# Log the arguments for debugging
logger.info(f"Room URL: {args.url}")
logger.info(f"Token: {args.token}")
logger.info(f"Body provided: {bool(args.body)}")
asyncio.run(main(args.url, args.token, args.body))

187
examples/phone-chatbot/simple_dialout.py Normal file
View File

@@ -0,0 +1,187 @@
#
# Copyright (c) 2024–2025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import argparse
import asyncio
import os
import sys
from call_connection_manager import CallConfigManager
from dotenv import load_dotenv
from loguru import logger
from pipecat.adapters.schemas.function_schema import FunctionSchema
from pipecat.adapters.schemas.tools_schema import ToolsSchema
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import EndTaskFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.services.llm_service import LLMService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport
load_dotenv(override=True)
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
daily_api_key = os.getenv("DAILY_API_KEY", "")
daily_api_url = os.getenv("DAILY_API_URL", "https://api.daily.co/v1")
async def main(
room_url: str,
token: str,
body: dict,
):
# ------------ CONFIGURATION AND SETUP ------------
# Create a config manager using the provided body
call_config_manager = CallConfigManager.from_json_string(body) if body else CallConfigManager()
# Get important configuration values
dialout_settings = call_config_manager.get_dialout_settings()
test_mode = call_config_manager.is_test_mode()
# ------------ TRANSPORT SETUP ------------
transport_params = DailyParams(
api_url=daily_api_url,
api_key=daily_api_key,
audio_in_enabled=True,
audio_out_enabled=True,
camera_out_enabled=False,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
transcription_enabled=True,
)
# Initialize transport with Daily
transport = DailyTransport(
room_url,
token,
"Simple Dial-out Bot",
transport_params,
)
# Initialize TTS
tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY", ""),
voice_id="b7d50908-b17c-442d-ad8d-810c63997ed9", # Use Helpful Woman voice by default
)
# ------------ FUNCTION DEFINITIONS ------------
async def terminate_call(
function_name, tool_call_id, args, llm: LLMService, context, result_callback
):
"""Function the bot can call to terminate the call upon completion of a voicemail message."""
await llm.queue_frame(EndTaskFrame(), FrameDirection.UPSTREAM)
# Define function schemas for tools
terminate_call_function = FunctionSchema(
name="terminate_call",
description="Call this function to terminate the call.",
properties={},
required=[],
)
# Create tools schema
tools = ToolsSchema(standard_tools=[terminate_call_function])
# ------------ LLM AND CONTEXT SETUP ------------
# Set up the system instruction for the LLM
system_instruction = """You are Chatbot, a friendly, helpful robot. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by introducing yourself. If the user ends the conversation, **IMMEDIATELY** call the `terminate_call` function. """
# Initialize LLM
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
# Register functions with the LLM
llm.register_function("terminate_call", terminate_call)
# Create system message and initialize messages list
messages = [call_config_manager.create_system_message(system_instruction)]
# Initialize LLM context and aggregator
context = OpenAILLMContext(messages, tools)
context_aggregator = llm.create_context_aggregator(context)
# ------------ PIPELINE SETUP ------------
# Build pipeline
pipeline = Pipeline(
[
transport.input(), # Transport user input
context_aggregator.user(), # User responses
llm, # LLM
tts, # TTS
transport.output(), # Transport bot output
context_aggregator.assistant(), # Assistant spoken responses
]
)
# Create pipeline task
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
# ------------ EVENT HANDLERS ------------
@transport.event_handler("on_joined")
async def on_joined(transport, data):
# Start dialout if needed
if not test_mode and dialout_settings:
logger.debug("Dialout settings detected; starting dialout")
await call_config_manager.start_dialout(transport, dialout_settings)
@transport.event_handler("on_dialout_connected")
async def on_dialout_connected(transport, data):
logger.debug(f"Dial-out connected: {data}")
@transport.event_handler("on_dialout_answered")
async def on_dialout_answered(transport, data):
logger.debug(f"Dial-out answered: {data}")
# Automatically start capturing transcription for the participant
await transport.capture_participant_transcription(data["sessionId"])
# The bot will wait to hear the user before the bot speaks
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
if test_mode:
logger.debug(f"First participant joined: {participant['id']}")
await transport.capture_participant_transcription(participant["id"])
# The bot will wait to hear the user before the bot speaks
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
logger.debug(f"Participant left: {participant}, reason: {reason}")
await task.cancel()
# ------------ RUN PIPELINE ------------
if test_mode:
logger.debug("Running in test mode (can be tested in Daily Prebuilt)")
runner = PipelineRunner()
await runner.run(task)
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Simple Dial-out Bot")
parser.add_argument("-u", "--url", type=str, help="Room URL")
parser.add_argument("-t", "--token", type=str, help="Room Token")
parser.add_argument("-b", "--body", type=str, help="JSON configuration string")
args = parser.parse_args()
# Log the arguments for debugging
logger.info(f"Room URL: {args.url}")
logger.info(f"Token: {args.token}")
logger.info(f"Body provided: {bool(args.body)}")
asyncio.run(main(args.url, args.token, args.body))

472
examples/phone-chatbot/voicemail_detection.py Normal file
View File

@@ -0,0 +1,472 @@
#
# Copyright (c) 2024–2025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import argparse
import asyncio
import functools
import os
import sys
from call_connection_manager import CallConfigManager, SessionManager
from dotenv import load_dotenv
from loguru import logger
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import (
EndFrame,
EndTaskFrame,
InputAudioRawFrame,
StopTaskFrame,
TranscriptionFrame,
UserStartedSpeakingFrame,
UserStoppedSpeakingFrame,
)
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.services.deepgram.stt import DeepgramSTTService
from pipecat.services.google.google import GoogleLLMContext
from pipecat.services.google.llm import GoogleLLMService
from pipecat.services.llm_service import LLMService # Base LLM service class
from pipecat.transports.services.daily import (
DailyParams,
DailyTransport,
)
load_dotenv(override=True)
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
daily_api_key = os.getenv("DAILY_API_KEY", "")
daily_api_url = os.getenv("DAILY_API_URL", "https://api.daily.co/v1")
# ------------ HELPER CLASSES ------------
class UserAudioCollector(FrameProcessor):
"""Collects audio frames in a buffer, then adds them to the LLM context when the user stops speaking."""
def __init__(self, context, user_context_aggregator):
super().__init__()
self._context = context
self._user_context_aggregator = user_context_aggregator
self._audio_frames = []
self._start_secs = 0.2 # this should match VAD start_secs (hardcoding for now)
self._user_speaking = False
async def process_frame(self, frame, direction):
await super().process_frame(frame, direction)
if isinstance(frame, TranscriptionFrame):
# Skip transcription frames - we're handling audio directly
return
elif isinstance(frame, UserStartedSpeakingFrame):
self._user_speaking = True
elif isinstance(frame, UserStoppedSpeakingFrame):
self._user_speaking = False
self._context.add_audio_frames_message(audio_frames=self._audio_frames)
await self._user_context_aggregator.push_frame(
self._user_context_aggregator.get_context_frame()
)
elif isinstance(frame, InputAudioRawFrame):
if self._user_speaking:
# When speaking, collect frames
self._audio_frames.append(frame)
else:
# Maintain a rolling buffer of recent audio (for start of speech)
self._audio_frames.append(frame)
frame_duration = len(frame.audio) / 16 * frame.num_channels / frame.sample_rate
buffer_duration = frame_duration * len(self._audio_frames)
while buffer_duration > self._start_secs:
self._audio_frames.pop(0)
buffer_duration -= frame_duration
await self.push_frame(frame, direction)
class FunctionHandlers:
"""Handlers for the voicemail detection bot functions."""
def __init__(self, session_manager):
self.session_manager = session_manager
self.prompt = None # Can be set externally
async def voicemail_response(
self,
function_name,
tool_call_id,
args,
llm: LLMService,
context,
result_callback,
):
"""Function the bot can call to leave a voicemail message."""
message = """You are Chatbot leaving a voicemail message. Say EXACTLY this message and then terminate the call:
'Hello, this is a message for Pipecat example user. This is Chatbot. Please call back on 123-456-7891. Thank you.'"""
await result_callback(message)
async def human_conversation(
self,
function_name,
tool_call_id,
args,
llm: LLMService,
context,
result_callback,
):
"""Function called when bot detects it's talking to a human."""
# Update state to indicate human was detected
self.session_manager.call_flow_state.set_human_detected()
await llm.push_frame(StopTaskFrame(), FrameDirection.UPSTREAM)
# ------------ MAIN FUNCTION ------------
async def main(
room_url: str,
token: str,
body: dict,
):
# ------------ CONFIGURATION AND SETUP ------------
# Create a configuration manager from the provided body
call_config_manager = CallConfigManager.from_json_string(body) if body else CallConfigManager()
# Get important configuration values
dialout_settings = call_config_manager.get_dialout_settings()
test_mode = call_config_manager.is_test_mode()
# Get caller info (might be None for dialout scenarios)
caller_info = call_config_manager.get_caller_info()
logger.info(f"Caller info: {caller_info}")
# Initialize the session manager
session_manager = SessionManager()
# ------------ TRANSPORT AND SERVICES SETUP ------------
# Initialize transport
transport = DailyTransport(
room_url,
token,
"Voicemail Detection Bot",
DailyParams(
api_url=daily_api_url,
api_key=daily_api_key,
audio_in_enabled=True,
audio_out_enabled=True,
camera_out_enabled=False,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
vad_audio_passthrough=True, # Important for audio collection
),
)
# Initialize TTS
tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY", ""),
voice_id="b7d50908-b17c-442d-ad8d-810c63997ed9", # Use Helpful Woman voice by default
)
# Initialize speech-to-text service (for human conversation phase)
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
# ------------ FUNCTION DEFINITIONS ------------
async def terminate_call(
function_name,
tool_call_id,
args,
llm: LLMService,
context,
result_callback,
session_manager=None,
):
"""Function the bot can call to terminate the call."""
if session_manager:
# Set call terminated flag in the session manager
session_manager.call_flow_state.set_call_terminated()
await llm.queue_frame(EndTaskFrame(), FrameDirection.UPSTREAM)
# ------------ VOICEMAIL DETECTION PHASE SETUP ------------
# Define tools for both LLMs
tools = [
{
"function_declarations": [
{
"name": "switch_to_voicemail_response",
"description": "Call this function when you detect this is a voicemail system.",
},
{
"name": "switch_to_human_conversation",
"description": "Call this function when you detect this is a human.",
},
{
"name": "terminate_call",
"description": "Call this function to terminate the call.",
},
]
}
]
# Get voicemail detection prompt
voicemail_detection_prompt = call_config_manager.get_prompt("voicemail_detection_prompt")
if voicemail_detection_prompt:
system_instruction = voicemail_detection_prompt
else:
system_instruction = """You are Chatbot trying to determine if this is a voicemail system or a human.
If you hear any of these phrases (or very similar ones):
- "Please leave a message after the beep"
- "No one is available to take your call"
- "Record your message after the tone"
- "You have reached voicemail for..."
- "You have reached [phone number]"
- "[phone number] is unavailable"
- "The person you are trying to reach..."
- "The number you have dialed..."
- "Your call has been forwarded to an automated voice messaging system"
Then call the function switch_to_voicemail_response.
If it sounds like a human (saying hello, asking questions, etc.), call the function switch_to_human_conversation.
DO NOT say anything until you've determined if this is a voicemail or human.
If you are asked to terminate the call, **IMMEDIATELY** call the `terminate_call` function. **FAILURE TO CALL `terminate_call` IMMEDIATELY IS A MISTAKE.**"""
# Initialize voicemail detection LLM
voicemail_detection_llm = GoogleLLMService(
model="models/gemini-2.0-flash-lite", # Lighter model for faster detection
api_key=os.getenv("GOOGLE_API_KEY"),
system_instruction=system_instruction,
tools=tools,
)
# Initialize context and context aggregator
voicemail_detection_context = GoogleLLMContext()
voicemail_detection_context_aggregator = voicemail_detection_llm.create_context_aggregator(
voicemail_detection_context
)
# Get custom voicemail prompt if available
voicemail_prompt = call_config_manager.get_prompt("voicemail_prompt")
# Set up function handlers
handlers = FunctionHandlers(session_manager)
handlers.prompt = voicemail_prompt # Set custom prompt if available
# Register functions with the voicemail detection LLM
voicemail_detection_llm.register_function(
"switch_to_voicemail_response",
handlers.voicemail_response,
)
voicemail_detection_llm.register_function(
"switch_to_human_conversation", handlers.human_conversation
)
voicemail_detection_llm.register_function(
"terminate_call", functools.partial(terminate_call, session_manager=session_manager)
)
# Set up audio collector for handling audio input
voicemail_detection_audio_collector = UserAudioCollector(
voicemail_detection_context, voicemail_detection_context_aggregator.user()
)
# Build voicemail detection pipeline
voicemail_detection_pipeline = Pipeline(
[
transport.input(), # Transport user input
voicemail_detection_audio_collector, # Collect audio frames
voicemail_detection_context_aggregator.user(), # User context
voicemail_detection_llm, # LLM
tts, # TTS
transport.output(), # Transport bot output
voicemail_detection_context_aggregator.assistant(), # Assistant context
]
)
# Create pipeline task
voicemail_detection_pipeline_task = PipelineTask(
voicemail_detection_pipeline,
params=PipelineParams(allow_interruptions=True),
)
# ------------ EVENT HANDLERS ------------
@transport.event_handler("on_joined")
async def on_joined(transport, data):
# Start dialout if needed
if not test_mode and dialout_settings:
logger.debug("Dialout settings detected; starting dialout")
await call_config_manager.start_dialout(transport, dialout_settings)
@transport.event_handler("on_dialout_connected")
async def on_dialout_connected(transport, data):
logger.debug(f"Dial-out connected: {data}")
@transport.event_handler("on_dialout_answered")
async def on_dialout_answered(transport, data):
logger.debug(f"Dial-out answered: {data}")
# Start capturing transcription
await transport.capture_participant_transcription(data["sessionId"])
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
logger.debug(f"First participant joined: {participant['id']}")
if test_mode:
await transport.capture_participant_transcription(participant["id"])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
# Mark that a participant left early
session_manager.call_flow_state.set_participant_left_early()
await voicemail_detection_pipeline_task.queue_frame(EndFrame())
# ------------ RUN VOICEMAIL DETECTION PIPELINE ------------
if test_mode:
logger.debug("Detect voicemail example. You can test this in Daily Prebuilt")
runner = PipelineRunner()
print("!!! starting voicemail detection pipeline")
try:
await runner.run(voicemail_detection_pipeline_task)
except Exception as e:
logger.error(f"Error in voicemail detection pipeline: {e}")
import traceback
logger.error(traceback.format_exc())
print("!!! Done with voicemail detection pipeline")
# Check if we should exit early
if (
session_manager.call_flow_state.participant_left_early
or session_manager.call_flow_state.call_terminated
):
if session_manager.call_flow_state.participant_left_early:
print("!!! Participant left early; terminating call")
elif session_manager.call_flow_state.call_terminated:
print("!!! Bot terminated call; not proceeding to human conversation")
return
# ------------ HUMAN CONVERSATION PHASE SETUP ------------
# Get human conversation prompt
human_conversation_prompt = call_config_manager.get_prompt("human_conversation_prompt")
if human_conversation_prompt:
human_conversation_system_instruction = human_conversation_prompt
else:
human_conversation_system_instruction = """You are Chatbot talking to a human. Be friendly and helpful.
Start with: "Hello! I'm a friendly chatbot. How can I help you today?"
Keep your responses brief and to the point. Listen to what the person says.
When the person indicates they're done with the conversation by saying something like:
- "Goodbye"
- "That's all"
- "I'm done"
- "Thank you, that's all I needed"
THEN say: "Thank you for chatting. Goodbye!" and call the terminate_call function."""
# Initialize human conversation LLM
human_conversation_llm = GoogleLLMService(
model="models/gemini-2.0-flash-001", # Full model for better conversation
api_key=os.getenv("GOOGLE_API_KEY"),
system_instruction=human_conversation_system_instruction,
tools=tools,
)
# Initialize context and context aggregator
human_conversation_context = GoogleLLMContext()
human_conversation_context_aggregator = human_conversation_llm.create_context_aggregator(
human_conversation_context
)
# Register terminate function with the human conversation LLM
human_conversation_llm.register_function(
"terminate_call", functools.partial(terminate_call, session_manager=session_manager)
)
# Build human conversation pipeline
human_conversation_pipeline = Pipeline(
[
transport.input(), # Transport user input
stt, # Speech-to-text
human_conversation_context_aggregator.user(), # User context
human_conversation_llm, # LLM
tts, # TTS
transport.output(), # Transport bot output
human_conversation_context_aggregator.assistant(), # Assistant context
]
)
# Create pipeline task
human_conversation_pipeline_task = PipelineTask(
human_conversation_pipeline,
params=PipelineParams(allow_interruptions=True),
)
# Update participant left handler for human conversation phase
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await voicemail_detection_pipeline_task.queue_frame(EndFrame())
await human_conversation_pipeline_task.queue_frame(EndFrame())
# ------------ RUN HUMAN CONVERSATION PIPELINE ------------
print("!!! starting human conversation pipeline")
# Initialize the context with system message
human_conversation_context_aggregator.user().set_messages(
[call_config_manager.create_system_message(human_conversation_system_instruction)]
)
# Queue the context frame to start the conversation
await human_conversation_pipeline_task.queue_frames(
[human_conversation_context_aggregator.user().get_context_frame()]
)
# Run the human conversation pipeline
try:
await runner.run(human_conversation_pipeline_task)
except Exception as e:
logger.error(f"Error in voicemail detection pipeline: {e}")
import traceback
logger.error(traceback.format_exc())
print("!!! Done with human conversation pipeline")
# ------------ SCRIPT ENTRY POINT ------------
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Pipecat Voicemail Detection Bot")
parser.add_argument("-u", "--url", type=str, help="Room URL")
parser.add_argument("-t", "--token", type=str, help="Room Token")
parser.add_argument("-b", "--body", type=str, help="JSON configuration string")
args = parser.parse_args()
# Log the arguments for debugging
logger.info(f"Room URL: {args.url}")
logger.info(f"Token: {args.token}")
logger.info(f"Body provided: {bool(args.body)}")
asyncio.run(main(args.url, args.token, args.body))

768
examples/simple-chatbot/client/javascript/package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

4
examples/simple-chatbot/client/javascript/package.json
View File

@@ -15,7 +15,7 @@
"vite": "^6.0.9"
},
"dependencies": {
"@pipecat-ai/client-js": "^0.3.2",
"@pipecat-ai/daily-transport": "^0.3.4"
"@pipecat-ai/client-js": "^0.3.5",
"@pipecat-ai/daily-transport": "^0.3.8"
}
}

146
examples/simple-chatbot/client/javascript/src/app.js
View File

@@ -29,6 +29,7 @@ class ChatbotClient {
this.rtviClient = null;
this.setupDOMElements();
this.setupEventListeners();
this.initializeClientAndTransport();
}
/**
@@ -57,6 +58,79 @@ class ChatbotClient {
this.disconnectBtn.addEventListener('click', () => this.disconnect());
}
/**
* Set up the RTVI client and Daily transport
*/
initializeClientAndTransport() {
// Initialize the RTVI client with a DailyTransport and our configuration
this.rtviClient = new RTVIClient({
transport: new DailyTransport(),
params: {
// The baseURL and endpoint of your bot server that the client will connect to
baseUrl: 'http://localhost:7860',
endpoints: {
connect: '/connect',
},
},
enableMic: true, // Enable microphone for user input
enableCam: false,
callbacks: {
// Handle connection state changes
onConnected: () => {
this.updateStatus('Connected');
this.connectBtn.disabled = true;
this.disconnectBtn.disabled = false;
this.log('Client connected');
},
onDisconnected: () => {
this.updateStatus('Disconnected');
this.connectBtn.disabled = false;
this.disconnectBtn.disabled = true;
this.log('Client disconnected');
},
// Handle transport state changes
onTransportStateChanged: (state) => {
this.updateStatus(`Transport: ${state}`);
this.log(`Transport state changed: ${state}`);
if (state === 'ready') {
this.setupMediaTracks();
}
},
// Handle bot connection events
onBotConnected: (participant) => {
this.log(`Bot connected: ${JSON.stringify(participant)}`);
},
onBotDisconnected: (participant) => {
this.log(`Bot disconnected: ${JSON.stringify(participant)}`);
},
onBotReady: (data) => {
this.log(`Bot ready: ${JSON.stringify(data)}`);
this.setupMediaTracks();
},
// Transcript events
onUserTranscript: (data) => {
// Only log final transcripts
if (data.final) {
this.log(`User: ${data.text}`);
}
},
onBotTranscript: (data) => {
this.log(`Bot: ${data.text}`);
},
// Error handling
onMessageError: (error) => {
console.log('Message error:', error);
},
onError: (error) => {
console.log('Error:', JSON.stringify(error));
},
},
});
// Set up listeners for media track events
this.setupTrackListeners();
}
/**
* Add a timestamped message to the debug log
*/
@@ -181,77 +255,6 @@ class ChatbotClient {
*/
async connect() {
try {
// Create a new Daily transport for WebRTC communication
const transport = new DailyTransport();
// Initialize the RTVI client with our configuration
this.rtviClient = new RTVIClient({
transport,
params: {
// The baseURL and endpoint of your bot server that the client will connect to
baseUrl: 'http://localhost:7860',
endpoints: {
connect: '/connect',
},
},
enableMic: true, // Enable microphone for user input
enableCam: false,
callbacks: {
// Handle connection state changes
onConnected: () => {
this.updateStatus('Connected');
this.connectBtn.disabled = true;
this.disconnectBtn.disabled = false;
this.log('Client connected');
},
onDisconnected: () => {
this.updateStatus('Disconnected');
this.connectBtn.disabled = false;
this.disconnectBtn.disabled = true;
this.log('Client disconnected');
},
// Handle transport state changes
onTransportStateChanged: (state) => {
this.updateStatus(`Transport: ${state}`);
this.log(`Transport state changed: ${state}`);
if (state === 'ready') {
this.setupMediaTracks();
}
},
// Handle bot connection events
onBotConnected: (participant) => {
this.log(`Bot connected: ${JSON.stringify(participant)}`);
},
onBotDisconnected: (participant) => {
this.log(`Bot disconnected: ${JSON.stringify(participant)}`);
},
onBotReady: (data) => {
this.log(`Bot ready: ${JSON.stringify(data)}`);
this.setupMediaTracks();
},
// Transcript events
onUserTranscript: (data) => {
// Only log final transcripts
if (data.final) {
this.log(`User: ${data.text}`);
}
},
onBotTranscript: (data) => {
this.log(`Bot: ${data.text}`);
},
// Error handling
onMessageError: (error) => {
console.log('Message error:', error);
},
onError: (error) => {
console.log('Error:', error);
},
},
});
// Set up listeners for media track events
this.setupTrackListeners();
// Initialize audio/video devices
this.log('Initializing devices...');
await this.rtviClient.initDevices();
@@ -286,7 +289,6 @@ class ChatbotClient {
try {
// Disconnect the RTVI client
await this.rtviClient.disconnect();
this.rtviClient = null;
// Clean up audio
if (this.botAudio.srcObject) {

1516
examples/simple-chatbot/client/react/package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

6
examples/simple-chatbot/client/react/package.json
View File

@@ -10,9 +10,9 @@
"preview": "vite preview"
},
"dependencies": {
"@pipecat-ai/client-js": "^0.3.2",
"@pipecat-ai/client-react": "^0.3.2",
"@pipecat-ai/daily-transport": "^0.3.4",
"@pipecat-ai/client-js": "^0.3.5",
"@pipecat-ai/client-react": "^0.3.5",
"@pipecat-ai/daily-transport": "^0.3.8",
"react": "^18.3.1",
"react-dom": "^18.3.1"
},

3
examples/simple-chatbot/server/env.example
View File

@@ -1,4 +1,5 @@
DAILY_SAMPLE_ROOM_URL=https://yourdomain.daily.co/yourroom # (for joining the bot to the same room repeatedly for local dev)
DAILY_SAMPLE_ROOM_URL=https://yourdomain.daily.co/yourroom # (optional: for joining the bot to the same room repeatedly for local dev)
DAILY_SAMPLE_ROOM_TOKEN=9c8... # (optional: if your room above requires a token)
DAILY_API_KEY=7df...
OPENAI_API_KEY=sk-PL...
GEMINI_API_KEY=AIza...

18
examples/simple-chatbot/server/server.py
View File

@@ -111,15 +111,19 @@ async def create_room_and_token() -> tuple[str, str]:
Raises:
HTTPException: If room creation or token generation fails
"""
room = await daily_helpers["rest"].create_room(DailyRoomParams())
if not room.url:
raise HTTPException(status_code=500, detail="Failed to create room")
room_url = os.getenv("DAILY_SAMPLE_ROOM_URL", None)
token = os.getenv("DAILY_SAMPLE_ROOM_TOKEN", None)
if not room_url:
room = await daily_helpers["rest"].create_room(DailyRoomParams())
if not room.url:
raise HTTPException(status_code=500, detail="Failed to create room")
room_url = room.url
token = await daily_helpers["rest"].get_token(room.url)
if not token:
raise HTTPException(status_code=500, detail=f"Failed to get token for room: {room.url}")
token = await daily_helpers["rest"].get_token(room_url)
if not token:
raise HTTPException(status_code=500, detail=f"Failed to get token for room: {room_url}")
return room.url, token
return room_url, token
@app.get("/")

7
pyproject.toml
View File

@@ -20,7 +20,7 @@ classifiers = [
"Topic :: Scientific/Engineering :: Artificial Intelligence"
]
dependencies = [
"aiohttp~=3.11.13",
"aiohttp~=3.11.12",
"audioop-lts~=0.2.1; python_version>='3.13'",
"loguru~=0.7.3",
"Markdown~=3.7",
@@ -31,7 +31,7 @@ dependencies = [
"pyloudnorm~=0.1.1",
"resampy~=0.4.3",
"soxr~=0.5.0",
"openai~=1.67.0"
"openai~=1.70.0"
]
[project.urls]
@@ -115,6 +115,9 @@ select = [
"D", # Docstring rules
"I", # Import rules
]
# We ignore D107 because class docstrings already document __init__ parameters
# and our Sphinx configuration uses napoleon_include_init_with_doc=True
ignore = ["D107"]
[tool.ruff.lint.pydocstyle]
convention = "google"

43
src/pipecat/adapters/schemas/function_schema.py
View File

@@ -8,16 +8,22 @@ from typing import Any, Dict, List
class FunctionSchema:
"""Standardized function schema representation for tool definition.
Provides a structured way to define function tools used with AI models like OpenAI.
This schema defines the function's name, description, parameter properties, and
required parameters, following specifications required by AI service providers.
Args:
name: Name of the function to be called.
description: Description of what the function does.
properties: Dictionary defining parameter types, descriptions, and constraints.
required: List of property names that are required parameters.
"""
def __init__(
self, name: str, description: str, properties: Dict[str, Any], required: List[str]
) -> None:
"""Standardized function schema representation.
:param name: Name of the function.
:param description: Description of the function.
:param properties: Dictionary defining properties types and descriptions.
:param required: List of required parameters.
"""
self._name = name
self._description = description
self._properties = properties
@@ -26,7 +32,8 @@ class FunctionSchema:
def to_default_dict(self) -> Dict[str, Any]:
"""Converts the function schema to a dictionary.
:return: Dictionary representation of the function schema.
Returns:
Dictionary representation of the function schema.
"""
return {
"name": self._name,
@@ -40,16 +47,36 @@ class FunctionSchema:
@property
def name(self) -> str:
"""Get the function name.
Returns:
The function name.
"""
return self._name
@property
def description(self) -> str:
"""Get the function description.
Returns:
The function description.
"""
return self._description
@property
def properties(self) -> Dict[str, Any]:
"""Get the function properties.
Returns:
Dictionary of parameter specifications.
"""
return self._properties
@property
def required(self) -> List[str]:
"""Get the required parameters.
Returns:
List of required parameter names.
"""
return self._required

38
src/pipecat/frames/frames.py
View File

@@ -377,25 +377,6 @@ class LLMEnablePromptCachingFrame(DataFrame):
enable: bool
@dataclass
class FunctionCallResultProperties:
"""Properties for a function call result frame."""
run_llm: Optional[bool] = None
on_context_updated: Optional[Callable[[], Awaitable[None]]] = None
@dataclass
class FunctionCallResultFrame(DataFrame):
"""A frame containing the result of an LLM function (tool) call."""
function_name: str
tool_call_id: str
arguments: Any
result: Any
properties: Optional[FunctionCallResultProperties] = None
@dataclass
class TTSSpeakFrame(DataFrame):
"""A frame that contains a text that should be spoken by the TTS in the
@@ -652,6 +633,25 @@ class FunctionCallCancelFrame(SystemFrame):
tool_call_id: str
@dataclass
class FunctionCallResultProperties:
"""Properties for a function call result frame."""
run_llm: Optional[bool] = None
on_context_updated: Optional[Callable[[], Awaitable[None]]] = None
@dataclass
class FunctionCallResultFrame(SystemFrame):
"""A frame containing the result of an LLM function (tool) call."""
function_name: str
tool_call_id: str
arguments: Any
result: Any
properties: Optional[FunctionCallResultProperties] = None
@dataclass
class STTMuteFrame(SystemFrame):
"""System frame to mute/unmute the STT service."""

2
src/pipecat/observers/loggers/llm_log_observer.py
View File

@@ -18,7 +18,7 @@ from pipecat.frames.frames import (
from pipecat.observers.base_observer import BaseObserver
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContextFrame
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.services.ai_services import LLMService
from pipecat.services.llm_service import LLMService
class LLMLogObserver(BaseObserver):

2
src/pipecat/observers/loggers/transcription_log_observer.py
View File

@@ -13,7 +13,7 @@ from pipecat.frames.frames import (
)
from pipecat.observers.base_observer import BaseObserver
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.services.ai_services import STTService
from pipecat.services.stt_service import STTService
class TranscriptionLogObserver(BaseObserver):

4
src/pipecat/processors/aggregators/llm_response.py
View File

@@ -149,7 +149,8 @@ class BaseLLMResponseAggregator(FrameProcessor):
@abstractmethod
def reset(self):
"""Reset the internals of this aggregator. This should not modify the
internal messages."""
internal messages.
"""
pass
@abstractmethod
@@ -446,6 +447,7 @@ class LLMAssistantContextAggregator(LLMContextResponseAggregator):
await self._handle_user_image_frame(frame)
elif isinstance(frame, BotStoppedSpeakingFrame):
await self.push_aggregation()
await self.push_frame(frame, direction)
else:
await self.push_frame(frame, direction)

65
src/pipecat/processors/consumer_processor.py Normal file
View File

@@ -0,0 +1,65 @@
#
# Copyright (c) 2024–2025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import asyncio
from typing import Awaitable, Callable, Optional
from pipecat.frames.frames import CancelFrame, EndFrame, Frame, StartFrame
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.processors.producer_processor import ProducerProcessor, identity_transformer
class ConsumerProcessor(FrameProcessor):
"""This class passes-through frames and also consumes frames from a
producer's queue. When a frame from a producer queue is received it will be
pushed to the specified direction. The frames can be transformed into a
different type of frame before being pushed.
"""
def __init__(
self,
*,
producer: ProducerProcessor,
transformer: Callable[[Frame], Awaitable[Frame]] = identity_transformer,
direction: FrameDirection = FrameDirection.DOWNSTREAM,
**kwargs,
):
super().__init__(**kwargs)
self._transformer = transformer
self._direction = direction
self._queue: asyncio.Queue = producer.add_consumer()
self._consumer_task: Optional[asyncio.Task] = None
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
if isinstance(frame, StartFrame):
await self._start(frame)
elif isinstance(frame, EndFrame):
await self._stop(frame)
elif isinstance(frame, CancelFrame):
await self._cancel(frame)
await self.push_frame(frame, direction)
async def _start(self, _: StartFrame):
if not self._consumer_task:
self._consumer_task = self.create_task(self._consumer_task_handler())
async def _stop(self, _: EndFrame):
if self._consumer_task:
await self.cancel_task(self._consumer_task)
async def _cancel(self, _: CancelFrame):
if self._consumer_task:
await self.cancel_task(self._consumer_task)
async def _consumer_task_handler(self):
while True:
frame = await self._queue.get()
new_frame = await self._transformer(frame)
await self.push_frame(new_frame, self._direction)

2
src/pipecat/processors/filters/stt_mute_filter.py
View File

@@ -108,7 +108,7 @@ class STTMuteFilter(FrameProcessor):
async def _handle_mute_state(self, should_mute: bool):
"""Handles both STT muting and interruption control."""
if should_mute != self.is_muted:
logger.debug(f"STT {'muting' if should_mute else 'unmuting'}")
logger.debug(f"STTMuteFilter {'muting' if should_mute else 'unmuting'}")
self._is_muted = should_mute
await self.push_frame(STTMuteFrame(mute=should_mute))

73
src/pipecat/processors/producer_processor.py Normal file
View File

@@ -0,0 +1,73 @@
#
# Copyright (c) 2024–2025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import asyncio
from typing import Awaitable, Callable, List
from pipecat.frames.frames import Frame
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
async def identity_transformer(frame: Frame):
return frame
class ProducerProcessor(FrameProcessor):
"""This class optionally passes-through received frames and decides if those
frames should be sent to consumers based on a user-defined filter. The
frames can be transformed into a different type of frame before being
sending them to the consumers. More than one consumer can be added.
"""
def __init__(
self,
*,
filter: Callable[[Frame], Awaitable[bool]],
transformer: Callable[[Frame], Awaitable[Frame]] = identity_transformer,
passthrough: bool = True,
):
super().__init__()
self._filter = filter
self._transformer = transformer
self._passthrough = passthrough
self._consumers: List[asyncio.Queue] = []
def add_consumer(self):
"""
Adds a new consumer and returns its associated queue.
Returns:
asyncio.Queue: The queue for the newly added consumer.
"""
queue = asyncio.Queue()
self._consumers.append(queue)
return queue
async def process_frame(self, frame: Frame, direction: FrameDirection):
"""
Processes an incoming frame and determines whether to produce it as a ProducerItem.
If the frame meets the produce criteria, it will be added to the consumer queues.
If passthrough is enabled, the frame will also be sent to consumers.
Args:
frame (Frame): The frame to process.
direction (FrameDirection): The direction of the frame.
"""
await super().process_frame(frame, direction)
if await self._filter(frame):
await self._produce(frame)
if self._passthrough:
await self.push_frame(frame, direction)
else:
await self.push_frame(frame, direction)
async def _produce(self, frame: Frame):
for consumer in self._consumers:
new_frame = await self._transformer(frame)
await consumer.put(new_frame)

24
src/pipecat/processors/transcript_processor.py
View File

@@ -175,22 +175,28 @@ class AssistantTranscriptProcessor(BaseTranscriptProcessor):
"""
await super().process_frame(frame, direction)
if isinstance(frame, TTSTextFrame):
if isinstance(frame, (StartInterruptionFrame, CancelFrame)):
# Push frame first otherwise our emitted transcription update frame
# might get cleaned up.
await self.push_frame(frame, direction)
# Emit accumulated text with interruptions
await self._emit_aggregated_text()
elif isinstance(frame, TTSTextFrame):
# Start timestamp on first text part
if not self._aggregation_start_time:
self._aggregation_start_time = time_now_iso8601()
self._current_text_parts.append(frame.text)
elif isinstance(frame, (BotStoppedSpeakingFrame, StartInterruptionFrame, CancelFrame)):
# Emit accumulated text when bot finishes speaking or is interrupted
# Push frame.
await self.push_frame(frame, direction)
elif isinstance(frame, (BotStoppedSpeakingFrame, EndFrame)):
# Emit accumulated text when bot finishes speaking or pipeline ends.
await self._emit_aggregated_text()
elif isinstance(frame, EndFrame):
# Emit any remaining text when pipeline ends
await self._emit_aggregated_text()
await self.push_frame(frame, direction)
# Push frame.
await self.push_frame(frame, direction)
else:
await self.push_frame(frame, direction)
class TranscriptProcessor:

7
src/pipecat/processors/user_idle_processor.py
View File

@@ -127,9 +127,10 @@ class UserIdleProcessor(FrameProcessor):
# Check for end frames before processing
if isinstance(frame, (EndFrame, CancelFrame)):
await self.push_frame(frame, direction) # Push the frame down the pipeline
if self._idle_task:
await self._stop() # Stop the idle task, if it exists
# Stop the idle task, if it exists
await self._stop()
# Push the frame down the pipeline
await self.push_frame(frame, direction)
return
await self.push_frame(frame, direction)

2
src/pipecat/services/__init__.py
View File

@@ -37,4 +37,4 @@ class DeprecatedModuleProxy:
def __getattr__(self, attr):
if attr in self._globals:
return _warn_deprecated_access(self._globals, attr, self._old, self._new)
raise AttributeError(f"module 'pipecat.{self._old}' has no attribute '{attr}'")
raise AttributeError(f"module 'pipecat.services.{self._old}' has no attribute '{attr}'")

105
src/pipecat/services/ai_service.py Normal file
View File

@@ -0,0 +1,105 @@
#
# Copyright (c) 2024–2025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
from typing import Any, AsyncGenerator, Dict, Mapping
from loguru import logger
from pipecat.frames.frames import (
CancelFrame,
EndFrame,
ErrorFrame,
Frame,
StartFrame,
)
from pipecat.metrics.metrics import MetricsData
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
class AIService(FrameProcessor):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self._model_name: str = ""
self._settings: Dict[str, Any] = {}
self._session_properties: Dict[str, Any] = {}
@property
def model_name(self) -> str:
return self._model_name
def set_model_name(self, model: str):
self._model_name = model
self.set_core_metrics_data(MetricsData(processor=self.name, model=self._model_name))
async def start(self, frame: StartFrame):
pass
async def stop(self, frame: EndFrame):
pass
async def cancel(self, frame: CancelFrame):
pass
async def _update_settings(self, settings: Mapping[str, Any]):
from pipecat.services.openai_realtime_beta.events import (
SessionProperties,
)
for key, value in settings.items():
logger.debug("Update request for:", key, value)
if key in self._settings:
logger.info(f"Updating LLM setting {key} to: [{value}]")
self._settings[key] = value
elif key in SessionProperties.model_fields:
logger.debug("Attempting to update", key, value)
try:
from pipecat.services.openai_realtime_beta.events import (
TurnDetection,
)
if isinstance(self._session_properties, SessionProperties):
current_properties = self._session_properties
else:
current_properties = SessionProperties(**self._session_properties)
if key == "turn_detection" and isinstance(value, dict):
turn_detection = TurnDetection(**value)
setattr(current_properties, key, turn_detection)
else:
setattr(current_properties, key, value)
validated_properties = SessionProperties.model_validate(
current_properties.model_dump()
)
logger.info(f"Updating LLM setting {key} to: [{value}]")
self._session_properties = validated_properties.model_dump()
except Exception as e:
logger.warning(f"Unexpected error updating session property {key}: {e}")
elif key == "model":
logger.info(f"Updating LLM setting {key} to: [{value}]")
self.set_model_name(value)
else:
logger.warning(f"Unknown setting for {self.name} service: {key}")
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
if isinstance(frame, StartFrame):
await self.start(frame)
elif isinstance(frame, CancelFrame):
await self.cancel(frame)
elif isinstance(frame, EndFrame):
await self.stop(frame)
async def process_generator(self, generator: AsyncGenerator[Frame | None, None]):
async for f in generator:
if f:
if isinstance(f, ErrorFrame):
await self.push_error(f)
else:
await self.push_frame(f)

1129
src/pipecat/services/ai_services.py
View File

File diff suppressed because it is too large Load Diff

2
src/pipecat/services/anthropic/llm.py
View File

@@ -43,7 +43,7 @@ from pipecat.processors.aggregators.openai_llm_context import (
OpenAILLMContextFrame,
)
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_services import LLMService
from pipecat.services.llm_service import LLMService
try:
from anthropic import NOT_GIVEN, AsyncAnthropic, NotGiven

2
src/pipecat/services/assemblyai/stt.py
View File

@@ -18,7 +18,7 @@ from pipecat.frames.frames import (
StartFrame,
TranscriptionFrame,
)
from pipecat.services.ai_services import STTService
from pipecat.services.stt_service import STTService
from pipecat.transcriptions.language import Language
from pipecat.utils.time import time_now_iso8601

2
src/pipecat/services/aws/tts.py
View File

@@ -18,7 +18,7 @@ from pipecat.frames.frames import (
TTSStartedFrame,
TTSStoppedFrame,
)
from pipecat.services.ai_services import TTSService
from pipecat.services.tts_service import TTSService
from pipecat.transcriptions.language import Language
try:

2
src/pipecat/services/azure/image.py
View File

@@ -13,7 +13,7 @@ from loguru import logger
from PIL import Image
from pipecat.frames.frames import ErrorFrame, Frame, URLImageRawFrame
from pipecat.services.ai_services import ImageGenService
from pipecat.services.image_service import ImageGenService
class AzureImageGenServiceREST(ImageGenService):

2
src/pipecat/services/azure/stt.py
View File

@@ -16,8 +16,8 @@ from pipecat.frames.frames import (
StartFrame,
TranscriptionFrame,
)
from pipecat.services.ai_services import STTService
from pipecat.services.azure.common import language_to_azure_language
from pipecat.services.stt_service import STTService
from pipecat.transcriptions.language import Language
from pipecat.utils.time import time_now_iso8601

2
src/pipecat/services/azure/tts.py
View File

@@ -18,8 +18,8 @@ from pipecat.frames.frames import (
TTSStartedFrame,
TTSStoppedFrame,
)
from pipecat.services.ai_services import TTSService
from pipecat.services.azure.common import language_to_azure_language
from pipecat.services.tts_service import TTSService
from pipecat.transcriptions.language import Language
try:

2
src/pipecat/services/canonical/metrics.py
View File

@@ -18,7 +18,7 @@ from pipecat.frames.frames import CancelFrame, EndFrame, Frame
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.processors.audio.audio_buffer_processor import AudioBufferProcessor
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_services import AIService
from pipecat.services.ai_service import AIService
try:
import aiofiles

2
src/pipecat/services/cartesia/tts.py
View File

@@ -24,7 +24,7 @@ from pipecat.frames.frames import (
TTSStoppedFrame,
)
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_services import AudioContextWordTTSService, TTSService
from pipecat.services.tts_service import AudioContextWordTTSService, TTSService
from pipecat.transcriptions.language import Language
from pipecat.utils.text.base_text_aggregator import BaseTextAggregator
from pipecat.utils.text.skip_tags_aggregator import SkipTagsAggregator

2
src/pipecat/services/deepgram/stt.py
View File

@@ -19,7 +19,7 @@ from pipecat.frames.frames import (
UserStoppedSpeakingFrame,
)
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_services import STTService
from pipecat.services.stt_service import STTService
from pipecat.transcriptions.language import Language
from pipecat.utils.time import time_now_iso8601

2
src/pipecat/services/deepgram/tts.py
View File

@@ -16,7 +16,7 @@ from pipecat.frames.frames import (
TTSStartedFrame,
TTSStoppedFrame,
)
from pipecat.services.ai_services import TTSService
from pipecat.services.tts_service import TTSService
try:
from deepgram import DeepgramClient, SpeakOptions

2
src/pipecat/services/elevenlabs/tts.py
View File

@@ -25,7 +25,7 @@ from pipecat.frames.frames import (
TTSStoppedFrame,
)
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_services import InterruptibleWordTTSService, TTSService
from pipecat.services.tts_service import InterruptibleWordTTSService, TTSService
from pipecat.transcriptions.language import Language
# See .env.example for ElevenLabs configuration needed

2
src/pipecat/services/fal/image.py
View File

@@ -15,7 +15,7 @@ from PIL import Image
from pydantic import BaseModel
from pipecat.frames.frames import ErrorFrame, Frame, URLImageRawFrame
from pipecat.services.ai_services import ImageGenService
from pipecat.services.image_service import ImageGenService
try:
import fal_client

2
src/pipecat/services/fal/stt.py
View File

@@ -11,7 +11,7 @@ from loguru import logger
from pydantic import BaseModel
from pipecat.frames.frames import ErrorFrame, Frame, TranscriptionFrame
from pipecat.services.ai_services import SegmentedSTTService
from pipecat.services.stt_service import SegmentedSTTService
from pipecat.transcriptions.language import Language
from pipecat.utils.time import time_now_iso8601

2
src/pipecat/services/fish/tts.py
View File

@@ -22,7 +22,7 @@ from pipecat.frames.frames import (
TTSStoppedFrame,
)
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_services import InterruptibleTTSService
from pipecat.services.tts_service import InterruptibleTTSService
from pipecat.transcriptions.language import Language
try:

2
src/pipecat/services/gemini_multimodal_live/gemini.py
View File

@@ -50,7 +50,7 @@ from pipecat.processors.aggregators.openai_llm_context import (
OpenAILLMContextFrame,
)
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_services import LLMService
from pipecat.services.llm_service import LLMService
from pipecat.services.openai.llm import (
OpenAIAssistantContextAggregator,
OpenAIUserContextAggregator,

253
src/pipecat/services/gladia.py
View File

@@ -1,253 +0,0 @@
#
# Copyright (c) 2024–2025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import base64
import json
from typing import AsyncGenerator, Optional
import aiohttp
from loguru import logger
from pydantic import BaseModel
from pipecat.frames.frames import (
CancelFrame,
EndFrame,
Frame,
InterimTranscriptionFrame,
StartFrame,
TranscriptionFrame,
)
from pipecat.services.ai_services import STTService
from pipecat.transcriptions.language import Language
from pipecat.utils.time import time_now_iso8601
try:
import websockets
except ModuleNotFoundError as e:
logger.error(f"Exception: {e}")
logger.error(
"In order to use Gladia, you need to `pip install pipecat-ai[gladia]`. Also, set `GLADIA_API_KEY` environment variable."
)
raise Exception(f"Missing module: {e}")
def language_to_gladia_language(language: Language) -> Optional[str]:
BASE_LANGUAGES = {
Language.AF: "af",
Language.AM: "am",
Language.AR: "ar",
Language.AS: "as",
Language.AZ: "az",
Language.BG: "bg",
Language.BN: "bn",
Language.BS: "bs",
Language.CA: "ca",
Language.CS: "cs",
Language.CY: "cy",
Language.DA: "da",
Language.DE: "de",
Language.EL: "el",
Language.EN: "en",
Language.ES: "es",
Language.ET: "et",
Language.EU: "eu",
Language.FA: "fa",
Language.FI: "fi",
Language.FR: "fr",
Language.GA: "ga",
Language.GL: "gl",
Language.GU: "gu",
Language.HE: "he",
Language.HI: "hi",
Language.HR: "hr",
Language.HU: "hu",
Language.HY: "hy",
Language.ID: "id",
Language.IS: "is",
Language.IT: "it",
Language.JA: "ja",
Language.JV: "jv",
Language.KA: "ka",
Language.KK: "kk",
Language.KM: "km",
Language.KN: "kn",
Language.KO: "ko",
Language.LO: "lo",
Language.LT: "lt",
Language.LV: "lv",
Language.MK: "mk",
Language.ML: "ml",
Language.MN: "mn",
Language.MR: "mr",
Language.MS: "ms",
Language.MT: "mt",
Language.MY: "my",
Language.NE: "ne",
Language.NL: "nl",
Language.NO: "no",
Language.OR: "or",
Language.PA: "pa",
Language.PL: "pl",
Language.PS: "ps",
Language.PT: "pt",
Language.RO: "ro",
Language.RU: "ru",
Language.SI: "si",
Language.SK: "sk",
Language.SL: "sl",
Language.SO: "so",
Language.SQ: "sq",
Language.SR: "sr",
Language.SU: "su",
Language.SV: "sv",
Language.SW: "sw",
Language.TA: "ta",
Language.TE: "te",
Language.TH: "th",
Language.TR: "tr",
Language.UK: "uk",
Language.UR: "ur",
Language.UZ: "uz",
Language.VI: "vi",
Language.ZH: "zh",
Language.ZU: "zu",
}
result = BASE_LANGUAGES.get(language)
# If not found in base languages, try to find the base language from a variant
if not result:
# Convert enum value to string and get the base language part (e.g. es-ES -> es)
lang_str = str(language.value)
base_code = lang_str.split("-")[0].lower()
# Look up the base code in our supported languages
result = base_code if base_code in BASE_LANGUAGES.values() else None
return result
class GladiaSTTService(STTService):
class InputParams(BaseModel):
language: Optional[Language] = Language.EN
endpointing: Optional[float] = 0.2
maximum_duration_without_endpointing: Optional[int] = 10
audio_enhancer: Optional[bool] = None
words_accurate_timestamps: Optional[bool] = None
speech_threshold: Optional[float] = 0.99
def __init__(
self,
*,
api_key: str,
url: str = "https://api.gladia.io/v2/live",
confidence: float = 0.5,
sample_rate: Optional[int] = None,
params: InputParams = InputParams(),
**kwargs,
):
super().__init__(sample_rate=sample_rate, **kwargs)
self._api_key = api_key
self._url = url
self._settings = {
"encoding": "wav/pcm",
"bit_depth": 16,
"sample_rate": 0,
"channels": 1,
"language_config": {
"languages": [self.language_to_service_language(params.language)]
if params.language
else [],
"code_switching": False,
},
"endpointing": params.endpointing,
"maximum_duration_without_endpointing": params.maximum_duration_without_endpointing,
"pre_processing": {
"audio_enhancer": params.audio_enhancer,
"speech_threshold": params.speech_threshold,
},
"realtime_processing": {
"words_accurate_timestamps": params.words_accurate_timestamps,
},
}
self._confidence = confidence
self._websocket = None
self._receive_task = None
def language_to_service_language(self, language: Language) -> Optional[str]:
return language_to_gladia_language(language)
async def start(self, frame: StartFrame):
await super().start(frame)
if self._websocket:
return
self._settings["sample_rate"] = self.sample_rate
response = await self._setup_gladia()
self._websocket = await websockets.connect(response["url"])
if not self._receive_task:
self._receive_task = self.create_task(self._receive_task_handler())
async def stop(self, frame: EndFrame):
await super().stop(frame)
await self._send_stop_recording()
if self._websocket:
await self._websocket.close()
self._websocket = None
if self._receive_task:
await self.wait_for_task(self._receive_task)
self._receive_task = None
async def cancel(self, frame: CancelFrame):
await super().cancel(frame)
await self._websocket.close()
if self._receive_task:
await self.cancel_task(self._receive_task)
self._receive_task = None
async def run_stt(self, audio: bytes) -> AsyncGenerator[Frame, None]:
await self.start_processing_metrics()
await self._send_audio(audio)
await self.stop_processing_metrics()
yield None
async def _setup_gladia(self):
async with aiohttp.ClientSession() as session:
async with session.post(
self._url,
headers={"X-Gladia-Key": self._api_key, "Content-Type": "application/json"},
json=self._settings,
) as response:
if response.ok:
return await response.json()
else:
logger.error(
f"Gladia error: {response.status}: {response.text or response.reason}"
)
raise Exception(f"Failed to initialize Gladia session: {response.status}")
async def _send_audio(self, audio: bytes):
data = base64.b64encode(audio).decode("utf-8")
message = {"type": "audio_chunk", "data": {"chunk": data}}
await self._websocket.send(json.dumps(message))
async def _send_stop_recording(self):
await self._websocket.send(json.dumps({"type": "stop_recording"}))
async def _receive_task_handler(self):
async for message in self._websocket:
content = json.loads(message)
if content["type"] == "transcript":
utterance = content["data"]["utterance"]
confidence = utterance.get("confidence", 0)
transcript = utterance["text"]
if confidence >= self._confidence:
if content["data"]["is_final"]:
await self.push_frame(
TranscriptionFrame(transcript, "", time_now_iso8601())
)
else:
await self.push_frame(
InterimTranscriptionFrame(transcript, "", time_now_iso8601())
)

13
src/pipecat/services/gladia/__init__.py Normal file
View File

@@ -0,0 +1,13 @@
#
# Copyright (c) 2024–2025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import sys
from pipecat.services import DeprecatedModuleProxy
from .stt import *
sys.modules[__name__] = DeprecatedModuleProxy(globals(), "gladia", "gladia.stt")

163
src/pipecat/services/gladia/config.py Normal file
View File

@@ -0,0 +1,163 @@
#
# Copyright (c) 2024–2025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
from typing import Any, Dict, List, Optional, Union
from pydantic import BaseModel
from pipecat.transcriptions.language import Language
class LanguageConfig(BaseModel):
"""Configuration for language detection and handling.
Attributes:
languages: List of language codes to use for transcription
code_switching: Whether to auto-detect language changes during transcription
"""
languages: Optional[List[str]] = None
code_switching: Optional[bool] = None
class PreProcessingConfig(BaseModel):
"""Configuration for audio pre-processing options.
Attributes:
speech_threshold: Sensitivity for speech detection (0-1)
"""
speech_threshold: Optional[float] = None
class CustomVocabularyItem(BaseModel):
"""Represents a custom vocabulary item with an intensity value.
Attributes:
value: The vocabulary word or phrase
intensity: The bias intensity for this vocabulary item (0-1)
"""
value: str
intensity: float
class CustomVocabularyConfig(BaseModel):
"""Configuration for custom vocabulary.
Attributes:
vocabulary: List of words/phrases or CustomVocabularyItem objects
default_intensity: Default intensity for simple string vocabulary items
"""
vocabulary: Optional[List[Union[str, CustomVocabularyItem]]] = None
default_intensity: Optional[float] = None
class CustomSpellingConfig(BaseModel):
"""Configuration for custom spelling rules.
Attributes:
spelling_dictionary: Mapping of correct spellings to phonetic variations
"""
spelling_dictionary: Optional[Dict[str, List[str]]] = None
class TranslationConfig(BaseModel):
"""Configuration for real-time translation.
Attributes:
target_languages: List of target language codes for translation
model: Translation model to use ("base" or "enhanced")
match_original_utterances: Whether to align translations with original utterances
"""
target_languages: Optional[List[str]] = None
model: Optional[str] = None
match_original_utterances: Optional[bool] = None
class RealtimeProcessingConfig(BaseModel):
"""Configuration for real-time processing features.
Attributes:
words_accurate_timestamps: Whether to provide per-word timestamps
custom_vocabulary: Whether to enable custom vocabulary
custom_vocabulary_config: Custom vocabulary configuration
custom_spelling: Whether to enable custom spelling
custom_spelling_config: Custom spelling configuration
translation: Whether to enable translation
translation_config: Translation configuration
named_entity_recognition: Whether to enable named entity recognition
sentiment_analysis: Whether to enable sentiment analysis
"""
words_accurate_timestamps: Optional[bool] = None
custom_vocabulary: Optional[bool] = None
custom_vocabulary_config: Optional[CustomVocabularyConfig] = None
custom_spelling: Optional[bool] = None
custom_spelling_config: Optional[CustomSpellingConfig] = None
translation: Optional[bool] = None
translation_config: Optional[TranslationConfig] = None
named_entity_recognition: Optional[bool] = None
sentiment_analysis: Optional[bool] = None
class MessagesConfig(BaseModel):
"""Configuration for controlling which message types are sent via WebSocket.
Attributes:
receive_partial_transcripts: Whether to receive intermediate transcription results
receive_final_transcripts: Whether to receive final transcription results
receive_speech_events: Whether to receive speech begin/end events
receive_pre_processing_events: Whether to receive pre-processing events
receive_realtime_processing_events: Whether to receive real-time processing events
receive_post_processing_events: Whether to receive post-processing events
receive_acknowledgments: Whether to receive acknowledgment messages
receive_errors: Whether to receive error messages
receive_lifecycle_events: Whether to receive lifecycle events
"""
receive_partial_transcripts: Optional[bool] = None
receive_final_transcripts: Optional[bool] = None
receive_speech_events: Optional[bool] = None
receive_pre_processing_events: Optional[bool] = None
receive_realtime_processing_events: Optional[bool] = None
receive_post_processing_events: Optional[bool] = None
receive_acknowledgments: Optional[bool] = None
receive_errors: Optional[bool] = None
receive_lifecycle_events: Optional[bool] = None
class GladiaInputParams(BaseModel):
"""Configuration parameters for the Gladia STT service.
Attributes:
encoding: Audio encoding format
bit_depth: Audio bit depth
channels: Number of audio channels
custom_metadata: Additional metadata to include with requests
endpointing: Silence duration in seconds to mark end of speech
maximum_duration_without_endpointing: Maximum utterance duration without silence
language: DEPRECATED - Use language_config instead
language_config: Detailed language configuration
pre_processing: Audio pre-processing options
realtime_processing: Real-time processing features
messages_config: WebSocket message filtering options
"""
encoding: Optional[str] = "wav/pcm"
bit_depth: Optional[int] = 16
channels: Optional[int] = 1
custom_metadata: Optional[Dict[str, Any]] = None
endpointing: Optional[float] = None
maximum_duration_without_endpointing: Optional[int] = 10
language: Optional[Language] = None # Deprecated
language_config: Optional[LanguageConfig] = None
pre_processing: Optional[PreProcessingConfig] = None
realtime_processing: Optional[RealtimeProcessingConfig] = None
messages_config: Optional[MessagesConfig] = None

365
src/pipecat/services/gladia/stt.py Normal file
View File

@@ -0,0 +1,365 @@
#
# Copyright (c) 2024–2025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import base64
import json
import warnings
from typing import Any, AsyncGenerator, Dict, Optional
import aiohttp
from loguru import logger
from pipecat.frames.frames import (
CancelFrame,
EndFrame,
Frame,
InterimTranscriptionFrame,
StartFrame,
TranscriptionFrame,
)
from pipecat.services.gladia.config import GladiaInputParams
from pipecat.services.stt_service import STTService
from pipecat.transcriptions.language import Language
from pipecat.utils.time import time_now_iso8601
try:
import websockets
except ModuleNotFoundError as e:
logger.error(f"Exception: {e}")
logger.error("In order to use Gladia, you need to `pip install pipecat-ai[gladia]`.")
raise Exception(f"Missing module: {e}")
def language_to_gladia_language(language: Language) -> Optional[str]:
"""Convert a Language enum to Gladia's language code format.
Args:
language: The Language enum value to convert
Returns:
The Gladia language code string or None if not supported
"""
BASE_LANGUAGES = {
Language.AF: "af",
Language.AM: "am",
Language.AR: "ar",
Language.AS: "as",
Language.AZ: "az",
Language.BA: "ba",
Language.BE: "be",
Language.BG: "bg",
Language.BN: "bn",
Language.BO: "bo",
Language.BR: "br",
Language.BS: "bs",
Language.CA: "ca",
Language.CS: "cs",
Language.CY: "cy",
Language.DA: "da",
Language.DE: "de",
Language.EL: "el",
Language.EN: "en",
Language.ES: "es",
Language.ET: "et",
Language.EU: "eu",
Language.FA: "fa",
Language.FI: "fi",
Language.FO: "fo",
Language.FR: "fr",
Language.GL: "gl",
Language.GU: "gu",
Language.HA: "ha",
Language.HAW: "haw",
Language.HE: "he",
Language.HI: "hi",
Language.HR: "hr",
Language.HT: "ht",
Language.HU: "hu",
Language.HY: "hy",
Language.ID: "id",
Language.IS: "is",
Language.IT: "it",
Language.JA: "ja",
Language.JV: "jv",
Language.KA: "ka",
Language.KK: "kk",
Language.KM: "km",
Language.KN: "kn",
Language.KO: "ko",
Language.LA: "la",
Language.LB: "lb",
Language.LN: "ln",
Language.LO: "lo",
Language.LT: "lt",
Language.LV: "lv",
Language.MG: "mg",
Language.MI: "mi",
Language.MK: "mk",
Language.ML: "ml",
Language.MN: "mn",
Language.MR: "mr",
Language.MS: "ms",
Language.MT: "mt",
Language.MY_MR: "mymr",
Language.NE: "ne",
Language.NL: "nl",
Language.NN: "nn",
Language.NO: "no",
Language.OC: "oc",
Language.PA: "pa",
Language.PL: "pl",
Language.PS: "ps",
Language.PT: "pt",
Language.RO: "ro",
Language.RU: "ru",
Language.SA: "sa",
Language.SD: "sd",
Language.SI: "si",
Language.SK: "sk",
Language.SL: "sl",
Language.SN: "sn",
Language.SO: "so",
Language.SQ: "sq",
Language.SR: "sr",
Language.SU: "su",
Language.SV: "sv",
Language.SW: "sw",
Language.TA: "ta",
Language.TE: "te",
Language.TG: "tg",
Language.TH: "th",
Language.TK: "tk",
Language.TL: "tl",
Language.TR: "tr",
Language.TT: "tt",
Language.UK: "uk",
Language.UR: "ur",
Language.UZ: "uz",
Language.VI: "vi",
Language.YI: "yi",
Language.YO: "yo",
Language.ZH: "zh",
}
result = BASE_LANGUAGES.get(language)
# If not found in base languages, try to find the base language from a variant
if not result:
# Convert enum value to string and get the base language part (e.g. es-ES -> es)
lang_str = str(language.value)
base_code = lang_str.split("-")[0].lower()
# Look up the base code in our supported languages
result = base_code if base_code in BASE_LANGUAGES.values() else None
return result
# Deprecation warning for nested InputParams
class _InputParamsDescriptor:
"""Descriptor for backward compatibility with deprecation warning."""
def __get__(self, obj, objtype=None):
warnings.warn(
"GladiaSTTService.InputParams is deprecated and will be removed in a future version. "
"Import and use GladiaInputParams directly instead.",
DeprecationWarning,
stacklevel=2,
)
return GladiaInputParams
class GladiaSTTService(STTService):
"""Speech-to-Text service using Gladia's API.
This service connects to Gladia's WebSocket API for real-time transcription
with support for multiple languages, custom vocabulary, and various processing options.
For complete API documentation, see: https://docs.gladia.io/api-reference/v2/live/init
"""
# Maintain backward compatibility
InputParams = _InputParamsDescriptor()
def __init__(
self,
*,
api_key: str,
url: str = "https://api.gladia.io/v2/live",
confidence: float = 0.5,
sample_rate: Optional[int] = None,
model: str = "solaria-1",
params: GladiaInputParams = GladiaInputParams(),
**kwargs,
):
"""Initialize the Gladia STT service.
Args:
api_key: Gladia API key
url: Gladia API URL
confidence: Minimum confidence threshold for transcriptions
sample_rate: Audio sample rate in Hz
model: Model to use ("solaria-1", "solaria-mini-1", "fast",
or "accurate")
params: Additional configuration parameters
**kwargs: Additional arguments passed to the STTService
"""
super().__init__(sample_rate=sample_rate, **kwargs)
# Warn about deprecated language parameter if it's used
if params.language is not None:
warnings.warn(
"The 'language' parameter is deprecated and will be removed in a future version. "
"Use 'language_config' instead.",
DeprecationWarning,
stacklevel=2,
)
self._api_key = api_key
self._url = url
self.set_model_name(model)
self._confidence = confidence
self._params = params
self._websocket = None
self._receive_task = None
def language_to_service_language(self, language: Language) -> Optional[str]:
"""Convert pipecat Language enum to Gladia's language code."""
return language_to_gladia_language(language)
def _prepare_settings(self) -> Dict[str, Any]:
settings = {
"encoding": self._params.encoding or "wav/pcm",
"bit_depth": self._params.bit_depth or 16,
"sample_rate": self.sample_rate,
"channels": self._params.channels or 1,
"model": self._model_name,
}
# Add custom_metadata if provided
if self._params.custom_metadata:
settings["custom_metadata"] = self._params.custom_metadata
# Add endpointing parameters if provided
if self._params.endpointing is not None:
settings["endpointing"] = self._params.endpointing
if self._params.maximum_duration_without_endpointing is not None:
settings["maximum_duration_without_endpointing"] = (
self._params.maximum_duration_without_endpointing
)
# Add language configuration (prioritize language_config over deprecated language)
if self._params.language_config:
settings["language_config"] = self._params.language_config.model_dump(exclude_none=True)
elif self._params.language: # Backward compatibility for deprecated parameter
language_code = self.language_to_service_language(self._params.language)
if language_code:
settings["language_config"] = {
"languages": [language_code],
"code_switching": False,
}
# Add pre_processing configuration if provided
if self._params.pre_processing:
settings["pre_processing"] = self._params.pre_processing.model_dump(exclude_none=True)
# Add realtime_processing configuration if provided
if self._params.realtime_processing:
settings["realtime_processing"] = self._params.realtime_processing.model_dump(
exclude_none=True
)
# Add messages_config if provided
if self._params.messages_config:
settings["messages_config"] = self._params.messages_config.model_dump(exclude_none=True)
return settings
async def start(self, frame: StartFrame):
"""Start the Gladia STT websocket connection."""
await super().start(frame)
if self._websocket:
return
settings = self._prepare_settings()
response = await self._setup_gladia(settings)
self._websocket = await websockets.connect(response["url"])
if not self._receive_task:
self._receive_task = self.create_task(self._receive_task_handler())
async def stop(self, frame: EndFrame):
"""Stop the Gladia STT websocket connection."""
await super().stop(frame)
await self._send_stop_recording()
if self._websocket:
await self._websocket.close()
self._websocket = None
if self._receive_task:
await self.wait_for_task(self._receive_task)
self._receive_task = None
async def cancel(self, frame: CancelFrame):
"""Cancel the Gladia STT websocket connection."""
await super().cancel(frame)
await self._websocket.close()
if self._receive_task:
await self.cancel_task(self._receive_task)
self._receive_task = None
async def run_stt(self, audio: bytes) -> AsyncGenerator[Frame, None]:
"""Run speech-to-text on audio data."""
await self.start_processing_metrics()
await self._send_audio(audio)
await self.stop_processing_metrics()
yield None
async def _setup_gladia(self, settings: Dict[str, Any]):
async with aiohttp.ClientSession() as session:
async with session.post(
self._url,
headers={"X-Gladia-Key": self._api_key, "Content-Type": "application/json"},
json=settings,
) as response:
if response.ok:
return await response.json()
else:
error_text = await response.text()
logger.error(
f"Gladia error: {response.status}: {error_text or response.reason}"
)
raise Exception(
f"Failed to initialize Gladia session: {response.status} - {error_text}"
)
async def _send_audio(self, audio: bytes):
data = base64.b64encode(audio).decode("utf-8")
message = {"type": "audio_chunk", "data": {"chunk": data}}
await self._websocket.send(json.dumps(message))
async def _send_stop_recording(self):
if self._websocket and not self._websocket.closed:
await self._websocket.send(json.dumps({"type": "stop_recording"}))
async def _receive_task_handler(self):
try:
async for message in self._websocket:
content = json.loads(message)
if content["type"] == "transcript":
utterance = content["data"]["utterance"]
confidence = utterance.get("confidence", 0)
transcript = utterance["text"]
if confidence >= self._confidence:
if content["data"]["is_final"]:
await self.push_frame(
TranscriptionFrame(transcript, "", time_now_iso8601())
)
else:
await self.push_frame(
InterimTranscriptionFrame(transcript, "", time_now_iso8601())
)
except websockets.exceptions.ConnectionClosed:
# Expected when closing the connection
pass
except Exception as e:
logger.error(f"Error in Gladia WebSocket handler: {e}")

2
src/pipecat/services/google/image.py
View File

@@ -17,7 +17,7 @@ from PIL import Image
from pydantic import BaseModel, Field
from pipecat.frames.frames import ErrorFrame, Frame, URLImageRawFrame
from pipecat.services.ai_services import ImageGenService
from pipecat.services.image_service import ImageGenService
try:
from google import genai

2
src/pipecat/services/google/llm.py
View File

@@ -44,8 +44,8 @@ from pipecat.processors.aggregators.openai_llm_context import (
OpenAILLMContextFrame,
)
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_services import LLMService
from pipecat.services.google.frames import LLMSearchResponseFrame
from pipecat.services.llm_service import LLMService
from pipecat.services.openai.llm import (
OpenAIAssistantContextAggregator,
OpenAIUserContextAggregator,

2
src/pipecat/services/google/stt.py
View File

@@ -26,7 +26,7 @@ from pipecat.frames.frames import (
StartFrame,
TranscriptionFrame,
)
from pipecat.services.ai_services import STTService
from pipecat.services.stt_service import STTService
from pipecat.transcriptions.language import Language
from pipecat.utils.time import time_now_iso8601

2
src/pipecat/services/google/tts.py
View File

@@ -23,7 +23,7 @@ from pipecat.frames.frames import (
TTSStartedFrame,
TTSStoppedFrame,
)
from pipecat.services.ai_services import TTSService
from pipecat.services.tts_service import TTSService
from pipecat.transcriptions.language import Language
try:

2
src/pipecat/services/groq/tts.py
View File

@@ -10,7 +10,7 @@ from loguru import logger
from pydantic import BaseModel
from pipecat.frames.frames import Frame, TTSAudioRawFrame, TTSStartedFrame, TTSStoppedFrame
from pipecat.services.ai_services import TTSService
from pipecat.services.tts_service import TTSService
from pipecat.transcriptions.language import Language
try:

33
src/pipecat/services/image_service.py Normal file
View File

@@ -0,0 +1,33 @@
#
# Copyright (c) 2024–2025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
from abc import abstractmethod
from typing import AsyncGenerator
from pipecat.frames.frames import Frame, TextFrame
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_service import AIService
class ImageGenService(AIService):
def __init__(self, **kwargs):
super().__init__(**kwargs)
# Renders the image. Returns an Image object.
@abstractmethod
async def run_image_gen(self, prompt: str) -> AsyncGenerator[Frame, None]:
pass
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
if isinstance(frame, TextFrame):
await self.push_frame(frame, direction)
await self.start_processing_metrics()
await self.process_generator(self.run_image_gen(frame.text))
await self.stop_processing_metrics()
else:
await self.push_frame(frame, direction)

257
src/pipecat/services/llm_service.py Normal file
View File

@@ -0,0 +1,257 @@
#
# Copyright (c) 2024–2025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import asyncio
from dataclasses import dataclass
from typing import Any, Mapping, Optional, Set, Tuple, Type
from loguru import logger
from pipecat.adapters.base_llm_adapter import BaseLLMAdapter
from pipecat.adapters.services.open_ai_adapter import OpenAILLMAdapter
from pipecat.frames.frames import (
Frame,
FunctionCallCancelFrame,
FunctionCallInProgressFrame,
FunctionCallResultFrame,
StartInterruptionFrame,
UserImageRequestFrame,
)
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_service import AIService
@dataclass
class FunctionEntry:
function_name: Optional[str]
callback: Any # TODO(aleix): add proper typing.
cancel_on_interruption: bool
class LLMService(AIService):
"""This class is a no-op but serves as a base class for LLM services."""
# OpenAILLMAdapter is used as the default adapter since it aligns with most LLM implementations.
# However, subclasses should override this with a more specific adapter when necessary.
adapter_class: Type[BaseLLMAdapter] = OpenAILLMAdapter
def __init__(self, **kwargs):
super().__init__(**kwargs)
self._functions = {}
self._start_callbacks = {}
self._adapter = self.adapter_class()
self._function_call_tasks: Set[Tuple[asyncio.Task, str, str]] = set()
self._register_event_handler("on_completion_timeout")
def get_llm_adapter(self) -> BaseLLMAdapter:
return self._adapter
def create_context_aggregator(
self,
context: OpenAILLMContext,
*,
user_kwargs: Mapping[str, Any] = {},
assistant_kwargs: Mapping[str, Any] = {},
) -> Any:
pass
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
if isinstance(frame, StartInterruptionFrame):
await self._handle_interruptions(frame)
async def _handle_interruptions(self, frame: StartInterruptionFrame):
for function_name, entry in self._functions.items():
if entry.cancel_on_interruption:
await self._cancel_function_call(function_name)
def register_function(
self,
function_name: Optional[str],
callback: Any,
start_callback=None,
*,
cancel_on_interruption: bool = False,
):
# Registering a function with the function_name set to None will run that callback
# for all functions
self._functions[function_name] = FunctionEntry(
function_name=function_name,
callback=callback,
cancel_on_interruption=cancel_on_interruption,
)
# Start callbacks are now deprecated.
if start_callback:
import warnings
with warnings.catch_warnings():
warnings.simplefilter("always")
warnings.warn(
"Parameter 'start_callback' is deprecated, just put your code on top of the actual function call instead.",
DeprecationWarning,
)
self._start_callbacks[function_name] = start_callback
def unregister_function(self, function_name: Optional[str]):
del self._functions[function_name]
if self._start_callbacks[function_name]:
del self._start_callbacks[function_name]
def has_function(self, function_name: str):
if None in self._functions.keys():
return True
return function_name in self._functions.keys()
async def call_function(
self,
*,
context: OpenAILLMContext,
tool_call_id: str,
function_name: str,
arguments: str,
run_llm: bool = True,
):
if not function_name in self._functions.keys() and not None in self._functions.keys():
return
task = self.create_task(
self._run_function_call(context, tool_call_id, function_name, arguments, run_llm)
)
self._function_call_tasks.add((task, tool_call_id, function_name))
task.add_done_callback(self._function_call_task_finished)
async def call_start_function(self, context: OpenAILLMContext, function_name: str):
if function_name in self._start_callbacks.keys():
await self._start_callbacks[function_name](function_name, self, context)
elif None in self._start_callbacks.keys():
return await self._start_callbacks[None](function_name, self, context)
async def request_image_frame(
self,
user_id: str,
*,
function_name: Optional[str] = None,
tool_call_id: Optional[str] = None,
text_content: Optional[str] = None,
):
await self.push_frame(
UserImageRequestFrame(
user_id=user_id,
function_name=function_name,
tool_call_id=tool_call_id,
context=text_content,
),
FrameDirection.UPSTREAM,
)
async def _run_function_call(
self,
context: OpenAILLMContext,
tool_call_id: str,
function_name: str,
arguments: str,
run_llm: bool = True,
):
if function_name in self._functions.keys():
entry = self._functions[function_name]
elif None in self._functions.keys():
entry = self._functions[None]
else:
return
logger.debug(
f"{self} Calling function [{function_name}:{tool_call_id}] with arguments {arguments}"
)
# NOTE(aleix): This needs to be removed after we remove the deprecation.
await self.call_start_function(context, function_name)
# Push a SystemFrame downstream. This frame will let our assistant context aggregator
# know that we are in the middle of a function call. Some contexts/aggregators may
# not need this. But some definitely do (Anthropic, for example).
# Also push a SystemFrame upstream for use by other processors, like STTMuteFilter.
progress_frame_downstream = FunctionCallInProgressFrame(
function_name=function_name,
tool_call_id=tool_call_id,
arguments=arguments,
cancel_on_interruption=entry.cancel_on_interruption,
)
progress_frame_upstream = FunctionCallInProgressFrame(
function_name=function_name,
tool_call_id=tool_call_id,
arguments=arguments,
cancel_on_interruption=entry.cancel_on_interruption,
)
# Push frame both downstream and upstream
await self.push_frame(progress_frame_downstream, FrameDirection.DOWNSTREAM)
await self.push_frame(progress_frame_upstream, FrameDirection.UPSTREAM)
# Define a callback function that pushes a FunctionCallResultFrame upstream & downstream.
async def function_call_result_callback(result, *, properties=None):
result_frame_downstream = FunctionCallResultFrame(
function_name=function_name,
tool_call_id=tool_call_id,
arguments=arguments,
result=result,
properties=properties,
)
result_frame_upstream = FunctionCallResultFrame(
function_name=function_name,
tool_call_id=tool_call_id,
arguments=arguments,
result=result,
properties=properties,
)
await self.push_frame(result_frame_downstream, FrameDirection.DOWNSTREAM)
await self.push_frame(result_frame_upstream, FrameDirection.UPSTREAM)
await entry.callback(
function_name, tool_call_id, arguments, self, context, function_call_result_callback
)
async def _cancel_function_call(self, function_name: str):
cancelled_tasks = set()
for task, tool_call_id, name in self._function_call_tasks:
if name == function_name:
# We remove the callback because we are going to cancel the task
# now, otherwise we will be removing it from the set while we
# are iterating.
task.remove_done_callback(self._function_call_task_finished)
logger.debug(f"{self} Cancelling function call [{name}:{tool_call_id}]...")
await self.cancel_task(task)
frame = FunctionCallCancelFrame(
function_name=function_name, tool_call_id=tool_call_id
)
await self.push_frame(frame)
logger.debug(f"{self} Function call [{name}:{tool_call_id}] has been cancelled")
cancelled_tasks.add(task)
# Remove all cancelled tasks from our set.
for task in cancelled_tasks:
self._function_call_task_finished(task)
def _function_call_task_finished(self, task: asyncio.Task):
tuple_to_remove = next((t for t in self._function_call_tasks if t[0] == task), None)
if tuple_to_remove:
self._function_call_tasks.discard(tuple_to_remove)
# The task is finished so this should exit immediately. We need to
# do this because otherwise the task manager would report a dangling
# task if we don't remove it.
asyncio.run_coroutine_threadsafe(self.wait_for_task(task), self.get_event_loop())

2
src/pipecat/services/lmnt/tts.py
View File

@@ -21,7 +21,7 @@ from pipecat.frames.frames import (
TTSStoppedFrame,
)
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_services import InterruptibleTTSService
from pipecat.services.tts_service import InterruptibleTTSService
from pipecat.transcriptions.language import Language
# See .env.example for LMNT configuration needed

2
src/pipecat/services/moondream/vision.py
View File

@@ -11,7 +11,7 @@ from loguru import logger
from PIL import Image
from pipecat.frames.frames import ErrorFrame, Frame, TextFrame, VisionImageRawFrame
from pipecat.services.ai_services import VisionService
from pipecat.services.vision_service import VisionService
try:
import torch

2
src/pipecat/services/neuphonic/tts.py
View File

@@ -27,7 +27,7 @@ from pipecat.frames.frames import (
TTSStoppedFrame,
)
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_services import InterruptibleTTSService, TTSService
from pipecat.services.tts_service import InterruptibleTTSService, TTSService
from pipecat.transcriptions.language import Language
try:

2
src/pipecat/services/openai/base_llm.py
View File

@@ -34,7 +34,7 @@ from pipecat.processors.aggregators.openai_llm_context import (
OpenAILLMContextFrame,
)
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_services import LLMService
from pipecat.services.llm_service import LLMService
class OpenAIUnhandledFunctionException(Exception):

2
src/pipecat/services/openai/image.py
View File

@@ -17,7 +17,7 @@ from pipecat.frames.frames import (
Frame,
URLImageRawFrame,
)
from pipecat.services.ai_services import ImageGenService
from pipecat.services.image_service import ImageGenService
class OpenAIImageGenService(ImageGenService):

11
src/pipecat/services/openai/tts.py
View File

@@ -17,17 +17,24 @@ from pipecat.frames.frames import (
TTSStartedFrame,
TTSStoppedFrame,
)
from pipecat.services.ai_services import TTSService
from pipecat.services.tts_service import TTSService
ValidVoice = Literal["alloy", "echo", "fable", "onyx", "nova", "shimmer"]
ValidVoice = Literal[
"alloy", "ash", "ballad", "coral", "echo", "fable", "onyx", "nova", "sage", "shimmer", "verse"
]
VALID_VOICES: Dict[str, ValidVoice] = {
"alloy": "alloy",
"ash": "ash",
"ballad": "ballad",
"coral": "coral",
"echo": "echo",
"fable": "fable",
"onyx": "onyx",
"nova": "nova",
"sage": "sage",
"shimmer": "shimmer",
"verse": "verse",
}

2
src/pipecat/services/openai_realtime_beta/openai.py
View File

@@ -53,7 +53,7 @@ from pipecat.processors.aggregators.openai_llm_context import (
OpenAILLMContextFrame,
)
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_services import LLMService
from pipecat.services.llm_service import LLMService
from pipecat.services.openai.llm import OpenAIContextAggregatorPair
from pipecat.utils.time import time_now_iso8601

2
src/pipecat/services/piper/tts.py
View File

@@ -16,7 +16,7 @@ from pipecat.frames.frames import (
TTSStartedFrame,
TTSStoppedFrame,
)
from pipecat.services.ai_services import TTSService
from pipecat.services.tts_service import TTSService
# This assumes a running TTS service running: https://github.com/rhasspy/piper/blob/master/src/python_run/README_http.md

2
src/pipecat/services/playht/tts.py
View File

@@ -27,7 +27,7 @@ from pipecat.frames.frames import (
TTSStoppedFrame,
)
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_services import InterruptibleTTSService, TTSService
from pipecat.services.tts_service import InterruptibleTTSService, TTSService
from pipecat.transcriptions.language import Language
try:

2
src/pipecat/services/rime/tts.py
View File

@@ -25,7 +25,7 @@ from pipecat.frames.frames import (
TTSStoppedFrame,
)
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_services import AudioContextWordTTSService, TTSService
from pipecat.services.tts_service import AudioContextWordTTSService, TTSService
from pipecat.transcriptions.language import Language
from pipecat.utils.text.base_text_aggregator import BaseTextAggregator
from pipecat.utils.text.skip_tags_aggregator import SkipTagsAggregator

2
src/pipecat/services/riva/stt.py
View File

@@ -18,7 +18,7 @@ from pipecat.frames.frames import (
StartFrame,
TranscriptionFrame,
)
from pipecat.services.ai_services import STTService
from pipecat.services.stt_service import STTService
from pipecat.transcriptions.language import Language
from pipecat.utils.time import time_now_iso8601

2
src/pipecat/services/riva/tts.py
View File

@@ -16,7 +16,7 @@ from pipecat.frames.frames import (
TTSStartedFrame,
TTSStoppedFrame,
)
from pipecat.services.ai_services import TTSService
from pipecat.services.tts_service import TTSService
from pipecat.transcriptions.language import Language
try:

171
src/pipecat/services/stt_service.py Normal file
View File

@@ -0,0 +1,171 @@
#
# Copyright (c) 2024–2025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import io
import wave
from abc import abstractmethod
from typing import Any, AsyncGenerator, Dict, Mapping, Optional
from loguru import logger
from pipecat.frames.frames import (
AudioRawFrame,
Frame,
StartFrame,
STTMuteFrame,
STTUpdateSettingsFrame,
UserStartedSpeakingFrame,
UserStoppedSpeakingFrame,
)
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_service import AIService
from pipecat.transcriptions.language import Language
class STTService(AIService):
"""STTService is a base class for speech-to-text services."""
def __init__(
self,
audio_passthrough=False,
# STT input sample rate
sample_rate: Optional[int] = None,
**kwargs,
):
super().__init__(**kwargs)
self._audio_passthrough = audio_passthrough
self._init_sample_rate = sample_rate
self._sample_rate = 0
self._settings: Dict[str, Any] = {}
self._muted: bool = False
@property
def is_muted(self) -> bool:
"""Returns whether the STT service is currently muted."""
return self._muted
@property
def sample_rate(self) -> int:
return self._sample_rate
async def set_model(self, model: str):
self.set_model_name(model)
async def set_language(self, language: Language):
pass
@abstractmethod
async def run_stt(self, audio: bytes) -> AsyncGenerator[Frame, None]:
"""Returns transcript as a string"""
pass
async def start(self, frame: StartFrame):
await super().start(frame)
self._sample_rate = self._init_sample_rate or frame.audio_in_sample_rate
async def _update_settings(self, settings: Mapping[str, Any]):
logger.info(f"Updating STT settings: {self._settings}")
for key, value in settings.items():
if key in self._settings:
logger.info(f"Updating STT setting {key} to: [{value}]")
self._settings[key] = value
if key == "language":
await self.set_language(value)
elif key == "model":
self.set_model_name(value)
else:
logger.warning(f"Unknown setting for STT service: {key}")
async def process_audio_frame(self, frame: AudioRawFrame, direction: FrameDirection):
if self._muted:
return
await self.process_generator(self.run_stt(frame.audio))
async def process_frame(self, frame: Frame, direction: FrameDirection):
"""Processes a frame of audio data, either buffering or transcribing it."""
await super().process_frame(frame, direction)
if isinstance(frame, AudioRawFrame):
# In this service we accumulate audio internally and at the end we
# push a TextFrame. We also push audio downstream in case someone
# else needs it.
await self.process_audio_frame(frame, direction)
if self._audio_passthrough:
await self.push_frame(frame, direction)
elif isinstance(frame, STTUpdateSettingsFrame):
await self._update_settings(frame.settings)
elif isinstance(frame, STTMuteFrame):
self._muted = frame.mute
logger.debug(f"STT service {'muted' if frame.mute else 'unmuted'}")
else:
await self.push_frame(frame, direction)
class SegmentedSTTService(STTService):
"""SegmentedSTTService is an STTService that uses VAD events to detect
speech and will run speech-to-text on speech segments only, instead of a
continous stream. Since it uses VAD it means that VAD needs to be enabled in
the pipeline.
This service always keeps a small audio buffer to take into account that VAD
events are delayed from when the user speech really starts.
"""
def __init__(self, *, sample_rate: Optional[int] = None, **kwargs):
super().__init__(sample_rate=sample_rate, **kwargs)
self._content = None
self._wave = None
self._audio_buffer = bytearray()
self._audio_buffer_size_1s = 0
self._user_speaking = False
async def start(self, frame: StartFrame):
await super().start(frame)
self._audio_buffer_size_1s = self.sample_rate * 2
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
if isinstance(frame, UserStartedSpeakingFrame):
await self._handle_user_started_speaking(frame)
elif isinstance(frame, UserStoppedSpeakingFrame):
await self._handle_user_stopped_speaking(frame)
async def _handle_user_started_speaking(self, frame: UserStartedSpeakingFrame):
if frame.emulated:
return
self._user_speaking = True
async def _handle_user_stopped_speaking(self, frame: UserStoppedSpeakingFrame):
if frame.emulated:
return
self._user_speaking = False
content = io.BytesIO()
wav = wave.open(content, "wb")
wav.setsampwidth(2)
wav.setnchannels(1)
wav.setframerate(self.sample_rate)
wav.writeframes(self._audio_buffer)
wav.close()
content.seek(0)
await self.process_generator(self.run_stt(content.read()))
# Start clean.
self._audio_buffer.clear()
async def process_audio_frame(self, frame: AudioRawFrame, direction: FrameDirection):
# If the user is speaking the audio buffer will keep growing.
self._audio_buffer += frame.audio
# If the user is not speaking we keep just a little bit of audio.
if not self._user_speaking and len(self._audio_buffer) > self._audio_buffer_size_1s:
discarded = len(self._audio_buffer) - self._audio_buffer_size_1s
self._audio_buffer = self._audio_buffer[discarded:]

Some files were not shown because too many files have changed in this diff Show More