Compare commits

...

267 Commits

Author SHA1 Message Date
James Hush
b29ac3c7a8 Remove logs 2025-01-23 16:31:41 +08:00
James Hush
5222488fb5 Have a default transfer 2025-01-23 16:24:40 +08:00
James Hush
c2fef9584b Add call transfer to bot_daily 2025-01-23 15:54:53 +08:00
Mark Backman
fc6aa6eae8 Merge pull request #1060 from chhao01/patch-1
[bug]TypeError: object of type 'NoneType' has no len()
2025-01-22 19:14:35 -05:00
Mark Backman
ddd5bf70ab Merge pull request #1061 from Allenmylath/patch-21
Update README.md
2025-01-22 19:13:15 -05:00
allenmylath
aa59744444 Update examples/README.md
Co-authored-by: Mark Backman <m.backman@gmail.com>
2025-01-23 05:38:37 +05:30
chadbailey59
067ddfe505 Storytelling chatbot updates (#1066)
* initial changes for gemini storybot

* storybot updates for gemini

* more storybot updates

* interim interruptible commit

* cleanup

* cleanup

* cleanup

* cleanup
2025-01-22 15:20:21 -06:00
Mark Backman
a64df978e7 Merge pull request #1046 from pipecat-ai/mb/transcript-tts
Modified `TranscriptProcessor` to use `TTSTextFrame`s
2025-01-22 15:46:01 -05:00
Mark Backman
7167719761 Emit a transcription callback when receiving a CancelFrame, update examples accordingly 2025-01-22 14:56:29 -05:00
Mark Backman
e1430be9f9 Code review fixes 2025-01-22 14:56:29 -05:00
Mark Backman
c2fe8e7fdb Updated CHANGELOG 2025-01-22 14:56:28 -05:00
Mark Backman
31c77d8e35 Update examples for the updated TranscriptProcessor 2025-01-22 14:56:00 -05:00
Mark Backman
2a60d54830 Update the AssistantTranscriptProcessor to use TTSTextFrames in place of OpenAILLMContextFrames 2025-01-22 14:56:00 -05:00
Aleix Conchillo Flaqué
b3c99887dc Merge pull request #1068 from Canonical-AI-Inc/import-fix
Fixing missing import
2025-01-22 11:37:49 -08:00
Mark Backman
38ad75cc17 Merge pull request #1065 from pipecat-ai/mb/fix-openai_realtime-function-calling
OpenAIRealtimeBetaLLMService: Fixed an error in function calling
2025-01-22 14:37:01 -05:00
Adrian Cowham
2debac314c fixing missing import 2025-01-22 11:06:53 -08:00
Mark Backman
e0c9a1a1a2 Merge pull request #1041 from Allenmylath/patch-20
Update bot.py
2025-01-22 09:18:19 -05:00
allenmylath
4cdcca588e Update examples/moondream-chatbot/bot.py
Co-authored-by: Mark Backman <m.backman@gmail.com>
2025-01-22 19:40:12 +05:30
allenmylath
a90e81e2eb Update examples/moondream-chatbot/bot.py
Co-authored-by: Mark Backman <m.backman@gmail.com>
2025-01-22 19:38:36 +05:30
Mark Backman
0ba60c9e28 Merge pull request #975 from imsakg/main
fix(gemini): prevent non-audio modality processing
2025-01-22 09:03:18 -05:00
Mark Backman
5ca5fbd825 OpenAIRealtimeBetaLLMService: Fixed an error in function calling 2025-01-22 08:54:03 -05:00
allenmylath
2b52e2c109 Update README.md
Silero-tts changed to VAD, also description regarding session handling added to websocket chatbot
2025-01-22 14:42:35 +05:30
Cheng Hao
7e8fc2e7e2 [bug]TypeError: object of type 'NoneType' has no len()
Sometimes the chunk.choices is None, and I got exception like: 
```
TypeError: object of type 'NoneType' has no len()
```
2025-01-22 15:31:27 +08:00
Aleix Conchillo Flaqué
0d79a9eaa6 update CHANGELOG.md 2025-01-21 18:00:10 -08:00
Aleix Conchillo Flaqué
f89b9ec23f Merge pull request #1057 from pipecat-ai/aleix/replace-resampy-soxr
improve audio resampling by switching from resampy to soxr
2025-01-21 17:52:49 -08:00
Mark Backman
20d5824e56 Merge pull request #1058 from pipecat-ai/mb/fix-trace-log 2025-01-21 20:44:50 -05:00
Aleix Conchillo Flaqué
f23baa78d8 test-requirements: add soxr and remove resampy 2025-01-21 17:40:17 -08:00
Aleix Conchillo Flaqué
cacd6ba3fa improve audio resampling by switching from resampy to soxr 2025-01-21 17:40:17 -08:00
Aleix Conchillo Flaqué
f87ecd3a51 Merge pull request #1048 from pipecat-ai/aleix/add-unittest-utils
tests: add some initial run_test() utilities
2025-01-21 17:39:06 -08:00
Mark Backman
b96a922aa8 Fix trace log line for resume_processing_frames 2025-01-21 18:15:03 -05:00
Aleix Conchillo Flaqué
401d3ff267 tests: added PipelineTask tests 2025-01-21 11:45:43 -08:00
Aleix Conchillo Flaqué
ab4221a4db task: added BaseTask 2025-01-21 11:45:43 -08:00
Aleix Conchillo Flaqué
bd6f82cf94 task: allow specifying heartbeat period 2025-01-21 11:45:43 -08:00
Aleix Conchillo Flaqué
dd21b424d6 pyproject: ignore 'audioop' deprecation warning 2025-01-21 10:27:34 -08:00
Aleix Conchillo Flaqué
76884877dd tests: add pytest-asyncio dependency 2025-01-21 10:23:19 -08:00
Aleix Conchillo Flaqué
0d6c680133 README: add unit tests badge 2025-01-21 10:14:37 -08:00
Aleix Conchillo Flaqué
a27fe4bde2 tests: move test_ai_services to test_utils_string 2025-01-21 10:06:14 -08:00
Aleix Conchillo Flaqué
177cb2ca8b tests: initial pipeline and parallelpipeline tests 2025-01-21 09:57:54 -08:00
Aleix Conchillo Flaqué
3c970a3cee tests: add more filter tests 2025-01-21 09:43:57 -08:00
Aleix Conchillo Flaqué
af02f8f1cd filters(frame_filter): allow more than one frame 2025-01-21 09:43:33 -08:00
Aleix Conchillo Flaqué
2e0fb198bf frame_processor: allow pushing more frames after EndFrame
This can be useful for testing purposes. In real practice, there shouldn't be
any frames after an EndFrame is pushed.
2025-01-21 09:42:15 -08:00
Filipi da Silva Fuchter
4f758c5a3b Merge pull request #1050 from pipecat-ai/fix_rtvi_warning_msg
Ignoring transport messages that are not intended to RTVI.
2025-01-21 13:36:50 -03:00
Filipi Fuchter
3e0836b340 Ignoring transport messages that are not intended to RTVI. 2025-01-21 10:08:14 -03:00
Aleix Conchillo Flaqué
2f23693bf3 tests: fix test_protobuf_serializer.py 2025-01-20 18:39:59 -08:00
Aleix Conchillo Flaqué
b7dd9748cf serializers: fix special fix initialization 2025-01-20 18:39:41 -08:00
Aleix Conchillo Flaqué
d4d9c3b7ae tests: fix test_aggregators.py 2025-01-20 18:16:14 -08:00
Aleix Conchillo Flaqué
090bc81ec5 tests: add some initial run_test() utilities 2025-01-20 17:41:21 -08:00
Mark Backman
e3d53d3d9a Merge pull request #1044 from pipecat-ai/mb/elevenlabs-http-fix-voice-settings
Fixed a type error when using voice_settings in ElevenLabsHttpTTSService
2025-01-20 08:11:38 -05:00
Mark Backman
262d3a19c9 Fixed a type error when using voice_settings in ElevenLabsHttpTTSService 2025-01-20 07:57:02 -05:00
allenmylath
491feb691c Update bot.py
quiet and talking frames are determined based on BotStartedSpeakingFrame and BotStoppedSpeakingFrame not ttsframe
2025-01-20 14:00:17 +05:30
Aleix Conchillo Flaqué
e4f83b237e update CHANGELOG (remove 07d-interruptible-elevenlabs-http.py) 2025-01-19 11:36:18 -08:00
Aleix Conchillo Flaqué
a169e0cde9 Merge pull request #1035 from pipecat-ai/aleix/prepare-0.0.53
update CHANGELOG for 0.0.53
2025-01-18 14:50:35 -08:00
Aleix Conchillo Flaqué
c6d643d4ec update CHANGELOG for 0.0.53 2025-01-18 14:48:48 -08:00
Aleix Conchillo Flaqué
2abbd4bb27 Merge pull request #1039 from pipecat-ai/aleix/fish-audio-websocket-service
services(fish): FishAudioTTSService to use WebsocketService
2025-01-18 14:48:20 -08:00
Aleix Conchillo Flaqué
e0011a3996 services(fish): FishAudioTTSService to use WebsocketService 2025-01-18 14:29:45 -08:00
Aleix Conchillo Flaqué
ea44c59ddd Merge pull request #1037 from Vaibhav159/fixing_unused_11labs_package
removing unused 11labs package imports
2025-01-17 22:08:04 -08:00
Vaibhav159
a9c7dbbc05 removing unused code 2025-01-18 10:58:07 +05:30
Vaibhav159
8a87e92b2b adding missing 11labs package 2025-01-18 10:48:57 +05:30
Mark Backman
982f2becc6 Merge pull request #1002 from pipecat-ai/mb/add-on-error-callback
Register the on_error handler
2025-01-17 21:58:59 -05:00
Mark Backman
e049ae470d Register the on_error handler 2025-01-17 21:49:42 -05:00
Mark Backman
e159f2dce1 Merge pull request #1024 from pipecat-ai/mb/elevenlabs-http
Add ElevenLabsHttpTTSService
2025-01-17 21:30:31 -05:00
Aleix Conchillo Flaqué
e9162ae467 Merge pull request #1004 from Fluentsai/feature/dtmf_input
Twilio serializer reading dtmf websocket messages
2025-01-17 18:14:46 -08:00
Aleix Conchillo Flaqué
bb65512ff4 Merge pull request #1034 from pipecat-ai/aleix/ulaw-resample-update
ulaw resample update
2025-01-17 17:47:18 -08:00
Mark Backman
b81323d676 Code review fixes + docstrings 2025-01-17 20:12:43 -05:00
Aleix Conchillo Flaqué
65fa77dfa5 audio: use resample_audio to resample ulaw bytes 2025-01-17 15:24:41 -08:00
Aleix Conchillo Flaqué
9ddd9ae27c Merge pull request #1011 from Vaibhav159/vl_deepgram_metrics_without_vad
adding metric generation without deepgram VAD
2025-01-17 14:47:19 -08:00
Aleix Conchillo Flaqué
12fc6e17ef Merge pull request #1033 from pipecat-ai/aleix/observers-performance
task: add TaskObserver and avoid pipeline blocking
2025-01-17 14:43:26 -08:00
Aleix Conchillo Flaqué
3e4020cdba task: add TaskObserver and avoid pipeline blocking
Observers now process frames in separate tasks. This avoids blocking the
pipeline while the observer is processing the frame.
2025-01-17 11:15:52 -08:00
Aleix Conchillo Flaqué
4f883ee31f Merge pull request #1023 from pipecat-ai/aleix/introduce-heartbeat-frames
introduce heartbeat frames
2025-01-17 10:31:07 -08:00
Mark Backman
3ff360f042 Merge pull request #1032 from pipecat-ai/mb/user-idle-fixes
Start UserIdleProcessor on speaking frame, fix bug not pushing EndFrame
2025-01-17 13:18:09 -05:00
Aleix Conchillo Flaqué
45cbad5b3e task: add HEARTBEAT_MONITOR_SECONDS 2025-01-17 10:11:28 -08:00
Aleix Conchillo Flaqué
477d0d154b frame_processor: make sure clock is initialized 2025-01-17 10:05:23 -08:00
Aleix Conchillo Flaqué
4b3c776f58 task: don't use push queue to send a heartbeat
This is because we might be waiting for the EndFrame. Currently, if we push an
EndFrame to the task, the task will block until the EndFrame traverses all the
pipeline.
2025-01-17 10:04:24 -08:00
Aleix Conchillo Flaqué
da0c4cfd99 task: increase heartbeat monitoring to 5 seconds 2025-01-17 10:04:05 -08:00
Aleix Conchillo Flaqué
f22a00570d task: start heartbeats task when push task starts 2025-01-17 10:03:13 -08:00
Mark Backman
85f4663a41 Start UserIdleProcessor on speaking frame, fix bug not pushing EndFrame 2025-01-17 12:54:17 -05:00
Aleix Conchillo Flaqué
915e3bb3c7 Merge pull request #1029 from Vaibhav159/vl_fixing_idle_frame_processor_logic
fixing IdleFrameProcessor and UserIdleProcessor init logic
2025-01-17 06:48:13 -08:00
Vaibhav159
80779c48d6 sort fix 2025-01-17 20:07:25 +05:30
Vaibhav159
c444557965 fixing IdleFrameProcessor and UserIdleProcessor init logic 2025-01-17 19:50:53 +05:30
Mark Backman
d51893f61c Refactor for aiohttp, correct use of settings 2025-01-16 23:49:53 -05:00
Mark Backman
740d2743df Add TTFB metrics 2025-01-16 23:05:53 -05:00
Mark Backman
0dd22fb879 Merge pull request #1022 from pipecat-ai/mb/fix-abstractmethod
Remove @abstractmethod from set_model and set_model in TTSService class
2025-01-16 22:59:26 -05:00
Mark Backman
225b65c3d2 Add ElevenLabsHttpTTSService 2025-01-16 22:46:32 -05:00
Aleix Conchillo Flaqué
2503f76107 examples: add 31-heartbeats.py 2025-01-16 19:31:13 -08:00
Aleix Conchillo Flaqué
ff8aa68942 introduce heartbeat frames 2025-01-16 19:31:13 -08:00
Maxim Makatchev
c5edbf4b75 Made InputDTMFFrame a DataFrame and moved up to data frames 2025-01-17 12:27:04 +09:00
Aleix Conchillo Flaqué
799777774b Merge pull request #1018 from pipecat-ai/aleix/streamline-thread-pool-executors
transports: streamline max_workers for ThreadPoolExecutors
2025-01-16 19:05:41 -08:00
Mark Backman
fdef8a97e2 Remove @abstractmethod from set_model and set_model in TTSService class 2025-01-16 21:36:51 -05:00
Mark Backman
0163247410 Merge pull request #1021 from pipecat-ai/mb/improve-30
Add a second observer to the 30-observer.py example
2025-01-16 21:19:35 -05:00
James Hush
221e044046 demo: Update translator bot example (#1005)
* docs: Update translator bot example

Updates the translator bot to do the following:

- Allow you to specify the in and out languages
- Uses TranscriptionProcessor to handle transcriptions

* Simplify the example, improve performance

---------

Co-authored-by: Mark Backman <mark@daily.co>
2025-01-17 10:08:15 +08:00
Mark Backman
532fd31fd7 Add a second observer to the 30-observer.py example 2025-01-16 19:46:18 -05:00
Mark Backman
3e178fd46f Merge pull request #1020 from pipecat-ai/mb/observer-foundational
Add foundational example 30 to show how to use an Observer
2025-01-16 19:28:26 -05:00
Mark Backman
07cb8b7a89 Extend the example to include BotStartedSpeakingFrame and BotStoppedSpeakingFrame 2025-01-16 19:24:01 -05:00
Mark Backman
e805738d4c Merge pull request #1009 from pipecat-ai/mb/tts-ignore-interim-transcripts
TTSService should only process LLMTextFrames
2025-01-16 17:09:24 -05:00
Mark Backman
119bc7e35f Update check to exclude transcription frames 2025-01-16 16:43:46 -05:00
Mark Backman
b9b02845a3 Add foundational example 30 to show how to use an Observer 2025-01-16 16:37:32 -05:00
Aleix Conchillo Flaqué
3714f12edc Merge pull request #1019 from Canonical-AI-Inc/canonical-transcripts
Add transcript to Canonical Metrics Service
2025-01-16 13:36:55 -08:00
Aleix Conchillo Flaqué
d2b8171197 transports: streamline max_workers for ThreadPoolExecutors 2025-01-16 13:34:04 -08:00
Adrian Cowham
d0b48c95bb updated the example to use stereo audio and pass in the context. also updated the service to send the transcripts if they're available 2025-01-16 13:12:38 -08:00
Aleix Conchillo Flaqué
73ed0c1ad7 Merge pull request #1017 from pipecat-ai/aleix/additional-trace-logging
additional trace logging
2025-01-16 12:38:47 -08:00
Vanessa Pyne
c211580fec Merge pull request #1016 from pipecat-ai/vp-1007-nonetype
services(gemini_multimodal_live): set content to [] if not present in messages
2025-01-16 14:14:50 -06:00
Aleix Conchillo Flaqué
359b55a85e additional trace logging 2025-01-16 11:19:42 -08:00
kompfner
8b602a3f62 Merge pull request #1010 from pipecat-ai/ios-simplechatbot-assorted-improvements
iOS SimpleChatbot assorted improvements
2025-01-16 13:59:45 -05:00
kompfner
485c231f69 Merge pull request #1012 from pipecat-ai/simplechatbot-readme-local-pipecat
Add to the SimpleChatbot server README a step for pointing to the loc…
2025-01-16 13:46:19 -05:00
vipyne
8ba3b150eb services(gemini_multimodal_live): set content to [] if not present in messages
... which it will be if the message is a tool call
2025-01-16 11:59:02 -06:00
Paul Kompfner
b5f72b4378 Add to the SimpleChatbot server README a step for pointing to the local version of pipecat 2025-01-16 11:59:44 -05:00
Vaibhav159
85e7d62f94 fixing log text 2025-01-16 21:36:51 +05:30
Vaibhav159
923d33eeff fixing ruff 2025-01-16 21:32:48 +05:30
Vaibhav159
7ee6e7193d adding metric generation without deepgram VAD 2025-01-16 21:23:56 +05:30
Paul Kompfner
156fffe6fc In iOS SimpleChatbot demo, add clarifying note to Audio Settings section header explaining that "(No selection = system default)".
Ideally we could add a row showing that the system default is selected, but this is OK as a short-term fix. Also, the presence of that row might suggest that "system default" is selectable, but it's not: this is currently a limitation in the Pipecat Client.
2025-01-16 10:32:55 -05:00
Paul Kompfner
c9834e2712 In iOS SimpleChatbot demo, remove unused LLMHelperDelegate protocol conformance 2025-01-16 10:31:17 -05:00
Paul Kompfner
1e7e307f69 In iOS SimpleChatbot demo, call release() when disconnecting the voice client, since we're not using it after disconnecting 2025-01-16 10:30:06 -05:00
Mark Backman
67e47a388d TTSService should only process LLMTextFrames 2025-01-16 10:03:24 -05:00
Maxim Makatchev
dcf317f2fa Twilio serializer reading dtmf websocket messages and generating InputDTMFFrame containing the corresponding value of KeypadEntry 2025-01-16 17:43:12 +09:00
Aleix Conchillo Flaqué
b8ffd7b16b Merge pull request #996 from pipecat-ai/aleix/introduce-observers
introduce pipeline frame observers
2025-01-15 18:05:33 -08:00
Aleix Conchillo Flaqué
08f1dda94e observers: add a timestamp to on_push_frame() 2025-01-15 17:45:00 -08:00
Aleix Conchillo Flaqué
45039e7cde update CHANGELOG.md 2025-01-15 17:45:00 -08:00
Aleix Conchillo Flaqué
e50c76d075 examples(simple-chatbot): use RTVIObserver for server-client messages 2025-01-15 17:45:00 -08:00
Aleix Conchillo Flaqué
dd9f9179cc rtvi(RTVIObserver): use observers for RTVI server->client messages 2025-01-15 17:45:00 -08:00
Aleix Conchillo Flaqué
c8da531402 pipeline(task): add support for pipeline frame observers 2025-01-15 17:43:59 -08:00
Aleix Conchillo Flaqué
25bcaf5c7c observers: introduce pipeline observers 2025-01-15 17:43:59 -08:00
Aleix Conchillo Flaqué
2d0f3341c3 frames: add LLMTextFrame and TTSTextFrame
This is to distinguish what type of service has generated the TextFrames.
2025-01-15 17:43:59 -08:00
Aleix Conchillo Flaqué
7626d7b04b Merge pull request #999 from pipecat-ai/aleix/add-pre-commit-hooks
add pre-commit hooks
2025-01-15 17:39:34 -08:00
Aleix Conchillo Flaqué
f78520f7d0 add pre-commit hooks
Fixes #945
2025-01-15 13:44:21 -08:00
Aleix Conchillo Flaqué
bb4766455d Merge pull request #997 from pipecat-ai/aleix/update-dependencies-01-15-25
update dependencies (go back to numpy1)
2025-01-15 13:35:46 -08:00
Aleix Conchillo Flaqué
9dacbbbbf4 fix ruff formatting 2025-01-15 13:02:13 -08:00
Aleix Conchillo Flaqué
4de192fbb0 update dependencies (go back to numpy1)
Fixes #911, #913
2025-01-15 12:04:28 -08:00
kompfner
80b6c28431 Merge pull request #992 from pipecat-ai/live-updates-to-selected-and-available-mics
In the iOS SimpleChatbot demo, wire up live updates to the selected m…
2025-01-15 15:00:14 -05:00
Mark Backman
f471744bca Merge pull request #995 from pipecat-ai/vp-riva-bump
deps(riva): bump to 2.18.0
2025-01-15 14:35:39 -05:00
Mark Backman
d5df4b064b Merge pull request #987 from pipecat-ai/mb/deepseek-typo
Fix error log in DeepSeekLLMService and CerebrasLLMService
2025-01-15 14:31:34 -05:00
Mark Backman
06a0e29920 Merge pull request #991 from pipecat-ai/mb/update-web-simple-chatbot
Update simple-chatbot example to use the latest client SDKs
2025-01-15 13:36:03 -05:00
Aleix Conchillo Flaqué
64eb8e7262 Merge pull request #994 from Vaibhav159/vl_deepgram_with_vad
finalize on DeepgramSTTService on VAD
2025-01-15 10:28:11 -08:00
Filipi da Silva Fuchter
d8386c12dc Merge pull request #990 from pipecat-ai/bumping_ios_example
Using PipecatClient version 0.3.2
2025-01-15 14:29:01 -03:00
vipyne
50e798bcd9 deps(riva): bump to 2.18.0 2025-01-15 10:24:57 -06:00
Vaibhav159
d1ac7751da finalize on DeepgramSTTService 2025-01-15 20:43:23 +05:30
Paul Kompfner
110ce27c91 In the iOS SimpleChatbot demo, wire up live updates to the selected mic and available mics list. This is beneficial for a few reasons:
- Live updates are nice! We can now more easily see what's going on when we connect or disconnect a mic.
- Resolves an issue where the initial selected mic was not shown.
- Let us see when the Pipecat client automatically switches to a new mic, like when one is connected.
2025-01-15 09:56:27 -05:00
Mark Backman
8b657158ca Update React simple-chatbot client to use latest client SDKs 2025-01-15 09:50:43 -05:00
Mark Backman
cce14fca97 Update JS simple-chatbot client to use latest client SDKs 2025-01-15 09:47:20 -05:00
Filipi Fuchter
7c051516d8 Using PipecatClient version 0.3.2 2025-01-15 09:57:57 -03:00
Mark Backman
5f402ad741 Merge pull request #988 from pipecat-ai/mb/readme-openrouter
Update README.md
2025-01-14 18:38:35 -05:00
Mark Backman
a80b186cea Update README.md
Add OpenRouter to the README
2025-01-14 18:08:14 -05:00
Mark Backman
c65aaf3b2e Merge pull request #967 from sahilsuman933/openrouter-integration
feat(services): Add OpenRouter LLM Service Integration
2025-01-14 18:06:13 -05:00
Mark Backman
e815d7776f Fix error log in DeepSeekLLMService and CerebrasLLMService 2025-01-14 18:03:29 -05:00
sahil suman
11fc08ef24 fix changelog
Signed-off-by: sahil suman <sahilsuman933@gmail.com>
2025-01-15 02:57:09 +05:30
sahil suman
6f3b0fdf73 fix changelog
Signed-off-by: sahil suman <sahilsuman933@gmail.com>
2025-01-15 02:56:16 +05:30
sahil suman
885bc32827 added changes in changelog.
Signed-off-by: sahil suman <sahilsuman933@gmail.com>
2025-01-15 02:53:04 +05:30
sahil suman
7339cc7197 Merge remote-tracking branch 'origin/main' into openrouter-integration 2025-01-15 02:52:19 +05:30
sahil suman
62e9e6bc5a changed the file name.
Signed-off-by: sahil suman <sahilsuman933@gmail.com>
2025-01-15 02:21:58 +05:30
Sahil Suman
329da50338 Update src/pipecat/services/openrouter.py
Co-authored-by: Mark Backman <m.backman@gmail.com>
2025-01-15 02:20:22 +05:30
sahil suman
4d307d26d8 made the required changes.
Signed-off-by: sahil suman <sahilsuman933@gmail.com>
2025-01-15 02:19:05 +05:30
Mark Backman
a74b9354ec Merge pull request #962 from pipecat-ai/mb/improve-tts-reconnection-logic
Improve websocket based TTS service reconnection logic
2025-01-14 14:48:00 -05:00
sahil suman
11381a536f added example for function calling and made the required changes.
Signed-off-by: sahil suman <sahilsuman933@gmail.com>
2025-01-15 01:00:33 +05:30
Mark Backman
b53bc8a879 _calculate_wait_times as private, add and use WebsocketServiceException 2025-01-14 13:20:13 -05:00
Mark Backman
e3d8910814 Update CHANGELOG 2025-01-14 13:12:40 -05:00
Mark Backman
e60a59434f Refactor LMNTTTSService to make a websocket connection directly, then use the WebsocketService base class 2025-01-14 13:09:58 -05:00
Mark Backman
5e5de618f3 Update PlayHTTTSService to use the WebsocketService base class 2025-01-14 13:09:58 -05:00
Mark Backman
8af92f7923 Update ElevenLabsTTSService to use the WebsocketService base class 2025-01-14 13:09:58 -05:00
Mark Backman
f39e17857e Add a WebsocketService base class to retry, ensure that retries reset after a successful connection, update Cartesia to use the new WebsocketService 2025-01-14 13:09:58 -05:00
Aleix Conchillo Flaqué
5b632de04a Merge pull request #982 from pipecat-ai/aleix/pipelinetask-cleanup-sink
pipeline(task): cleanup Sink processor
2025-01-14 09:14:03 -08:00
Mark Backman
6bcc196489 Merge pull request #969 from pipecat-ai/mb/deepseek
Add support for DeepSeek LLM
2025-01-14 09:40:06 -05:00
Mark Backman
66375e9dff Update dot-env.template API keys 2025-01-14 09:34:34 -05:00
Mark Backman
bc839492b6 Add support for DeepSeek LLM 2025-01-14 09:34:33 -05:00
Filipi da Silva Fuchter
4854645637 Merge pull request #960 from pipecat-ai/example_gemini_with_goolge_search
Example with Gemini using google search to retrieve news.
2025-01-14 10:07:15 -03:00
Mark Backman
98e80b7d4a Merge pull request #970 from pipecat-ai/mb/user-controlled-run-llm
Add an override_run_llm option to optionally defer function call completion
2025-01-13 18:48:00 -05:00
Mark Backman
8c0ecb89de Refactor for new on_context_updated callback and new frame properties 2025-01-13 17:20:41 -05:00
Aleix Conchillo Flaqué
4c8fcb2cfc pipeline(task): cleanup Sink processor
Fixes #953
2025-01-13 13:29:44 -08:00
Aleix Conchillo Flaqué
92313d6ce7 Merge pull request #972 from pipecat-ai/aleix/simple-chatbot-android-workflow-update
github: only run android simple-chatbot worflow if android example modified
2025-01-13 13:26:12 -08:00
Mark Backman
1ca6ecc46e Update CHANGELOG 2025-01-13 09:49:09 -05:00
Mark Backman
f1947d7d38 Update Anthropic and Gemini to allow overriding run_llm 2025-01-13 09:48:43 -05:00
Mark Backman
0852570212 Update Grok for function call override 2025-01-13 09:48:43 -05:00
Mark Backman
874b8bb136 Allow for an override of running a completion after a function call completes, OpenAI 2025-01-13 09:48:43 -05:00
Mark Backman
da1878537b Merge pull request #974 from pipecat-ai/mb/26d-example
Align 26d example with foundation norms
2025-01-12 19:44:31 -05:00
Mark Backman
f406d93b0f Align 26d example with foundation norms 2025-01-12 19:19:16 -05:00
Aleix Conchillo Flaqué
3cd2b90177 Merge pull request #971 from pipecat-ai/aleix/update-copyright-keep-original-year
update copyright keeping original year (2024)
2025-01-12 11:37:15 -08:00
Aleix Conchillo Flaqué
c4f0c7bcfd github: only run android simple-chatbot worflow if android example modified 2025-01-12 11:35:34 -08:00
Aleix Conchillo Flaqué
95e69597f3 update copyright keeping original year (2024) 2025-01-12 11:34:00 -08:00
Aleix Conchillo Flaqué
710baa5e17 Merge pull request #973 from pipecat-ai/aleix/simple-chatbot-clients
examples/simple-chatbot: move clients to client directory
2025-01-12 11:28:21 -08:00
Mert Sefa AKGUN
14e5419913 fix(gemini): prevent non-audio modality processing
Add an early return in the _handle_transcribe_model_audio method to
prevent unnecessary processing when the modalities setting is not set
to audio. This change ensures that audio transcription only occurs
when appropriate.
2025-01-12 22:17:10 +03:00
Mark Backman
8c953bac41 Merge pull request #966 from imsakg/main
fix(services): handle TranscriptionFrame separately in TTSService
2025-01-12 11:33:38 -05:00
Mark Backman
4c0861ce39 Some addition links and README changes 2025-01-12 09:27:23 -05:00
Mark Backman
12b1e1db9d Merge pull request #965 from pipecat-ai/mb/aws-add-session-token
Add optional aws_session_token for PollyTTSService
2025-01-12 09:13:03 -05:00
Mark Backman
53bfdfd83f Merge pull request #963 from pipecat-ai/mb/cleanup-examples
Update examples to align with latest best practices
2025-01-12 09:12:34 -05:00
Mark Backman
2a5593afea Merge pull request #968 from pipecat-ai/mb/readme-websocket
Update README.md
2025-01-12 09:12:19 -05:00
Aleix Conchillo Flaqué
a04a920e54 examples/simple-chatbot: move clients to client directory 2025-01-11 19:16:05 -08:00
Aleix Conchillo Flaqué
2ce6d92455 Merge pull request #959 from KevGTL/fix-livekit-transport
fix: push input audio frame only via push_audio_frame()
2025-01-11 19:03:35 -08:00
Mark Backman
1ecd5da219 Update README.md
Add websocket docs links to README.
2025-01-11 08:37:17 -05:00
sahil suman
e04da334d7 add support for openrouter.
Signed-off-by: sahil suman <sahilsuman933@gmail.com>
2025-01-11 17:50:58 +05:30
Mert Sefa AKGUN
7ec351813c style(ai_services): fix import order with ruff 2025-01-11 13:04:26 +03:00
Mert Sefa AKGUN
df6c2fc403 fix(services): handle TranscriptionFrame separately in TTSService
Exclude TranscriptionFrame from text frame processing in TTSService by updating the type check condition. This resolves unintended processing behavior when handling different frame types.
2025-01-11 13:00:38 +03:00
Mark Backman
71e107725c Add optional aws_session_token for PollyTTSService 2025-01-10 19:33:47 -05:00
Mark Backman
4d0c11fcab Update examples to align with latest best practices 2025-01-10 15:07:06 -05:00
Mark Backman
a8ae79831e Merge pull request #921 from pipecat-ai/mb/playht-http
PlayHTHttpTTSService fixes
2025-01-10 13:26:45 -05:00
Mark Backman
86516d2415 PlayHTHttpTTSService fixes 2025-01-10 13:21:27 -05:00
Vanessa Pyne
5cd9dab14b Merge pull request #949 from imsakg/main
fix(examples): correct TTS service import and setup
2025-01-10 10:58:50 -06:00
Kwindla Hultman Kramer
a3e2e06975 Merge pull request #961 from pipecat-ai/khk/tiny-chatbot-readme-fix
fixed 404 in SimpleChatbot iOS example README
2025-01-10 08:45:05 -08:00
Kwindla Hultman Kramer
e7107b99c5 fixed 404 in SimpleChatbot iOS example README 2025-01-10 08:37:13 -08:00
Filipi Fuchter
aa1b8879ee Fixing ruff format 2025-01-10 13:21:51 -03:00
Mark Backman
6802459165 Merge pull request #956 from pipecat-ai/mb/tavus
Update the Tavus example and comment about using the PERSONA_ID
2025-01-10 11:18:05 -05:00
Filipi Fuchter
6719d1fddc Example with Gemini using google search to retrieve news. 2025-01-10 13:13:59 -03:00
kompfner
a798bf18f2 Merge pull request #955 from pipecat-ai/ios-simple-chatbot-mainactor-fixes
iOS SimpleChatbot @MainActor fixes
2025-01-10 09:37:02 -05:00
Kevin Oury
f9d0cca60f fix: push input audio frame only via push_audio_frame() 2025-01-10 15:02:38 +01:00
Mark Backman
cb22de0d13 Update the Tavus example and comment about using the PERSONA_ID 2025-01-10 08:01:00 -05:00
marcus-daily
7d161cc53b Setting target SDK to 35 2025-01-10 09:50:37 +00:00
marcus-daily
255abf46ef Updating Gradle and AGP 2025-01-10 09:50:37 +00:00
marcus-daily
27579bcb70 Fixing imports 2025-01-10 09:50:37 +00:00
marcus-daily
1295b64879 Updating library dependencies 2025-01-10 09:50:37 +00:00
marcus-daily
ca57670f65 Removing unnecessary drawables 2025-01-10 09:50:37 +00:00
marcus-daily
06d0a231b9 Android demo app for simple-chatbot example 2025-01-10 09:50:37 +00:00
Mert Sefa AKGUN
67af4e619b style(examples): fix ruff formatting in Gemini text example
Refactor `CartesiaTTSService` instantiation to comply with line
length requirements from the ruff linter.
2025-01-10 12:32:53 +03:00
Mert Sefa AKGUN
21c274944e Update examples/foundational/26d-gemini-multimodal-live-text.py
Co-authored-by: Vanessa Pyne <vipyne@gmail.com>
2025-01-10 12:28:13 +03:00
Paul Kompfner
3239249feb In the iOS SimpleChatbot, fix @MainActor-related warnings (which would be errors in Swift 6). The delegate methods aren't contractually guaranteed to run on the main thread, so we can't mark them as @MainActor. 2025-01-09 17:35:44 -05:00
Paul Kompfner
216979c377 Bump iOS SimpleChatbot's pipecat-client-ios-daily dependency to version 0.3.1 2025-01-09 16:22:26 -05:00
Filipi da Silva Fuchter
b9db53d3cd Merge pull request #952 from pipecat-ai/fixing_gemini_function_calling
Fixing GeminiMultimodalLiveLLMService function calling to work with pipecat-flows
2025-01-09 17:50:25 -03:00
Filipi Fuchter
58bfcc8370 Fixing GeminiMultimodalLiveLLMService function calling when using with pipecat-flows. 2025-01-09 12:22:37 -03:00
Mert Sefa AKGUN
6664c492ac feat(gemini): enable audio transcription in live text example
Add options to transcribe both user and model audio during the GeminiMultimodalLiveLLMService setup in the 26d-gemini-multimodal-live-text.py example.
2025-01-09 15:38:33 +03:00
Mert Sefa AKGUN
7634058f97 fix(examples): correct TTS service import and setup
- Update import to use CartesiaTTSService instead of CartesiaMultiLingualTTSService.
- Adjust GeminiMultimodalLiveLLMService setup to use set_model_modalities with TEXT modality.
2025-01-09 02:19:08 +03:00
Mark Backman
39c6446bdc Merge pull request #947 from pipecat-ai/mb/add-rime-set-voices
Add setters for model and voice to RimeHttpTTSService
2025-01-08 14:25:24 -05:00
Filipi da Silva Fuchter
2df7dfcc91 Merge pull request #943 from pipecat-ai/simple_chat_bot_ios
SimpleChatbot iOS app.
2025-01-08 16:17:39 -03:00
Mark Backman
c23c9e046c Add setters for model and voice to RimeHttpTTSService 2025-01-08 14:17:32 -05:00
Mark Backman
9dae753e8c Merge pull request #926 from imsakg/main
feat(gemini): add text handling to GeminiMultimodalLive
2025-01-08 13:42:17 -05:00
Mert Sefa AKGUN
40e9ee6d63 fix(examples): correct import order in Gemini example
- Move `CartesiaMultiLingualTTSService` import to maintain proper order.
- Reorganize `enum` import to adhere to styling standards.
2025-01-08 21:14:29 +03:00
Mert Sefa AKGUN
a342fe732e docs: update CHANGELOG with Gemini modalities and examples 2025-01-08 19:34:42 +03:00
Mert Sefa AKGUN
a729834482 refactor(gemini): reposition WebSocket connection code
Move WebSocket connection setup earlier in the function for better
organization and to prepare for subsequent configuration steps.
2025-01-08 19:29:36 +03:00
Mert Sefa AKGUN
94a6f1086e feat(gemini): change default modality to AUDIO
Modify the default modality in the `InputParams` class from TEXT to AUDIO
to better align with the intended use case for GeminiMultimodalLive
service.
2025-01-08 19:29:36 +03:00
Mert Sefa AKGUN
b42d3a8257 feat(gemini): add modality configuration for GeminiMultimodalLive
- Introduce `GeminiMultimodalModalities` enum for modality options.
- Add modality field to `InputParams`, defaulting to text.
- Simplify modality setup with `set_model_modalities` method.
- Refactor WebSocket configuration to support dynamic response modalities.
2025-01-08 19:29:36 +03:00
Mert Sefa AKGUN
12ae980abe feat(gemini): handle full text response in GeminiMultimodalLive
- Add a buffer to store bot text responses.
- Push a `LLMFullResponseStartFrame` when text begins.
- Clear the text buffer and send `LLMFullResponseEndFrame` after processing.
2025-01-08 19:29:36 +03:00
Mert Sefa AKGUN
cdb909958c feat(examples): add Gemini multimodal live text example
Introduce a new example `26d-gemini-multimodal-live-text.py` to
demonstrate the use of GeminiMultimodalLiveLLMService with text-only
responses. This example sets up a pipeline for audio input via DailyTransport,
processing with Gemini, and output via Cartesia TTS.
2025-01-08 19:29:35 +03:00
Mert Sefa AKGUN
c72c3025f6 feat(gemini): add configuration methods for response modalities
- Introduce `set_model_only_audio` and `set_model_only_text` methods
  to toggle between audio-only and text-only response modes in
  `GeminiMultimodalLiveLLMService`.
- Refactor configuration setup to a class attribute for improved
  reusability and maintenance.
- Remove redundant configuration instantiation in the WebSocket
  connection setup process.
2025-01-08 19:29:35 +03:00
Mert Sefa AKGUN
5cbd719780 feat(gemini): add text handling to GeminiMultimodalLive
- Introduce text attribute in Part class for handling string data.
- Incorporate text processing in GeminiMultimodalLiveLLMService to push TextFrame if text is present.
2025-01-08 19:29:35 +03:00
Filipi Fuchter
23d6290672 Removing not used class. 2025-01-08 12:05:04 -03:00
Filipi Fuchter
d4e7e11981 SimpleChatbot iOS app. 2025-01-08 12:00:11 -03:00
Mark Backman
8057fe3fcf Merge pull request #742 from Vaibhav159/vl_feature_websocket_fastapi_timeout
adding session_timeout param
2025-01-08 09:05:41 -05:00
Vaibhav159
3b446234a7 fix hyperlink 2025-01-08 10:54:27 +05:30
Vaibhav159
768487ffb3 final changelog 2025-01-08 10:53:32 +05:30
Vaibhav159
2da5620d10 adding changelog 2025-01-08 10:50:09 +05:30
Vaibhav159
af90d65b3b adding session timeout example in websocket-server example 2025-01-08 10:43:10 +05:30
Vaibhav159
c8569a7b67 Merge remote-tracking branch 'upstream/main' into vl_feature_websocket_fastapi_timeout 2025-01-08 10:21:36 +05:30
Vaibhav159
0ecd98c873 Merge branch 'main' into vl_feature_websocket_fastapi_timeout 2025-01-08 10:20:55 +05:30
Mark Backman
6f863ba2c6 Merge pull request #938 from jcbjoe/jg/optional-authentication-polly
Changed Polly authentication params to be optional
2025-01-07 15:37:23 -05:00
Mark Backman
602ca5ebe6 Merge pull request #939 from Vaibhav159/vl_adding_daily_room_properties
adding more daily room params
2025-01-07 14:33:59 -05:00
Vaibhav159
787ade41f3 adding missing doc string 2025-01-08 00:58:01 +05:30
Joe Garlick
bb767831d5 Added: Changelog entry 2025-01-07 19:05:02 +00:00
Mark Backman
bc25a771dc Merge pull request #935 from pipecat-ai/hush/modalUpdate
docs: update dependencies for modal demo
2025-01-07 13:57:46 -05:00
Vaibhav159
f37626f81d adding more daily room params 2025-01-07 21:38:05 +05:30
Mark Backman
9d54578e65 Merge pull request #934 from pipecat-ai/mb/bump-open-ai-version
Bump openai version to 1.59.0 for realtime and model updates
2025-01-07 08:29:45 -05:00
Joe Garlick
79afe7ec2a Changed: Polly authentication information to be optional 2025-01-07 11:43:57 +00:00
James Hush
2c1fd3c3cc docs: update dependencies for modal demo 2025-01-07 15:45:55 +08:00
Mark Backman
b0dd8e03a6 Bump openai version to 1.59.0 for realtime and model updates 2025-01-06 17:05:22 -05:00
Mark Backman
ee20e48ef8 Merge pull request #931 from pipecat-ai/mb/fix-openai-realtime-
Fix truncation timing of OpenAIRealtimeBetaLLMService
2025-01-06 16:25:09 -05:00
Mark Backman
12b5c5a646 Fix truncation timing of OpenAIRealtimeBetaLLMService 2025-01-06 15:37:58 -05:00
Mark Backman
7a021cc82d Merge pull request #929 from pipecat-ai/mb/add-google-journey-support
Added support for Google Journey TTS voices
2025-01-06 15:13:00 -05:00
Mark Backman
3e1ec4a8ee Added support for Google Journey TTS voices 2025-01-06 14:54:34 -05:00
Mark Backman
a1377b7f1a Merge pull request #924 from xtreme-sameer-vohra/patch-1
Update frames.py
2025-01-06 14:13:10 -05:00
Mark Backman
d6335886e2 Merge pull request #848 from Vaibhav159/vl_add_audio_and_chat_livekit_example
adding example for livekit audio and chat version
2025-01-06 13:27:38 -05:00
Vaibhav159
b3b7a5f023 adding 2025 license 2025-01-06 22:10:46 +05:30
Vaibhav159
5138017b57 ruff changes 2025-01-06 22:07:59 +05:30
Vaibhav159
87670067d7 adding changelog 2025-01-06 22:03:11 +05:30
Vaibhav159
656cd2859e Merge branch 'main' into vl_add_audio_and_chat_livekit_example 2025-01-06 21:57:43 +05:30
Mark Backman
15b2cc210c Merge pull request #927 from pipecat-ai/mb/update-copyright
Update copyright to 2025
2025-01-06 10:33:04 -05:00
Mark Backman
4667624b60 Update copyright to 2025 2025-01-06 10:19:37 -05:00
Sameer Vohra
d07ba80572 Update frames.py
fix minor typo in docs
2025-01-05 22:57:54 -05:00
Vaibhav159
62fc95300b adding livekit audio and chat version 2024-12-13 01:09:47 +05:30
Vaibhav159
6e8e7fa19a adding session_timeout in fastapi 2024-11-21 14:56:42 +05:30
Vaibhav159
7dfa886669 moving logic to WebsocketServerInputTransport 2024-11-21 14:45:24 +05:30
Vaibhav159
da254c5143 correcting _monitor_websocket 2024-11-21 12:36:51 +05:30
Vaibhav159
e11f128110 adding on_session_timeout 2024-11-21 12:34:32 +05:30
Vaibhav-Lodha
3aa89fb13a adding session_timeout param 2024-11-21 12:20:51 +05:30
388 changed files with 8843 additions and 1806 deletions

48
.github/workflows/android.yaml vendored Normal file
View File

@@ -0,0 +1,48 @@
name: android
on:
push:
branches:
- main
paths:
- "examples/simple-chatbot/client/android/**"
pull_request:
branches:
- "**"
paths:
- "examples/simple-chatbot/client/android/**"
workflow_dispatch:
inputs:
sdk_git_ref:
type: string
description: "Which git ref of the app to build"
concurrency:
group: build-android-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
sdk:
name: "Simple chatbot demo"
runs-on: ubuntu-latest
steps:
- name: Checkout repo
uses: actions/checkout@v4
with:
ref: ${{ github.event.inputs.sdk_git_ref || github.ref }}
- name: "Install Java"
uses: actions/setup-java@v4
with:
distribution: 'temurin'
java-version: '17'
- name: Build demo app
working-directory: examples/simple-chatbot/client/android
run: ./gradlew :simple-chatbot-client:assembleDebug
- name: Upload demo APK
uses: actions/upload-artifact@v4
with:
name: Simple Chatbot Android Client
path: examples/simple-chatbot/client/android/simple-chatbot-client/build/outputs/apk/debug/simple-chatbot-client-debug.apk

View File

@@ -1,4 +1,4 @@
name: test
name: tests
on:
workflow_dispatch:
@@ -49,4 +49,4 @@ jobs:
- name: Test with pytest
run: |
source .venv/bin/activate
pytest --ignore-glob="*to_be_updated*" --ignore-glob=*pipeline_source* src tests
pytest

7
.pre-commit-config.yaml Normal file
View File

@@ -0,0 +1,7 @@
repos:
- repo: local
hooks:
- id: ruff-format-hook
name: Check ruff formatting
entry: sh scripts/pre-commit.sh
language: system

View File

@@ -5,6 +5,159 @@ All notable changes to **Pipecat** will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Added
- It is now possible to specify the period of the `PipelineTask` heartbeat
frames with `heartbeats_period_secs`.
### Changed
- Modified `TranscriptProcessor` to use TTS text frames for more accurate assistant
transcripts. Assistant messages are now aggregated based on bot speaking boundaries
rather than LLM context, providing better handling of interruptions and partial
utterances.
- Updated foundational examples `28a-transcription-processor-openai.py`,
`28b-transcript-processor-anthropic.py`, and
`28c-transcription-processor-gemini.py` to use the updated
`TranscriptProcessor`.
### Fixed
- Fixed a type error when using `voice_settings` in `ElevenLabsHttpTTSService`.
- Fixed an issue where `OpenAIRealtimeBetaLLMService` function calling resulted
in an error.
### Performance
- Replaced audio resampling library `resampy` with `soxr`. Resampling a 2:21s
audio file from 24KHz to 16KHz took 1.41s with `resampy` and 0.031s with
`soxr` with similar audio quality.
### Other
- Added initial unit test infrastructure.
## [0.0.53] - 2025-01-18
### Added
- Added `ElevenLabsHttpTTSService` which uses EleveLabs' HTTP API instead of the
websocket one.
- Introduced pipeline frame observers. Observers can view all the frames that go
through the pipeline without the need to inject processors in the
pipeline. This can be useful, for example, to implement frame loggers or
debuggers among other things. The example
`examples/foundational/30-observer.py` shows how to add an observer to a
pipeline for debugging.
- Introduced heartbeat frames. The pipeline task can now push periodic
heartbeats down the pipeline when `enable_heartbeats=True`. Heartbeats are
system frames that are supposed to make it all the way to the end of the
pipeline. When a heartbeat frame is received the traversing time (i.e. the
time it took to go through the whole pipeline) will be displayed (with TRACE
logging) otherwise a warning will be shown. The example
`examples/foundational/31-heartbeats.py` shows how to enable heartbeats and
forces warnings to be displayed.
- Added `LLMTextFrame` and `TTSTextFrame` which should be pushed by LLM and TTS
services respectively instead of `TextFrame`s.
- Added `OpenRouter` for OpenRouter integration with an OpenAI-compatible
interface. Added foundational example `14m-function-calling-openrouter.py`.
- Added a new `WebsocketService` based class for TTS services, containing
base functions and retry logic.
- Added `DeepSeekLLMService` for DeepSeek integration with an OpenAI-compatible
interface. Added foundational example `14l-function-calling-deepseek.py`.
- Added `FunctionCallResultProperties` dataclass to provide a structured way to
control function call behavior, including:
- `run_llm`: Controls whether to trigger LLM completion
- `on_context_updated`: Optional callback triggered after context update
- Added a new foundational example `07e-interruptible-playht-http.py` for easy
testing of `PlayHTHttpTTSService`.
- Added support for Google TTS Journey voices in `GoogleTTSService`.
- Added `29-livekit-audio-chat.py`, as a new foundational examples for
`LiveKitTransportLayer`.
- Added `enable_prejoin_ui`, `max_participants` and `start_video_off` params
to `DailyRoomProperties`.
- Added `session_timeout` to `FastAPIWebsocketTransport` and
`WebsocketServerTransport` for configuring session timeouts (in
seconds). Triggers `on_session_timeout` for custom timeout handling.
See [examples/websocket-server/bot.py](https://github.com/pipecat-ai/pipecat/blob/main/examples/websocket-server/bot.py).
- Added the new modalities option and helper function to set Gemini output
modalities.
- Added `examples/foundational/26d-gemini-multimodal-live-text.py` which is
using Gemini as TEXT modality and using another TTS provider for TTS process.
### Changed
- Modified `UserIdleProcessor` to start monitoring only after first
conversation activity (`UserStartedSpeakingFrame` or
`BotStartedSpeakingFrame`) instead of immediately.
- Modified `OpenAIAssistantContextAggregator` to support controlled completions
and to emit context update callbacks via `FunctionCallResultProperties`.
- Added `aws_session_token` to the `PollyTTSService`.
- Changed the default model for `PlayHTHttpTTSService` to `Play3.0-mini-http`.
- `api_key`, `aws_access_key_id` and `region` are no longer required parameters
for the PollyTTSService (AWSTTSService)
- Added `session_timeout` example in `examples/websocket-server/bot.py` to
handle session timeout event.
- Changed `InputParams` in
`src/pipecat/services/gemini_multimodal_live/gemini.py` to support different
modalities.
- Changed `DeepgramSTTService` to send `finalize` event whenever VAD detects
`UserStoppedSpeakingFrame`. This helps in faster transcriptions and clearing
the `Deepgram` audio buffer.
### Fixed
- Fixed an issue where `DeepgramSTTService` was not generating metrics using
pipeline's VAD.
- Fixed `UserIdleProcessor` not properly propagating `EndFrame`s through the
pipeline.
- Fixed an issue where websocket based TTS services could incorrectly terminate
their connection due to a retry counter not resetting.
- Fixed a `PipelineTask` issue that would cause a dangling task after stopping
the pipeline with an `EndFrame`.
- Fixed an import issue for `PlayHTHttpTTSService`.
- Fixed an issue where languages couldn't be used with the `PlayHTHttpTTSService`.
- Fixed an issue where `OpenAIRealtimeBetaLLMService` audio chunks were hitting
an error when truncating audio content.
- Fixed an issue where setting the voice and model for `RimeHttpTTSService`
wasn't working.
- Fixed an issue where `IdleFrameProcessor` and `UserIdleProcessor` were getting
initialized before the start of the pipeline.
## [0.0.52] - 2024-12-24
### Added

View File

@@ -1,6 +1,6 @@
BSD 2-Clause License
Copyright (c) 2024, Daily
Copyright (c) 20242025, Daily
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

View File

@@ -2,7 +2,7 @@
 <img alt="pipecat" width="300px" height="auto" src="https://raw.githubusercontent.com/pipecat-ai/pipecat/main/pipecat.png">
</div></h1>
[![PyPI](https://img.shields.io/pypi/v/pipecat-ai)](https://pypi.org/project/pipecat-ai) [![Docs](https://img.shields.io/badge/Documentation-blue)](https://docs.pipecat.ai) [![Discord](https://img.shields.io/discord/1239284677165056021)](https://discord.gg/pipecat) <a href="https://app.commanddash.io/agent/github_pipecat-ai_pipecat"><img src="https://img.shields.io/badge/AI-Code%20Agent-EB9FDA"></a>
[![PyPI](https://img.shields.io/pypi/v/pipecat-ai)](https://pypi.org/project/pipecat-ai) ![Tests](https://github.com/pipecat-ai/pipecat/actions/workflows/tests.yaml/badge.svg) [![Docs](https://img.shields.io/badge/Documentation-blue)](https://docs.pipecat.ai) [![Discord](https://img.shields.io/discord/1239284677165056021)](https://discord.gg/pipecat) <a href="https://app.commanddash.io/agent/github_pipecat-ai_pipecat"><img src="https://img.shields.io/badge/AI-Code%20Agent-EB9FDA"></a>
Pipecat is an open source Python framework for building voice and multimodal conversational agents. It handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions, letting you focus on creating engaging experiences.
@@ -55,17 +55,17 @@ pip install "pipecat-ai[option,...]"
Available options include:
| Category | Services | Install Command Example |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- |
| Speech-to-Text | [AssemblyAI](https://docs.pipecat.ai/server/services/stt/assemblyai), [Azure](https://docs.pipecat.ai/server/services/stt/azure), [Deepgram](https://docs.pipecat.ai/server/services/stt/deepgram), [Gladia](https://docs.pipecat.ai/server/services/stt/gladia), [Whisper](https://docs.pipecat.ai/server/services/stt/whisper) | `pip install "pipecat-ai[deepgram]"` |
| LLMs | [Anthropic](https://docs.pipecat.ai/server/services/llm/anthropic), [Azure](https://docs.pipecat.ai/server/services/llm/azure), [Cerebras](https://docs.pipecat.ai/server/services/llm/cerebras), [Fireworks AI](https://docs.pipecat.ai/server/services/llm/fireworks), [Gemini](https://docs.pipecat.ai/server/services/llm/gemini), [Grok](https://docs.pipecat.ai/server/services/llm/grok), [Groq](https://docs.pipecat.ai/server/services/llm/groq), [NVIDIA NIM](https://docs.pipecat.ai/server/services/llm/nim), [Ollama](https://docs.pipecat.ai/server/services/llm/ollama), [OpenAI](https://docs.pipecat.ai/server/services/llm/openai), [Together AI](https://docs.pipecat.ai/server/services/llm/together) | `pip install "pipecat-ai[openai]"` |
| Text-to-Speech | [AWS](https://docs.pipecat.ai/server/services/tts/aws), [Azure](https://docs.pipecat.ai/server/services/tts/azure), [Cartesia](https://docs.pipecat.ai/server/services/tts/cartesia), [Deepgram](https://docs.pipecat.ai/server/services/tts/deepgram), [ElevenLabs](https://docs.pipecat.ai/server/services/tts/elevenlabs), [Fish](https://docs.pipecat.ai/server/services/tts/fish), [Google](https://docs.pipecat.ai/server/services/tts/google), [LMNT](https://docs.pipecat.ai/server/services/tts/lmnt), [OpenAI](https://docs.pipecat.ai/server/services/tts/openai), [PlayHT](https://docs.pipecat.ai/server/services/tts/playht), [Rime](https://docs.pipecat.ai/server/services/tts/rime), [XTTS](https://docs.pipecat.ai/server/services/tts/xtts) | `pip install "pipecat-ai[cartesia]"` |
| Speech-to-Speech | [Gemini Multimodal Live](https://docs.pipecat.ai/server/services/s2s/gemini), [OpenAI Realtime](https://docs.pipecat.ai/server/services/s2s/openai) | `pip install "pipecat-ai[openai]"` |
| Transport | [Daily (WebRTC)](https://docs.pipecat.ai/server/services/transport/daily), WebSocket, Local | `pip install "pipecat-ai[daily]"` |
| Video | [Tavus](https://docs.pipecat.ai/server/services/video/tavus), [Simli](https://docs.pipecat.ai/server/services/video/simli) | `pip install "pipecat-ai[tavus,simli]"` |
| Vision & Image | [Moondream](https://docs.pipecat.ai/server/services/vision/moondream), [fal](https://docs.pipecat.ai/server/services/image-generation/fal) | `pip install "pipecat-ai[moondream]"` |
| Audio Processing | [Silero VAD](https://docs.pipecat.ai/server/utilities/audio/silero-vad-analyzer), [Krisp](https://docs.pipecat.ai/server/utilities/audio/krisp-filter), [Koala](https://docs.pipecat.ai/server/utilities/audio/koala-filter), [Noisereduce](https://docs.pipecat.ai/server/utilities/audio/noisereduce-filter) | `pip install "pipecat-ai[silero]"` |
| Analytics & Metrics | [Canonical AI](https://docs.pipecat.ai/server/services/analytics/canonical), [Sentry](https://docs.pipecat.ai/server/services/analytics/sentry) | `pip install "pipecat-ai[canonical]"` |
| Category | Services | Install Command Example |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- |
| Speech-to-Text | [AssemblyAI](https://docs.pipecat.ai/server/services/stt/assemblyai), [Azure](https://docs.pipecat.ai/server/services/stt/azure), [Deepgram](https://docs.pipecat.ai/server/services/stt/deepgram), [Gladia](https://docs.pipecat.ai/server/services/stt/gladia), [Whisper](https://docs.pipecat.ai/server/services/stt/whisper) | `pip install "pipecat-ai[deepgram]"` |
| LLMs | [Anthropic](https://docs.pipecat.ai/server/services/llm/anthropic), [Azure](https://docs.pipecat.ai/server/services/llm/azure), [Cerebras](https://docs.pipecat.ai/server/services/llm/cerebras), [DeepSeek](https://docs.pipecat.ai/server/services/llm/deepseek), [Fireworks AI](https://docs.pipecat.ai/server/services/llm/fireworks), [Gemini](https://docs.pipecat.ai/server/services/llm/gemini), [Grok](https://docs.pipecat.ai/server/services/llm/grok), [Groq](https://docs.pipecat.ai/server/services/llm/groq), [NVIDIA NIM](https://docs.pipecat.ai/server/services/llm/nim), [Ollama](https://docs.pipecat.ai/server/services/llm/ollama), [OpenAI](https://docs.pipecat.ai/server/services/llm/openai), [OpenRouter](https://docs.pipecat.ai/server/services/llm/openrouter), [Together AI](https://docs.pipecat.ai/server/services/llm/together) | `pip install "pipecat-ai[openai]"` |
| Text-to-Speech | [AWS](https://docs.pipecat.ai/server/services/tts/aws), [Azure](https://docs.pipecat.ai/server/services/tts/azure), [Cartesia](https://docs.pipecat.ai/server/services/tts/cartesia), [Deepgram](https://docs.pipecat.ai/server/services/tts/deepgram), [ElevenLabs](https://docs.pipecat.ai/server/services/tts/elevenlabs), [Fish](https://docs.pipecat.ai/server/services/tts/fish), [Google](https://docs.pipecat.ai/server/services/tts/google), [LMNT](https://docs.pipecat.ai/server/services/tts/lmnt), [OpenAI](https://docs.pipecat.ai/server/services/tts/openai), [PlayHT](https://docs.pipecat.ai/server/services/tts/playht), [Rime](https://docs.pipecat.ai/server/services/tts/rime), [XTTS](https://docs.pipecat.ai/server/services/tts/xtts) | `pip install "pipecat-ai[cartesia]"` |
| Speech-to-Speech | [Gemini Multimodal Live](https://docs.pipecat.ai/server/services/s2s/gemini), [OpenAI Realtime](https://docs.pipecat.ai/server/services/s2s/openai) | `pip install "pipecat-ai[openai]"` |
| Transport | [Daily (WebRTC)](https://docs.pipecat.ai/server/services/transport/daily), [FastAPI Websocket](https://docs.pipecat.ai/server/services/transport/fastapi-websocket), [WebSocket Server](https://docs.pipecat.ai/server/services/transport/websocket-server), Local | `pip install "pipecat-ai[daily]"` |
| Video | [Tavus](https://docs.pipecat.ai/server/services/video/tavus), [Simli](https://docs.pipecat.ai/server/services/video/simli) | `pip install "pipecat-ai[tavus,simli]"` |
| Vision & Image | [Moondream](https://docs.pipecat.ai/server/services/vision/moondream), [fal](https://docs.pipecat.ai/server/services/image-generation/fal) | `pip install "pipecat-ai[moondream]"` |
| Audio Processing | [Silero VAD](https://docs.pipecat.ai/server/utilities/audio/silero-vad-analyzer), [Krisp](https://docs.pipecat.ai/server/utilities/audio/krisp-filter), [Koala](https://docs.pipecat.ai/server/utilities/audio/koala-filter), [Noisereduce](https://docs.pipecat.ai/server/utilities/audio/noisereduce-filter) | `pip install "pipecat-ai[silero]"` |
| Analytics & Metrics | [Canonical AI](https://docs.pipecat.ai/server/services/analytics/canonical), [Sentry](https://docs.pipecat.ai/server/services/analytics/sentry) | `pip install "pipecat-ai[canonical]"` |
📚 [View full services documentation →](https://docs.pipecat.ai/server/services/supported-services)
@@ -160,15 +160,24 @@ From the root of this repo, run the following:
```shell
pip install -r dev-requirements.txt
python -m build
```
This builds the package. To use the package locally (e.g. to run sample files), run
This will install the necessary development dependencies. Also, make sure you install the git pre-commit hooks:
```shell
pre-commit install
```
The hooks will just save you time when you submit a PR by making sure your code follows the project rules.
To use the package locally (e.g. to run sample files), run:
```shell
pip install --editable ".[option,...]"
```
The `--editable` option makes sure you don't have to run `pip install` again and you can just edit the project files locally.
If you want to use this package from another directory, you can run:
```shell
@@ -180,7 +189,7 @@ pip install "path_to_this_repo[option,...]"
From the root directory, run:
```shell
pytest --doctest-modules --ignore-glob="*to_be_updated*" --ignore-glob=*pipeline_source* src tests
pytest
```
## Setting up your editor

View File

@@ -1,9 +1,11 @@
build~=1.2.2
grpcio-tools~=1.68.1
grpcio-tools~=1.69.0
pip-tools~=7.4.1
pyright~=1.1.390
pre-commit~=4.0.1
pyright~=1.1.392
pytest~=8.3.4
ruff~=0.8.3
setuptools~=75.6.0
pytest-asyncio~=0.25.2
ruff~=0.9.1
setuptools~=75.8.0
setuptools_scm~=8.1.0
python-dotenv~=1.0.1

View File

@@ -60,3 +60,27 @@ SIMLI_FACE_ID=...
# Krisp
KRISP_MODEL_PATH=...
# DeepSeek
DEEPSEEK_API_KEY=...
# Groq
GROQ_API_KEY=...
# Grok
GROK_API_KEY=...
# Together.ai
TOGETHER_API_KEY=...
# Cerebras
CEREBRAS_API_KEY=...
# Fish Audio
FISH_API_KEY=...
# Assembly AI
ASSEMBLYAI_API_KEY=...
# OpenRouter
OPENROUTER_API_KEY=...

View File

@@ -42,7 +42,7 @@ Next, follow the steps in the README for each demo.
| [Dialin Chatbot](dialin-chatbot) | A chatbot that connects to an incoming phone call from Daily or Twilio. | Deepgram, ElevenLabs, OpenAI, Daily, Twilio |
| [Twilio Chatbot](twilio-chatbot) | A chatbot that connects to an incoming phone call from Twilio. | Deepgram, ElevenLabs, OpenAI, Daily, Twilio |
| [studypal](studypal) | A chatbot to have a conversation about any article on the web | |
| [WebSocket Chatbot Server](websocket-server) | A real-time websocket server that handles audio streaming and bot interactions with speech-to-text and text-to-speech capabilities | `python-websockets`, `openai`, `deepgram`, `silero-tts`, `numpy` |
| [WebSocket Chatbot Server](websocket-server) | A real-time websocket server that handles audio streaming and bot interactions with speech-to-text and text-to-speech capabilities. | Cartesia, Deepgram, OpenAI, Websockets |
> [!IMPORTANT]
> These example projects use Daily as a WebRTC transport and can be joined using their hosted Prebuilt UI.

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -15,7 +15,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import EndFrame, LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -97,7 +97,7 @@ async def main():
call completion, CanonicalMetrics will send the audio buffer to Canonical for
analysis. Visit https://voice.canonical.chat to learn more.
"""
audio_buffer_processor = AudioBufferProcessor()
audio_buffer_processor = AudioBufferProcessor(num_channels=2)
canonical = CanonicalMetricsService(
audio_buffer_processor=audio_buffer_processor,
aiohttp_session=session,
@@ -105,6 +105,7 @@ async def main():
call_id=str(uuid.uuid4()),
assistant="pipecat-chatbot",
assistant_speaks_first=True,
context=context,
)
pipeline = Pipeline(
[
@@ -124,7 +125,7 @@ async def main():
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -18,7 +18,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import EndFrame, LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -134,7 +134,7 @@ async def main():
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -7,7 +7,7 @@ from dotenv import load_dotenv
from loguru import logger
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import EndFrame, LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -75,7 +75,7 @@ async def main(room_url: str, token: str):
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -13,7 +13,7 @@ logger.add(sys.stderr, level="DEBUG")
async def main(room_url: str, token: str):
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import EndFrame, LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -75,7 +75,7 @@ async def main(room_url: str, token: str):
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):

View File

@@ -1,5 +1,5 @@
python-dotenv==1.0.1
modal==0.65.48
pipecat-ai[daily,silero,cartesia,openai]==0.0.48
fastapi==0.115.4
aiohttp==3.11.9
modal==0.71.3
pipecat-ai[daily,silero,cartesia,openai]==0.0.52
fastapi==0.115.6
aiohttp==3.11.11

View File

@@ -37,7 +37,16 @@ Run `bot_runner.py` to handle incoming HTTP requests:
Then target the following URL:
`POST /daily_start_bot`
```bash
curl -X POST 'http://localhost:7860/daily_start_bot' \
-H 'Content-Type: application/json' \
-d '{
"callId": "callId-from-call",
"callDomain": "callDomain-from-call"
}'
```
Use [this guide](https://docs.pipecat.ai/guides/telephony/daily-webrtc) to connect a phone number purchased from Daily to the bot.
For more configuration options, please consult Daily's API documentation.
@@ -82,4 +91,4 @@ If you're using Twilio as a number vendor:
## Need to do something more advanced?
This demo covers the basics of bot telephony. If you want to know more about working with PSTN / SIP, please ping us on [Discord](https://discord.gg/pipecat).
This demo covers the basics of bot telephony. If you want to know more about working with PSTN / SIP, please ping us on [Discord](https://discord.gg/pipecat).

View File

@@ -1,3 +1,8 @@
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import argparse
import asyncio
import os
@@ -5,13 +10,16 @@ import sys
from dotenv import load_dotenv
from loguru import logger
from openai.types.chat import ChatCompletionToolParam
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import EndFrame, LLMMessagesFrame
from pipecat.frames.frames import EndFrame, TextFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.services.ai_services import LLMService
from pipecat.services.deepgram import DeepgramSTTService
from pipecat.services.elevenlabs import ElevenLabsTTSService
from pipecat.services.openai import OpenAILLMService
from pipecat.transports.services.daily import DailyDialinSettings, DailyParams, DailyTransport
@@ -55,16 +63,62 @@ async def main(room_url: str, token: str, callId: str, callDomain: str):
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
content = f"""
You are a delivery service customer support specialist supporting customers with their orders.
Begin with: "Hello, this is Hailey from customer support. What can I help you with today?"
"""
messages = [
{
"role": "system",
"content": "You are Chatbot, a friendly, helpful robot. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by saying 'Oh, hello! Who dares dial me at this hour?!'.",
"content": content,
},
]
context = OpenAILLMContext(messages)
tools = [
ChatCompletionToolParam(
type="function",
function={
"name": "transfer_call",
"description": "Transfer the call to a person. This function is used to connect the call to a real person. Examples of real people are: managers, supervisors, or other customer support specialists. Any person is okay as long as they are not a bot.",
"parameters": {
"type": "object",
"properties": {
"call_id": {
"type": "string",
"description": "This is always {callId}.",
},
"summary": {
"type": "string",
"description": """
Provide a concise summary in 3-5 sentences. Highlight any important details or unusual aspects of the conversation.
""",
},
},
},
},
)
]
context = OpenAILLMContext(messages, tools)
context_aggregator = llm.create_context_aggregator(context)
async def default_transfer_call(
function_name, tool_call_id, args, llm: LLMService, context, result_callback
):
logger.debug(f"default_transfer_call: {function_name} {tool_call_id} {args}")
await result_callback(
{
"transfer_call": False,
"reason": "To transfer call calls, please dial in to the room using a phone or a SIP client.",
}
)
llm.register_function(
function_name="transfer_call",
callback=default_transfer_call,
)
pipeline = Pipeline(
[
transport.input(),
@@ -81,12 +135,50 @@ async def main(room_url: str, token: str, callId: str, callDomain: str):
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
@transport.event_handler("on_dialin_ready")
async def on_dialin_ready(_, sip_endpoint):
logger.info(f"on_dialin_ready: {sip_endpoint}")
@transport.event_handler("on_dialin_connected")
async def on_dialin_connected(transport, event):
logger.info(f"on_dialin_connected: {event}")
sip_session_id = event["sessionId"]
async def transfer_call(
function_name, tool_call_id, args, llm: LLMService, context, result_callback
):
logger.debug(f"transfer_call: {function_name} {tool_call_id} {args}")
# sip_url = "sip:your_user_name@sip.linphone.org"
sip_url = (
f"sip:your_username@dailyco.sip.twilio.com?x-daily_id={room_url.split('/')[-1]}"
)
try:
await transport.sip_refer(
settings={
"sessionId": sip_session_id,
"toEndPoint": sip_url,
}
)
except Exception as e:
logger.error(f"An error occurred during SIP refer: {e}")
await result_callback({"transfer_call": False})
await result_callback({"transfer_call": True})
llm.register_function(
function_name="transfer_call",
callback=transfer_call,
)
runner = PipelineRunner()
await runner.run(task)

View File

@@ -8,7 +8,7 @@ from loguru import logger
from twilio.rest import Client
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import EndFrame, LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -83,7 +83,7 @@ async def main(room_url: str, token: str, callId: str, sipUri: str):
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,7 +14,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import Frame, LLMMessagesFrame, MetricsFrame
from pipecat.frames.frames import EndFrame, Frame, MetricsFrame
from pipecat.metrics.metrics import (
LLMUsageMetricsData,
ProcessingMetricsData,
@@ -47,9 +47,7 @@ class MetricsLogger(FrameProcessor):
elif isinstance(d, LLMUsageMetricsData):
tokens = d.value
print(
f"!!! MetricsFrame: {frame}, tokens: {
tokens.prompt_tokens}, characters: {
tokens.completion_tokens}"
f"!!! MetricsFrame: {frame}, tokens: {tokens.prompt_tokens}, characters: {tokens.completion_tokens}"
)
elif isinstance(d, TTSUsageMetricsData):
print(f"!!! MetricsFrame: {frame}, characters: {d.value}")
@@ -113,7 +111,11 @@ async def main():
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -15,7 +15,7 @@ from PIL import Image
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import Frame, OutputImageRawFrame, SystemFrame, TextFrame
from pipecat.frames.frames import EndFrame, Frame, OutputImageRawFrame, SystemFrame, TextFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask
@@ -126,6 +126,10 @@ async def main():
await transport.capture_participant_transcription(participant["id"])
await task.queue_frames([TextFrame(f"Hi there {participant_name}!")])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()
await runner.run(task)

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -13,7 +13,7 @@ from dotenv import load_dotenv
from loguru import logger
from runner import configure
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -90,7 +90,11 @@ async def main():
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,7 +14,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -88,7 +88,11 @@ async def main():
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,7 +14,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -78,13 +78,25 @@ async def main():
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
report_only_initial_ttfb=True,
),
)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -19,7 +19,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame, LLMMessagesFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -101,7 +101,15 @@ async def main():
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
report_only_initial_ttfb=True,
),
)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
@@ -114,6 +122,10 @@ async def main():
messages = [({"content": "Please briefly introduce yourself to the user."})]
await task.queue_frames([LLMMessagesFrame(messages)])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()
await runner.run(task)

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -16,7 +16,7 @@ from runner import configure
from pipecat.frames.frames import (
BotInterruptionFrame,
LLMMessagesFrame,
EndFrame,
StopInterruptionFrame,
UserStartedSpeakingFrame,
UserStoppedSpeakingFrame,
@@ -80,7 +80,15 @@ async def main():
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
report_only_initial_ttfb=True,
),
)
@stt.event_handler("on_speech_started")
async def on_speech_started(stt, *args, **kwargs):
@@ -94,7 +102,11 @@ async def main():
async def on_first_participant_joined(transport, participant):
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,7 +14,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -73,13 +73,25 @@ async def main():
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
report_only_initial_ttfb=True,
),
)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,7 +14,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -88,7 +88,11 @@ async def main():
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -0,0 +1,105 @@
#
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import asyncio
import os
import sys
import aiohttp
from dotenv import load_dotenv
from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.services.openai import OpenAILLMService
from pipecat.services.playht import PlayHTHttpTTSService
from pipecat.transcriptions.language import Language
from pipecat.transports.services.daily import DailyParams, DailyTransport
load_dotenv(override=True)
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
async def main():
async with aiohttp.ClientSession() as session:
(room_url, token) = await configure(session)
transport = DailyTransport(
room_url,
token,
"Respond bot",
DailyParams(
audio_out_enabled=True,
transcription_enabled=True,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
),
)
tts = PlayHTHttpTTSService(
user_id=os.getenv("PLAYHT_USER_ID"),
api_key=os.getenv("PLAYHT_API_KEY"),
voice_url="s3://voice-cloning-zero-shot/d9ff78ba-d016-47f6-b0ef-dd630f59414e/female-cs/manifest.json",
)
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
messages = [
{
"role": "system",
"content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
},
]
context = OpenAILLMContext(messages)
context_aggregator = llm.create_context_aggregator(context)
pipeline = Pipeline(
[
transport.input(), # Transport user input
context_aggregator.user(), # User responses
llm, # LLM
tts, # TTS
transport.output(), # Transport bot output
context_aggregator.assistant(), # Assistant spoken responses
]
)
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
report_only_initial_ttfb=True,
),
)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()
await runner.run(task)
if __name__ == "__main__":
asyncio.run(main())

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,7 +14,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -91,7 +91,11 @@ async def main():
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,7 +14,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -82,14 +82,26 @@ async def main():
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
report_only_initial_ttfb=True,
),
)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,7 +14,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -70,14 +70,26 @@ async def main():
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
report_only_initial_ttfb=True,
),
)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -15,7 +15,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -80,14 +80,26 @@ async def main():
]
)
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
report_only_initial_ttfb=True,
),
)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,7 +14,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -74,14 +74,26 @@ async def main():
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
report_only_initial_ttfb=True,
),
)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,7 +14,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import EndFrame, LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -79,14 +79,22 @@ async def main():
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
report_only_initial_ttfb=True,
),
)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
# Register an event handler to exit the application when the user leaves.
@transport.event_handler("on_participant_left")

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,7 +14,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -71,14 +71,26 @@ async def main():
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
report_only_initial_ttfb=True,
),
)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,7 +14,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -90,7 +90,10 @@ async def main():
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True, enable_metrics=True, enable_usage_metrics=True
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
report_only_initial_ttfb=True,
),
)
@@ -98,7 +101,11 @@ async def main():
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,7 +14,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -80,14 +80,26 @@ async def main():
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
report_only_initial_ttfb=True,
),
)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,7 +14,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -22,6 +22,7 @@ from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.services.deepgram import DeepgramSTTService
from pipecat.services.google import GoogleTTSService
from pipecat.services.openai import OpenAILLMService
from pipecat.transcriptions.language import Language
from pipecat.transports.services.daily import DailyParams, DailyTransport
load_dotenv(override=True)
@@ -50,8 +51,8 @@ async def main():
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
tts = GoogleTTSService(
voice_id="en-US-Neural2-J",
params=GoogleTTSService.InputParams(language="en-US", rate="1.05"),
voice_id="en-US-Journey-F",
params=GoogleTTSService.InputParams(language=Language.EN_US),
)
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")
@@ -78,14 +79,26 @@ async def main():
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
report_only_initial_ttfb=True,
),
)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,7 +14,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -79,14 +79,26 @@ async def main():
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
report_only_initial_ttfb=True,
),
)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,14 +14,11 @@ from loguru import logger
from runner import configure
from pipecat.audio.filters.krisp_filter import KrispFilter
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.services.deepgram import DeepgramSTTService, DeepgramTTSService
from pipecat.services.openai import OpenAILLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport
@@ -63,28 +60,40 @@ async def main():
},
]
tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages)
context = OpenAILLMContext(messages)
context_aggregator = llm.create_context_aggregator(context)
pipeline = Pipeline(
[
transport.input(), # Transport user input
stt, # STT
tma_in, # User responses
context_aggregator.user(), # User responses
llm, # LLM
tts, # TTS
transport.output(), # Transport bot output
tma_out, # Assistant spoken responses
context_aggregator.assistant(), # Assistant spoken responses
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
report_only_initial_ttfb=True,
),
)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,7 +14,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -89,7 +89,11 @@ async def main():
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,7 +14,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -81,7 +81,11 @@ async def main():
async def on_first_participant_joined(transport, participant):
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -17,6 +17,7 @@ from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import (
EndFrame,
Frame,
InputAudioRawFrame,
LLMFullResponseEndFrame,
@@ -268,6 +269,10 @@ async def main():
# Kick off the conversation.
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()
await runner.run(task)

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,7 +14,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -88,7 +88,11 @@ async def main():
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -0,0 +1,148 @@
#
# Copyright (c) 2025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import asyncio
import os
import sys
import aiohttp
from dotenv import load_dotenv
from loguru import logger
from openai.types.chat import ChatCompletionToolParam
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.services.cartesia import CartesiaTTSService
from pipecat.services.deepseek import DeepSeekLLMService
from pipecat.services.openai import OpenAILLMContext
from pipecat.transports.services.daily import DailyParams, DailyTransport
load_dotenv(override=True)
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
async def start_fetch_weather(function_name, llm, context):
# note: we can't push a frame to the LLM here. the bot
# can interrupt itself and/or cause audio overlapping glitches.
# possible question for Aleix and Chad about what the right way
# to trigger speech is, now, with the new queues/async/sync refactors.
# await llm.push_frame(TextFrame("Let me check on that."))
logger.debug(f"Starting fetch_weather_from_api with function_name: {function_name}")
async def fetch_weather_from_api(function_name, tool_call_id, args, llm, context, result_callback):
await result_callback({"conditions": "nice", "temperature": "75"})
async def main():
async with aiohttp.ClientSession() as session:
(room_url, token) = await configure(session)
transport = DailyTransport(
room_url,
token,
"Respond bot",
DailyParams(
audio_out_enabled=True,
transcription_enabled=True,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
),
)
tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY"),
voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady
)
llm = DeepSeekLLMService(api_key=os.getenv("DEEPSEEK_API_KEY"), model="deepseek-chat")
# Register a function_name of None to get all functions
# sent to the same callback with an additional function_name parameter.
llm.register_function(None, fetch_weather_from_api, start_callback=start_fetch_weather)
tools = [
ChatCompletionToolParam(
type="function",
function={
"name": "get_current_weather",
"description": "Get the current weather for a specific location. You MUST use this function whenever asked about weather.",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA",
},
"format": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "The temperature unit to use. Use fahrenheit for US locations, celsius for others.",
},
},
"required": ["location", "format"],
},
},
)
]
messages = [
{
"role": "system",
"content": """You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way.
You have one functions available:
1. get_current_weather is used to get current weather information.
Infer whether to use Fahrenheit or Celsius automatically based on the location, unless the user specifies a preference.
Start by asking me for my location. Then, use 'get_weather_current' to give me a forecast.
Respond to what the user said in a creative and helpful way.""",
},
]
context = OpenAILLMContext(messages, tools)
context_aggregator = llm.create_context_aggregator(context)
pipeline = Pipeline(
[
transport.input(),
context_aggregator.user(),
llm,
tts,
transport.output(),
context_aggregator.assistant(),
]
)
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
report_only_initial_ttfb=True,
),
)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
await task.queue_frames([context_aggregator.user().get_context_frame()])
runner = PipelineRunner()
await runner.run(task)
if __name__ == "__main__":
asyncio.run(main())

View File

@@ -0,0 +1,142 @@
#
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import asyncio
import os
import sys
import aiohttp
from dotenv import load_dotenv
from loguru import logger
from openai.types.chat import ChatCompletionToolParam
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.services.azure import AzureTTSService
from pipecat.services.openrouter import OpenRouterLLMService
from pipecat.transports.services.daily import DailyParams, DailyTransport
load_dotenv(override=True)
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
async def start_fetch_weather(function_name, llm, context):
# note: we can't push a frame to the LLM here. the bot
# can interrupt itself and/or cause audio overlapping glitches.
# possible question for Aleix and Chad about what the right way
# to trigger speech is, now, with the new queues/async/sync refactors.
# await llm.push_frame(TextFrame("Let me check on that."))
logger.debug(f"Starting fetch_weather_from_api with function_name: {function_name}")
async def fetch_weather_from_api(function_name, tool_call_id, args, llm, context, result_callback):
await result_callback({"conditions": "nice", "temperature": "75"})
async def main():
async with aiohttp.ClientSession() as session:
(room_url, token) = await configure(session)
transport = DailyTransport(
room_url,
token,
"Respond bot",
DailyParams(
audio_out_enabled=True,
transcription_enabled=True,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
),
)
tts = AzureTTSService(
api_key=os.getenv("AZURE_API_KEY"),
region="eastus",
voice="en-US-JennyNeural",
params=AzureTTSService.InputParams(language="en-US", rate="1.1", style="cheerful"),
)
llm = OpenRouterLLMService(
api_key=os.getenv("OPENROUTER_API_KEY"), model="openai/gpt-4o-2024-11-20"
)
# Register a function_name of None to get all functions
# sent to the same callback with an additional function_name parameter.
llm.register_function(None, fetch_weather_from_api, start_callback=start_fetch_weather)
tools = [
ChatCompletionToolParam(
type="function",
function={
"name": "get_current_weather",
"description": "Get the current weather",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA",
},
"format": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "The temperature unit to use. Infer this from the users location.",
},
},
"required": ["location", "format"],
},
},
)
]
messages = [
{
"role": "system",
"content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
},
]
context = OpenAILLMContext(messages, tools)
context_aggregator = llm.create_context_aggregator(context)
pipeline = Pipeline(
[
transport.input(),
context_aggregator.user(),
llm,
tts,
transport.output(),
context_aggregator.assistant(),
]
)
task = PipelineTask(
pipeline,
PipelineParams(
allow_interruptions=True,
enable_metrics=True,
enable_usage_metrics=True,
report_only_initial_ttfb=True,
),
)
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
await task.queue_frames([context_aggregator.user().get_context_frame()])
runner = PipelineRunner()
await runner.run(task)
if __name__ == "__main__":
asyncio.run(main())

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -15,7 +15,6 @@ from openai.types.chat import ChatCompletionToolParam
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.pipeline.parallel_pipeline import ParallelPipeline
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
@@ -146,7 +145,7 @@ async def main():
"content": f"Please introduce yourself to the user and let them know the voices you can do. Your initial responses should be as if you were a {current_voice}.",
}
)
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -16,7 +16,6 @@ from openai.types.chat import ChatCompletionToolParam
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.pipeline.parallel_pipeline import ParallelPipeline
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
@@ -139,7 +138,7 @@ async def main():
"content": f"Please introduce yourself to the user and let them know the languages you speak. Your initial responses should be in {current_language}.",
}
)
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,7 +14,6 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
@@ -98,7 +97,7 @@ async def main():
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
# Handle "latency-ping" messages. The client will send app messages that look like
# this:

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -100,7 +100,7 @@ async def main():
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,14 +14,11 @@ from dotenv import load_dotenv
from loguru import logger
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.frames.frames import EndFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.services.cartesia import CartesiaTTSService
from pipecat.services.deepgram import DeepgramSTTService
from pipecat.services.openai import OpenAILLMService
@@ -39,7 +36,6 @@ async def main():
tavus = TavusVideoService(
api_key=os.getenv("TAVUS_API_KEY"),
replica_id=os.getenv("TAVUS_REPLICA_ID"),
persona_id=os.getenv("TAVUS_PERSONA_ID", "pipecat0"),
session=session,
)
@@ -74,19 +70,19 @@ async def main():
},
]
tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages)
context = OpenAILLMContext(messages)
context_aggregator = llm.create_context_aggregator(context)
pipeline = Pipeline(
[
transport.input(), # Transport user input
stt, # STT
tma_in, # User responses
context_aggregator.user(), # User responses
llm, # LLM
tts, # TTS
tavus, # Tavus output layer
transport.output(), # Transport bot output
tma_out, # Assistant spoken responses
context_aggregator.assistant(), # Assistant spoken responses
]
)
@@ -120,7 +116,11 @@ async def main():
messages.append(
{"role": "system", "content": "Please introduce yourself to the user."}
)
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
await task.queue_frame(EndFrame())
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -14,7 +14,7 @@ from loguru import logger
from runner import configure
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMMessagesFrame, TextFrame
from pipecat.frames.frames import TextFrame
from pipecat.pipeline.parallel_pipeline import ParallelPipeline
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
@@ -155,7 +155,7 @@ async def main():
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
runner = PipelineRunner()

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -311,7 +311,7 @@ async def main():
await transport.capture_participant_transcription(participant["id"])
# Kick off the conversation.
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_app_message")
async def on_app_message(transport, message, sender):

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
@@ -525,7 +525,7 @@ async def main():
"content": "Start by just saying \"Hello I'm ready.\" Don't say anything else.",
}
)
await task.queue_frames([LLMMessagesFrame(messages)])
await task.queue_frames([context_aggregator.user().get_context_frame()])
@transport.event_handler("on_app_message")
async def on_app_message(transport, message, sender):

View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2024, Daily
# Copyright (c) 20242025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#

Some files were not shown because too many files have changed in this diff Show More