Aleix Conchillo Flaqué
c0f9ad19fe
all frame processors are asynchrnous
...
In this commit we make all frame processors asynchronous, that is, they have an
internal queue and they push frames using a task from that queue.
2024-09-30 13:17:50 -07:00
Aleix Conchillo Flaqué
e276dcbab7
initialize task variables and add minor description
2024-09-24 19:19:00 -07:00
Aleix Conchillo Flaqué
eeb8338dce
introduce Ruff formatting
2024-09-23 09:53:37 -07:00
Aleix Conchillo Flaqué
7e39d9ad3d
introduce input/output audio and image frames
...
We now distinguish between input and output audio and image frames. We introduce
`InputAudioRawFrame`, `OutputAudioRawFrame`, `InputImageRawFrame` and
`OutputImageRawFrame` (and other subclasses of those). The input frames usually
come from an input transport and are meant to be processed inside the pipeline
to generate new frames. However, the input frames will not be sent through an
output transport. The output frames can also be processed by any frame processor
in the pipeline and they are allowed to be sent by the output transport.
2024-09-19 23:11:03 -07:00
Aleix Conchillo Flaqué
fbf6eef68f
transports(base_output): wait for sink tasks before canceling audio/video tasks
2024-09-19 01:31:12 -07:00
Aleix Conchillo Flaqué
f078d156de
frames: StartFrame is now a SystemFrame
2024-09-19 01:31:12 -07:00
Aleix Conchillo Flaqué
337f048864
introduce synchronous and asynchronous frame processors
...
Pipecat has a pipeline-based architecture. The pipeline consists of frame
processors linked to each other. The elements travelling across the pipeline are
called frames.
To have a deterministic behavior the frames travelling through the pipeline
should always be ordered, except system frames which are out-of-band frames. To
achieve that, each frame processor should only output frames from a single task.
There are synchronous and asynchronous frame processors. The synchronous
processors push output frames from the same task that they receive input frames,
and therefore only pushing frames from one task. Asynchrnous frame processors
can have internal tasks to perform things asynchrnously (e.g. receiving data
from a websocket) but they also have a single task where they push frames from.
2024-09-19 01:31:10 -07:00
Aleix Conchillo Flaqué
3665734972
transports(output): initial sink clock synchronization
2024-09-12 00:37:34 -07:00
Aleix Conchillo Flaqué
37e2388758
StartFrame should be the first frame every processor receives
...
Fixes #427
2024-08-29 22:43:44 -07:00
Aleix Conchillo Flaqué
21de8e0a35
transport(out): log bot started/stopped speaking
2024-08-21 17:23:44 -07:00
Aleix Conchillo Flaqué
67d565930e
services: send TTSStartFrame/TTSStopFrame when really needed
2024-08-15 11:05:10 -07:00
Aleix Conchillo Flaqué
425a730d7c
transports(base_output): send urgent transport messages immediately
2024-08-15 11:05:10 -07:00
Aleix Conchillo Flaqué
83a037a7ce
Merge pull request #345 from pipecat-ai/aleix/base-output-render-time-fixes
...
transports(base_output): improve render sleep computation
2024-08-06 17:30:47 -07:00
Aleix Conchillo Flaqué
541072f8e0
transports(base_output): improve render sleep computation
2024-08-06 17:20:41 -07:00
Aleix Conchillo Flaqué
881248cbd6
transport(base_output): push audio downstream
2024-08-05 14:00:09 -07:00
Aleix Conchillo Flaqué
d5d88f756a
transport(output): improve audio and image handling for video use cases
2024-08-04 12:29:08 -07:00
Aleix Conchillo Flaqué
77723b34c7
EndFrame tries to end gracefully CancelFrame cancels tasks
2024-07-30 11:41:19 -07:00
Aleix Conchillo Flaqué
76aca32f2e
transport(output): emit new bot start|stop speaking frames
2024-07-25 14:50:33 -07:00
Aleix Conchillo Flaqué
65cdf50774
processors(rtvi): fix task cleanup
2024-07-22 15:01:45 -07:00
Aleix Conchillo Flaqué
d7c3e380a5
added BotSpeakingFrame
2024-07-01 14:57:18 -07:00
Aleix Conchillo Flaqué
fc0920504d
improve error handling and don't swallow exceptions
2024-07-01 09:35:45 -07:00
Aleix Conchillo Flaqué
d3e410b2ac
transports: fully use asyncio in all read/write operations
2024-06-21 15:55:15 -07:00
Aleix Conchillo Flaqué
2a29cb18a5
transports(base_output): chunk audio into 20ms instead of 10ms
2024-06-13 15:30:41 -07:00
Aleix Conchillo Flaqué
099e65f3b6
report processor name in error logs
2024-06-13 10:20:45 -07:00
Aleix Conchillo Flaqué
0225443ec8
transports(base): always send MetricsFrame
2024-06-12 12:15:29 -07:00
Aleix Conchillo Flaqué
83f69e02fd
allow specifying frame processor names
2024-06-12 12:15:29 -07:00
Aleix Conchillo Flaqué
7410fe1d1e
transport: chunk longer audio frames
2024-06-11 17:50:51 -07:00
Aleix Conchillo Flaqué
6554479d39
transports: don't queue system frames
2024-06-10 21:00:01 -07:00
Aleix Conchillo Flaqué
a1f1d1995c
transports: allow sending metrics
2024-06-06 14:35:34 -07:00
Aleix Conchillo Flaqué
390582d7f3
services: use start/stop_ttfb_metrics to report TTFB metrics
2024-06-06 14:00:10 -07:00
Aleix Conchillo Flaqué
e765a29ca2
processors: implement base process_frame(). all subsclassed should call it
2024-06-06 10:54:21 -07:00
Aleix Conchillo Flaqué
891b7b22ea
transports: push EndFrame/CancelFrame before stopping push task
2024-06-04 15:43:54 -07:00
Aleix Conchillo Flaqué
a674b43243
transport: remove redundant camera thread and switch audio pull for push
2024-06-04 15:43:54 -07:00
chadbailey59
4c3d19cc8b
Function calling ( #175 )
...
* added function calling code back
* removed old llm_context file
* added integration testing for openai
* added function calling example
* added function callbacks
* added function start callback
* fixup
* fixup
* added different return type support for function calling
* intake example working
* added frame loggers
* cleanup
* fixup
* Update openai.py
* removed function call frame types
* fixup
* re-added example
* renumbered wake phrase
* fixup for autopep8
* remove unused imports
2024-05-30 12:25:39 -05:00
Aleix Conchillo Flaqué
aecb9f5816
transport(output): fix high CPU usage with camera_out_enabled and no images
2024-05-30 10:18:43 -07:00
Aleix Conchillo Flaqué
34384881bc
transport(base): clear audio output buffer if interrupted
2024-05-22 16:30:43 -07:00
Aleix Conchillo Flaqué
825738440e
transports: run threads in their own ThreadPoolExecutor
2024-05-21 18:52:27 -07:00
Aleix Conchillo Flaqué
0e8c7a9b28
transports(output): create an downstream push frame task
2024-05-17 17:47:24 -07:00
Aleix Conchillo Flaqué
34762bf604
transports: allows update allow_interruptinos when receiving StartFrame
2024-05-17 14:15:37 -07:00
Aleix Conchillo Flaqué
0bef44c2ff
introduce StartInterruptionFrame and StopInterruptionFrame
2024-05-17 13:51:06 -07:00
Aleix Conchillo Flaqué
dc9377fb92
add missing queue task_done()
2024-05-17 13:50:48 -07:00
Aleix Conchillo Flaqué
7384b63b1d
initial interruptions support
2024-05-17 13:50:45 -07:00
Aleix Conchillo Flaqué
af10adb7fe
some minor event loop updates
2024-05-15 17:00:43 -07:00
Aleix Conchillo Flaqué
3d6783ddb0
transports: resize output image if it doesn't match camera
2024-05-15 15:36:20 -07:00
Aleix Conchillo Flaqué
acf6dc0a30
transports: more start and stop fixes
2024-05-15 15:23:03 -07:00
Aleix Conchillo Flaqué
20c679988c
transports: allow base transports to be reused
2024-05-14 00:28:43 -07:00
Aleix Conchillo Flaqué
1b21867a6f
transports: add support for sending transport messages
2024-05-13 16:22:11 -07:00
Aleix Conchillo Flaqué
b254525d3c
go back to using @dataclass since they can be inspected
2024-05-12 22:35:43 -07:00
Aleix Conchillo Flaqué
b026915d19
initial commit for new pipecat architecture
2024-05-12 10:07:25 -07:00
