We now distinguish between input and output audio and image frames. We introduce
`InputAudioRawFrame`, `OutputAudioRawFrame`, `InputImageRawFrame` and
`OutputImageRawFrame` (and other subclasses of those). The input frames usually
come from an input transport and are meant to be processed inside the pipeline
to generate new frames. However, the input frames will not be sent through an
output transport. The output frames can also be processed by any frame processor
in the pipeline and they are allowed to be sent by the output transport.
Pipecat has a pipeline-based architecture. The pipeline consists of frame
processors linked to each other. The elements travelling across the pipeline are
called frames.
To have a deterministic behavior the frames travelling through the pipeline
should always be ordered, except system frames which are out-of-band frames. To
achieve that, each frame processor should only output frames from a single task.
There are synchronous and asynchronous frame processors. The synchronous
processors push output frames from the same task that they receive input frames,
and therefore only pushing frames from one task. Asynchrnous frame processors
can have internal tasks to perform things asynchrnously (e.g. receiving data
from a websocket) but they also have a single task where they push frames from.
* processors(rtvi): rtvi 0.1 message protocol
* added a single function call handler
* wip - function calling
* fixup
* fixup
* fixup
* processors(rtvi): no need for configure_on_start()
* processors(rtvi): add new option values if they haven't been set yet
* Add the model name to the LLM usage metrics
* wip - anthropic tool calling
* still wip - anthropic tool use and vision
* anthropic tools and vision working
* anthropic tool calling and vision
* Cartesia error handling
* Anthropic tool use core Pipecat pieces refactored as per plan
* aleix has good ideas
* Usage metrics for Anthropic LLMs
* fix function call result state not getting cleared bug
* Pass **kwargs through from AnthropicLLMService constructor
* about to tinker with anthropic
* added openai function calling
* openai function calling
* fixup
---------
Co-authored-by: Aleix Conchillo Flaqué <aleix@daily.co>
Co-authored-by: Chad Bailey <chadbailey@gmail.com>
Co-authored-by: mattie ruth backman <mattieruth@gmail.com>
Co-authored-by: chadbailey59 <chadbailey59@users.noreply.github.com>
