If a FrameProcessor needs to create a task it should use
FrameProcessor.create_task() and FrameProcessor.cancel_task(). This gives
Pipecat more control over all the tasks that are created in Pipecat.
Both functions internally use the utils module: utils.create_task() and
utils.cancel_task() which should also be used outside of FrameProcessors. That
is, unless strictly necessary, we should avoid using asyncio.create_task().
Frame processors can now decide if they should continue processing frames or
not, and if so also decide when to continue processing frames. For example,
asynchronous TTS services will stop processing frames until they have generated
all the audio for an LLM response.
* feat: Add Sentry support in FrameProcessor
This update add optional Sentry integration for performance tracking and error monitoring.
Key changes include:
- Add conditional Sentry import and initialization check
- Implement Sentry spans in FrameProcessorMetrics to measure TTFB (Time To First Byte) and processing time when Sentry is available
- Maintain existing metrics functionality with MetricsFrame regardless of Sentry availability
* feat: Enable metrics in DeepgramSTTService for Sentry
This commit enhances the DeepgramSTTService class to enable metrics generation for use with Sentry.
Key changes include:
1. Enable general metrics generation:
- Implement `can_generate_metrics` method, returning True when VAD is enabled
- This allows metrics to be collected and used by both Sentry and the metrics system in frame_processor.py
2. Integrate Sentry-compatible performance tracking:
- Add start_ttfb_metrics and start_processing_metrics calls in the VAD speech detection handler
- Implement stop_ttfb_metrics call when receiving transcripts
- Add stop_processing_metrics for final transcripts
3. Enhance VAD support for metrics:
- Add `vad_enabled` property to check VAD event availability
- Implement VAD-based speech detection handler for precise metric timing
These changes enable detailed performance tracking via both Sentry and the general metrics system when VAD is active. This allows for better monitoring and analysis of the speech-to-text process, providing valuable insights through Sentry and any other metrics consumers in the pipeline.
* Update frame_processor.py
* Refactor to support flexible metrics implementation
- Modified the __init__ method to accept a metrics parameter that is either FrameProcessorMetrics or one of its subclasses
- Updated the metrics initialization to create an instance with the processor's name
- Moved all FrameProcessorMetrics-related logic to a new processors\metrics\base.py file
* Implement flexible metrics system with Sentry integration
1. Created a new metrics module in processors/metrics/
2. Implemented FrameProcessorMetrics base class in base.py:
3. Implemented SentryMetrics class in sentry.py:
- Inherits from FrameProcessorMetrics
- Integrates with Sentry SDK for advanced metrics tracking
- Implements Sentry-specific span creation and management for TTFB and processing metrics
- Handles cases where Sentry is not available or initialized
1. Fleshed out MetricsFrames and broke it into a proper set of types
2. Add model_name as a property to the AIService so that it can be
automatically included in metrics and also remove that
overhead from all the various services themselves
Breaking change!
Because of the types improvements, the MetricsFrame type has
changed. Each frame will have a list of metrics simlilar to before
except each item in the list will only contain one type of metric:
"ttfb", "tokens", "characters", or "processing". Previously these
fields would be in every entry but set to None if they didn't apply.
While this changes internal handling of the MetricsFrame, it does NOT
break the RTVI/daily messaging of metrics. That format remains the same.
Also. Remember to use model_name for accessing a service's current
model and set_model_name for setting it.
Pipecat has a pipeline-based architecture. The pipeline consists of frame
processors linked to each other. The elements travelling across the pipeline are
called frames.
To have a deterministic behavior the frames travelling through the pipeline
should always be ordered, except system frames which are out-of-band frames. To
achieve that, each frame processor should only output frames from a single task.
There are synchronous and asynchronous frame processors. The synchronous
processors push output frames from the same task that they receive input frames,
and therefore only pushing frames from one task. Asynchrnous frame processors
can have internal tasks to perform things asynchrnously (e.g. receiving data
from a websocket) but they also have a single task where they push frames from.
