This new processor wraps an aggregator that can be overridden for the purposes
of customizing how the llm output gets categorized and handled in the pipeline.
Along with this, we are deprecating the ability to override the default
aggregator in the TTS to encourage use of the LLMTextProcessor in cases where
custome aggregation is needed.
This PR also:
- Introduces TTSService.transform_aggregation_type():
This function provides the ability to provide callbacks to the TTS to
transform text based on its aggregated type prior to sending the text to the
underlying TTS service. This makes it possible to do things like introduce
TTS-specific tags for spelling or emotion or change the pronunciation of
something on the fly.
- Introduces to the RTVIObserver:
- new init field skip_aggregator_types: A way to provide a list of aggregation
types that should not be included in bot-output (or tts-text) messages
- transform_aggregation_type(): Same as with TTSService, this allows you
to provide a callback to transform text being sent as bot-output before
it gets sent.
1. Added support for turning off bot-output messages with the bot_output_enabled flag
2. Cleaned up logic and comments around TTSService:_push_tts_frames to hopefully make
it easier to understand
3. Other minor cleanup
This allows any given TextFrame to be marked in a way such that it does not get
added to the context.
Specifically, this fixes a problem with the new AggregatedTextFrames where we
need to send LLM text both in an aggregated form as well as word-by-word but
avoid duplicating the text in the context.
1. Fixed pattern_pair_aggregator to support various ways of handling
pattern matches (remove, keep and just trigger a callback, or
aggregate
2. Fixed ivr_navigator use of pattern_pair_aggregator
3. Test fixes -- Tests now pass
1. TTSTextFrames now include metadata about whether the text was spoken
or not along with a type string to describe what the text represents:
ex. "sentence", "word", "custom aggregation"
2. Expanded how aggregators work so that the aggregate method returns
aggregated text along with the type of aggregation used to create it
3. Deprecated the RTVI bot-transcription event in lieu of...
4. Introduced support for a new bot-output event. This event is meant
to be the one stop shop for communicating what the bot actually "says".
It is based off TTSTextFrames to communicate both sentence by sentence
(or whatever aggregation is used) as well as word by word. In addition,
it will include LLMTextFrames, aggregated by sentence when tts is
turned off (i.e. skip_tts is true).
Resolvespipecat-ai/pipecat-client-web#158
* feat: Add ErrorFrame emission to TTS/STT services for pipeline error detection
- Add ErrorFrame emission to all major TTS/STT services during initialization and runtime failures
- Services updated: Cartesia, ElevenLabs, Deepgram, AssemblyAI, Rime, Azure
- ErrorFrame objects emitted with fatal=False for graceful degradation
- Enables on_pipeline_error event handler to detect service failures programmatically
- Add comprehensive pytest test suite to verify ErrorFrame emission
- Fixes issue where services failed gracefully but didn't emit ErrorFrame objects
This allows developers to implement real-time error monitoring and alerting
using the on_pipeline_error event handler introduced in v0.0.90.
* Update STT and TTS services to use consistent error handling pattern
- Improves error handling consistency across all services
* Add changelog entry for STT/TTS error handling improvements
* Linting issues Resolved
* Azure STT ErrorFrames added with consistent patterns
* Cartesia STT and Deepgram STT; additional fixes made
* Removed Fatal Flags across services, removed duplication
* Moving the changelog entry to the correct place.
* Refactoring some classes to use yield instead of push_error directly.
* Fixing ruff format.
---------
Co-authored-by: Filipi Fuchter <filipi87@gmail.com>
- `GoogleHttpTTSService`
- `OpenAITTSService`
The reason I skipped this work in an earlier PR was because these services seemed to be emitting long, punctuation-free text frames. It turns out that the issue was with the LLM prompt, though, resulting in the LLM nondeterministically excluding all punctuation. An upcoming commit will address that prompt issue.
Note that for `LLMTextFrame`s, the right behavior is pretty much always `includes_inter_frame_spaces = True`. I decided *not* to go ahead and make that the default for `LLMTextFrame`s, though, simply to not introduce a subtle behavior change for creative/unexpected use-cases that were relying on text in hand-crafted `LLMTextFrame`s being handled a certain way. Ditto for `TTSTextFrame`s.
Also, fix an issue in `NeuphonicTTSService` where it wasn't pushing `TTSTextFrame`s.
Also, fix the broken `SarvamHttpTTSService` example.
Also, add a couple of missing examples.
* Fix Langfuse tracing for GoogleLLMService with universal LLMContext
- Fixed issue where input appeared as null in Langfuse dashboard for GoogleLLMService
- Added fallback to use adapter's get_messages_for_logging() for universal LLMContext
- Ensures proper message format conversion for Google/Gemini services
- Handles system message conversion to system_instruction format
- Also fixes serialization of empty message lists ([] now serializes correctly)
This fix ensures Langfuse tracing works correctly for Google services using
both OpenAILLMContext/GoogleLLMContext and the universal LLMContext.
* Add unit tests for Langfuse tracing with GoogleLLMService
- Test that tracing correctly captures messages with universal LLMContext
- Test that empty message lists are properly serialized
- Test that adapter's get_messages_for_logging is used instead of context method
- All tests verify that input is correctly added to Langfuse spans
* Fix test mocking to patch opentelemetry.trace.get_tracer correctly
The tests were failing in CI because they were trying to patch
'pipecat.utils.tracing.service_decorators.trace' which doesn't exist as
an attribute. The trace module is imported from opentelemetry, so we need
to patch 'opentelemetry.trace.get_tracer' instead.
* Skip tracing tests when opentelemetry is not installed
The tracing dependencies (opentelemetry) are optional in Pipecat and not
installed in the CI environment. Added a skipif marker to skip these tests
when opentelemetry is not available, preventing CI failures while still
allowing the tests to run when tracing dependencies are installed locally.
* Install tracing dependencies in GitHub Actions CI
Instead of skipping the tracing tests, install the 'tracing' extra
(opentelemetry) in the CI environment so the tests can run properly.
Removed the skipif condition from the tests since opentelemetry will
now be available in CI.
* Use the context type to determine which messages to use, fix tool_count and tools (#3032)
---------
Co-authored-by: Mark Backman <mark@daily.co>
