diff --git a/CHANGELOG.md b/CHANGELOG.md index 7cda253c7..ca85c1aa2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,44 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 services that subclass `TTSService` can indicate whether the text in the `TTSTextFrame`s they push already contain any necessary inter-frame spaces. +- New bot-output RTVI message to represent what the bot actually "says". + - RTVIBotOutputMessage / RTVIBotOutputMessageData — includes: + - spoken: bool — whether the text was spoken by TTS + - aggregated_by: Optional[str|\"word\"|\"sentence\"] — how the text was aggregated + - RTVIObserver now emits bot-output messages (bot-tts-text and bot-llm-text are still + supported and generated. bot-transcript is now deprecated in lieu of this new, more + thorough, message). +- Introduced new `AggregatedLLMTextFrame` type to support representing effective llm + output whether or not it is processed by the TTS. This new frame type includes the + field `aggregated_by` to represent the conceptual formaty by which the given text + is aggregated. `TTSTextFrame`s now inherit from `AggregatedLLMTextFrame`. + +- Updated the base aggregator type: + - Introduced a new `Aggregation` dataclass to represent both the aggregated `text` and + a string identifying the `type` of aggregation (ex. "sentence", "word", "my custom + aggregation") + - `BaseTextAggregator.text` now returns an `Aggregation` (instead of `str`). + - `BaseTextAggregator.aggregate()` now returns `Optional[Aggregation]` + (instead of `Optional[str]`) + - `SimpleTextAggregator`, `SkipTagsAggregator`, `PatternPairAggregator` updated to + produce/consume `Aggregation` objects. + +- Augmented the `PatterPairAggregator`: + - `PatternPairAggregator` now supports `type` and `action` per pattern. + - New `MatchAction` enum: `REMOVE`, `KEEP`, `AGGREGATE`, allowing customization for how + a match should be handled. + - `REMOVE`: The text along with its delimiters will be removed from the streaming text. + Sentence aggregation will continue on as if this text did not exist. + - `KEEP`: The delimiters will be removed, but the content between them will be kept. + Sentence aggregation will continue on with the internal text included. + - `AGGREGATE`: The delimiters will be removed and the content between will be treated + as a separate aggregation. Any text before the start of the pattern will be + returned early, whether or not a complete sentence was found. Then the pattern + will be returned. Then the aggregation will continue on sentence matching after + the closing delimeter is found. The content between the delimeters is not + aggregated by sentence. It is aggregated as one single block of text. + - `PatternMatch` now extends `Aggregation` and provides richer info to handlers. + ### Changed - Updated all STT and TTS services to use consistent error handling pattern with @@ -33,11 +71,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Updated language mappings for the Google and Gemini TTS services to match official documentation. +- TTS flow respects aggregation metadata + - `TTSService` accepts a new `skip_aggregator_types` to avoid speaking certain aggregation types (as + now determined/returned by the aggregator) + - TTS services push `AggregatedLLMTextFrame` in addition to `TTSTextFrame`s when either an aggregation + occurs that should not be spoken or when the TTS service supports word-by-word timestamping. In the + latter case, the `TTSService` preliminarily generates an `AggregatedLLMTextFrame`, aggregated by sentence + to generate the full sentence content as early as possible. + ### Deprecated - The `api_key` parameter in `GeminiTTSService` is deprecated. Use `credentials` or `credentials_path` instead for Google Cloud authentication. +- The RTVI `bot-transcription` event is deprecated in favor of the new `bot-output` message which is the canonical representation of bot output (spoken or not). The code still emits a transcription message for backwards compatibility while transition occurs. + ### Fixed - Fixed subtle issue of assistant context messages ending up with double spaces diff --git a/src/pipecat/utils/text/pattern_pair_aggregator.py b/src/pipecat/utils/text/pattern_pair_aggregator.py index 66cf4da9f..0551f7cc3 100644 --- a/src/pipecat/utils/text/pattern_pair_aggregator.py +++ b/src/pipecat/utils/text/pattern_pair_aggregator.py @@ -8,7 +8,7 @@ This module provides an aggregator that identifies and processes content between pattern pairs (like XML tags or custom delimiters) in streaming text, with -support for custom handlers and configurable pattern removal. +support for custom handlers and configurable actions for when a pattern is found. """ import re @@ -72,13 +72,18 @@ class PatternPairAggregator(BaseTextAggregator): This aggregator buffers text until it can identify complete pattern pairs (defined by start and end patterns), processes the content between these - patterns using registered handlers, and returns text at sentence boundaries. - It's particularly useful for processing structured content in streaming text, - such as XML tags, markdown formatting, or custom delimiters. + patterns using registered handlers. By default, its aggregation method + returns text at sentence boundaries, and remove the content found between + any matched patterns. However, matched patterns can also be configured to + returned as a separate aggregation object containing the content between + their start and end patterns or left in, so that only the delimiters are + removed and a callback can be triggered. + + This aggregator is particularly useful for processing structured content in + streaming text, such as XML tags, markdown formatting, or custom delimiters. The aggregator ensures that patterns spanning multiple text chunks are - correctly identified and handles cases where patterns contain sentence - boundaries. + correctly identified. """ def __init__(self, **kwargs):