From a6202c4d1a7c7a7f12f6e382d267efeef2c67853 Mon Sep 17 00:00:00 2001 From: mattie ruth backman Date: Thu, 20 Nov 2025 14:58:53 -0500 Subject: [PATCH] Fixed CHANGELOG post rebase --- CHANGELOG.md | 254 +++++++++--------- .../utils/text/base_text_aggregator.py | 2 +- 2 files changed, 132 insertions(+), 124 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7ee108c11..c9208b8d9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,34 +11,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Added optional speaking rate control to `InworldTTSService`. -### Changed - -- Updated `daily-python` to 0.22.0. - -### Fixed - -- Fixed `InworldTTSService` audio config payload to use camelCase keys expected - by the Inworld API. - -## [0.0.95] - 2025-11-18 - -### Added - -- Added ai-coustics integrated VAD (`AICVADAnalyzer`) with `AICFilter` factory and - example wiring; leverages the enhancement model for robust detection with no - ONNX dependency or added processing complexity. - -- Added a watchdog to `DeepgramFluxSTTService` to prevent dangling tasks in case the - user was speaking and we stop receiving audio. - -- Introduced a minimum confidence parameter in `DeepgramFluxSTTService` to avoid - generating transcriptions below a defined threshold. - -- Added `ElevenLabsRealtimeSTTService` which implements the Realtime STT - service from ElevenLabs. - -- Added word-level timestamps support to Hume TTS service - - Introduced a new `AggregatedTextFrame` type to support passing text along with an `aggregated_by` field to describe the type of text included. `TTSTextFrame`s now inherit from `AggregatedTextFrame`. With this inheritance, an observer can watch for @@ -71,6 +43,137 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `bot_output_enabled`: Defaults to True. Set to false to disable bot-output messages. - `skip_aggregator_types`: Defaults to `None`. Set to a list of strings that match aggregation types that should not be included in bot-output messages. (Ex. `credit_card`) + - Introduced new methods, `add_text_transformer()` and `remove_text_transformer()`, to + `RTVIObserver` to support providing (and subsequently removing) callbacks for various types of + aggregations (or all aggregations with `*`) that can modify the text before being sent as a + `bot-output` or `tts-text` message. (Think obscuring the credit card or inserting extra detail + the client might want that the context doesn't need.) + +### Changed + +- Updated `daily-python` to 0.22.0. + +- `BaseTextAggregator` changes: + Modified the BaseTextAggregator type so that when text gets aggregated, metadata can + be associated with it. Currently, that just means a `type`, so that the aggregation + can be classified or described. Changes made to support this: + - ⚠️ IMPORTANT: Aggregators are now expected to strip leading/trailing white space + characters before returning their aggregation from `aggregation()` or `.text`. This + way all aggregators have a consistent contract allowing downstream use to know how + to stitch aggregations back together. + - Introduced a new `Aggregation` dataclass to represent both the aggregated `text` and + a string identifying the `type` of aggregation (ex. "sentence", "word", "my custom + aggregation") + - ⚠️ Breaking change: `BaseTextAggregator.text` now returns an `Aggregation` (instead of `str`). + To update: `aggregated_text = myAggregator.text` -> `aggregated_text = myAggregator.text.text` + - ⚠️ Breaking change: `BaseTextAggregator.aggregate()` now returns `Optional[Aggregation]` + (instead of `Optional[str]`). To update: + ``` + aggregation = myAggregator.aggregate(text) + if (aggregation): + print(f"successfully aggregated text: {aggregation.text}") // instead of {aggregation} + ``` + - `SimpleTextAggregator`, `SkipTagsAggregator`, `PatternPairAggregator` updated to + produce/consume `Aggregation` objects. + - All uses of the above Aggregators have been updated accordingly. + +- Augmented the `PatternPairAggregator` so that matched patterns can be treated as their own + aggregation, taking advantage of the new. To that end: + - Introduced a new, preferred version of `add_pattern` to support a new option for treating a + match as a separate aggregation returned from `aggregate()`. This replaces the now + deprecated `add_pattern_pair` method and you provide a `MatchAction` in lieu of the `remove_match` field. + - `MatchAction` enum: `REMOVE`, `KEEP`, `AGGREGATE`, allowing customization for how + a match should be handled. + - `REMOVE`: The text along with its delimiters will be removed from the streaming text. + Sentence aggregation will continue on as if this text did not exist. + - `KEEP`: The delimiters will be removed, but the content between them will be kept. + Sentence aggregation will continue on with the internal text included. + - `AGGREGATE`: The delimiters will be removed and the content between will be treated + as a separate aggregation. Any text before the start of the pattern will be + returned early, whether or not a complete sentence was found. Then the pattern + will be returned. Then the aggregation will continue on sentence matching after + the closing delimiter is found. The content between the delimiters is not + aggregated by sentence. It is aggregated as one single block of text. + - `PatternMatch` now extends `Aggregation` and provides richer info to handlers. + - ⚠️ Breaking change: The `PatternMatch` type returned to handlers registered via `on_pattern_match` + has been updated to subclass from the new `Aggregation` type, which means that `content` + has been replaced with `text` and `pattern_id` has been replaced with `type`: + ``` + async dev on_match_tag(match: PatternMatch): + pattern = match.type # instead of match.pattern_id + text = match.text # instead of match.content + ``` + +- `TextFrame` now includes the field `append_to_context` to support setting whether or not the + encompassing text should be added to the LLM context (by the LLM assistant aggregator). It + defaults to `True`. + +- `TTSService` base class updates: + - `TTSService`s now accept a new `skip_aggregator_types` to avoid speaking certain aggregation + types (now determined/returned by the aggregator) + - Introduced the ability to do a just-in-time transform of text before it gets sent to the + TTS service via callbacks you can set up via a new init field, `text_transforms` or a new + method `add_text_transformer()`. This makes it possible to do things like introduce + TTS-specific tags for spelling or emotion or change the pronunciation of something on the + fly. `remove_text_transformer` has also been added to support removing a registered + transform callback. + - TTS services push `AggregatedTextFrame` in addition to `TTSTextFrame`s when either an + aggregation occurs that should not be spoken or when the TTS service supports word-by-word + timestamping. In the latter case, the `TTSService` preliminarily generates an + `AggregatedTextFrame`, aggregated by sentence to generate the full sentence content as early + as possible. + +- Updated `CartesiaTTSService`: + - Modified use of custom default text_aggregator to avoid deprecation warnings and push users + towards use of transformers or the `LLMTextProcessor` + - Added convenience methods for taking advantage of Cartesia's SSML tags: spell, emotion, + pauses, volume, and speed. + +- Updated `RimeTTSService`: + - Modified use of custom default text_aggregator to avoid deprecation warnings and push users + towards use of transformers or the `LLMTextProcessor` + - Added convenience methods for taking advantage of Rime's customization options: spell, + pauses, pronunciations, and inline speed control. + +### Deprecated + +- The TTS constructor field, `text_aggregator` is deprecated in favor of the new + `LLMTextProcessor`. TTSServices still have an internal aggregator for support of default + behavior, but if you want to override the aggregation behavior, you should use the new + processor. + +- The RTVI `bot-transcription` event is deprecated in favor of the new `bot-output` + message which is the canonical representation of bot output (spoken or not). The code + still emits a transcription message for backwards compatibility while transition occurs. + +- Deprecated `add_pattern_pair` in the `PatternPairAggregator` which takes a `pattern_id` + and `remove_match` field in favor of the new `add_pattern` method which takes a `type` and an + `action` + + +### Fixed + +- Fixed `InworldTTSService` audio config payload to use camelCase keys expected + by the Inworld API. + +## [0.0.95] - 2025-11-18 + +### Added + +- Added ai-coustics integrated VAD (`AICVADAnalyzer`) with `AICFilter` factory and + example wiring; leverages the enhancement model for robust detection with no + ONNX dependency or added processing complexity. + +- Added a watchdog to `DeepgramFluxSTTService` to prevent dangling tasks in case the + user was speaking and we stop receiving audio. + +- Introduced a minimum confidence parameter in `DeepgramFluxSTTService` to avoid + generating transcriptions below a defined threshold. + +- Added `ElevenLabsRealtimeSTTService` which implements the Realtime STT + service from ElevenLabs. + +- Added word-level timestamps support to Hume TTS service ### Changed @@ -117,88 +220,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 Croatian, Hungarian, Malay, Norwegian, Nynorsk, Slovak, Slovenian, Swedish, and Tamil -- Added new emotions: calm and fluent -- `BaseTextAggregator` changes: - Modified the BaseTextAggregator type so that when text gets aggregated, metadata can - be associated with it. Currently, that just means a `type`, so that the aggregation - can be classified or described. Changes made to support this: - - **IMPORTANT**: Aggregators are now expected to strip leading/trailing white space - characters before returning their aggregation from `aggregation()` or `.text`. This - way all aggregators have a consistent contract allowing downstream use to know how - to stitch aggregations back together. - - Introduced a new `Aggregation` dataclass to represent both the aggregated `text` and - a string identifying the `type` of aggregation (ex. "sentence", "word", "my custom - aggregation") - - **BREAKING**: `BaseTextAggregator.text` now returns an `Aggregation` (instead of `str`). - To update: `aggregated_text = myAggregator.text` -> `aggregated_text = myAggregator.text.text` - - **BREAKING**: `BaseTextAggregator.aggregate()` now returns `Optional[Aggregation]` - (instead of `Optional[str]`). To update: - ``` - aggregation = myAggregator.aggregate(text) - if (aggregation): - print(f"successfully aggregated text: {aggregation.text}") // instead of {aggregation} - ``` - - `SimpleTextAggregator`, `SkipTagsAggregator`, `PatternPairAggregator` updated to - produce/consume `Aggregation` objects. - - All uses of the above Aggregators have been updated accordingly. - -- Augmented the `PatternPairAggregator` so that matched patterns can be treated as their own - aggregation, taking advantage of the new. To that end: - - Introduced a new, preferred version of `add_pattern` to support a new option for treating a - match as a separate aggregation returned from `aggregate()`. This replaces the now - deprecated `add_pattern_pair` method and you provide a `MatchAction` in lieu of the `remove_match` field. - - `MatchAction` enum: `REMOVE`, `KEEP`, `AGGREGATE`, allowing customization for how - a match should be handled. - - `REMOVE`: The text along with its delimiters will be removed from the streaming text. - Sentence aggregation will continue on as if this text did not exist. - - `KEEP`: The delimiters will be removed, but the content between them will be kept. - Sentence aggregation will continue on with the internal text included. - - `AGGREGATE`: The delimiters will be removed and the content between will be treated - as a separate aggregation. Any text before the start of the pattern will be - returned early, whether or not a complete sentence was found. Then the pattern - will be returned. Then the aggregation will continue on sentence matching after - the closing delimiter is found. The content between the delimiters is not - aggregated by sentence. It is aggregated as one single block of text. - - `PatternMatch` now extends `Aggregation` and provides richer info to handlers. - - **BREAKING**: The `PatternMatch` type returned to handlers registered via `on_pattern_match` - has been updated to subclass from the new `Aggregation` type, which means that `content` - has been replaced with `text` and `pattern_id` has been replaced with `type`: - ``` - async dev on_match_tag(match: PatternMatch): - pattern = match.type # instead of match.pattern_id - text = match.text # instead of match.content - ``` - - - `TextFrame` now includes the field `append_to_context` to support setting whether or not the - encompassing text should be added to the LLM context (by the LLM assistant aggregator). It - defaults to `True`. - - - `TTSService` base class updates: - - `TTSService`s now accept a new `skip_aggregator_types` to avoid speaking certain aggregation - types (now determined/returned by the aggregator) - - Introduced the ability to do a just-in-time transform of text before it gets sent to the - TTS service via callbacks you can set up via a new init field, `text_transforms` or a new - method `add_text_transformer()`. This makes it possible to do things like introduce - TTS-specific tags for spelling or emotion or change the pronunciation of something on the - fly. `remove_text_transformer` has also been added to support removing a registered - transform callback. - - TTS services push `AggregatedTextFrame` in addition to `TTSTextFrame`s when either an - aggregation occurs that should not be spoken or when the TTS service supports word-by-word - timestamping. In the latter case, the `TTSService` preliminarily generates an - `AggregatedTextFrame`, aggregated by sentence to generate the full sentence content as early - as possible. - - - Updated `CartesiaTTSService`: - - Modified use of custom default text_aggregator to avoid deprecation warnings and push users - towards use of transformers or the `LLMTextProcessor` - - Added convenience methods for taking advantage of Cartesia's SSML tags: spell, emotion, - pauses, volume, and speed. - - - Updated `RimeTTSService`: - - Modified use of custom default text_aggregator to avoid deprecation warnings and push users - towards use of transformers or the `LLMTextProcessor` - - Added convenience methods for taking advantage of Rime's customization options: spell, - pauses, pronunciations, and inline speed control. - ### Deprecated - The `api_key` parameter in `GeminiTTSService` is deprecated. Use @@ -207,19 +228,6 @@ Croatian, Hungarian, Malay, Norwegian, Nynorsk, Slovak, Slovenian, Swedish, and - `english_normalization` input parameter for `MiniMaxHttpTTSService` is deprecated, use `test_normalization` instead. -- The TTS constructor field, `text_aggregator` is deprecated in favor of the new - `LLMTextProcessor`. TTSServices still have an internal aggregator for support of default - behavior, but if you want to override the aggregation behavior, you should use the new - processor. - -- The RTVI `bot-transcription` event is deprecated in favor of the new `bot-output` - message which is the canonical representation of bot output (spoken or not). The code - still emits a transcription message for backwards compatibility while transition occurs. - -- Deprecated `add_pattern_pair` in the `PatternPairAggregator` which takes a `pattern_id` - and `remove_match` field in favor of the new `add_pattern` method which takes a `type` and an - `action` - ### Fixed - Fixed a `SimliVideoService` connection issue. diff --git a/src/pipecat/utils/text/base_text_aggregator.py b/src/pipecat/utils/text/base_text_aggregator.py index 07cb6c097..5c39ce769 100644 --- a/src/pipecat/utils/text/base_text_aggregator.py +++ b/src/pipecat/utils/text/base_text_aggregator.py @@ -99,7 +99,7 @@ class BaseTextAggregator(ABC): format. Args: - text: The text to be aggregated + text: The text to be aggregated. Returns: An Aggregation object if ready for processing, or None if more