diff --git a/CHANGELOG.md b/CHANGELOG.md index 24734e799..aaaafd512 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,156 +7,235 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -### Fixed - -- Fixed an issue in `ElevenLabsRealtimeSTTService` where dynamic language updates were not working. - ### Added - Added optional speaking rate control to `InworldTTSService`. -- Introduced a new `AggregatedTextFrame` type to support passing text along with an - `aggregated_by` field to describe the type of text included. `TTSTextFrame`s now - inherit from `AggregatedTextFrame`. With this inheritance, an observer can watch for - `AggregatedTextFrame`s to accumlate the perceived output and determine whether or not - the text was spoken based on if that frame is also a `TTSTextFrame`. +- Introduced a new `AggregatedTextFrame` type to support passing text along with + an `aggregated_by` field to describe the type of text + included. `TTSTextFrame`s now inherit from `AggregatedTextFrame`. With this + inheritance, an observer can watch for `AggregatedTextFrame`s to accumlate the + perceived output and determine whether or not the text was spoken based on if + that frame is also a `TTSTextFrame`. - With this frame, the llm token stream can be transformed into custom composable - chunks, allowing for aggregation outside the TTS service. This makes it possible to - listen for or handle those aggregations and sets the stage for doing things like - composing a best effort of the perceived llm output in a more digestable form and - to do so whether or not it is processed by a TTS or if even a TTS exists. + With this frame, the llm token stream can be transformed into custom + composable chunks, allowing for aggregation outside the TTS service. This + makes it possible to listen for or handle those aggregations and sets the + stage for doing things like composing a best effort of the perceived llm + output in a more digestable form and to do so whether or not it is processed + by a TTS or if even a TTS exists. -- Introduced `LLMTextProcessor`: A new processor meant to allow customization for how - LLMTextFrames should be aggregated and considered. It's purpose is to turn - `LLMTextFrame`s into `AggregatedTextFrame`s. By default, a TTSService will still - aggregate `LLMTextFrame`s by sentence for the service to consume. However, if you - wish to override how the llm text is aggregated, you should no longer override the - TTS's internal text_aggregator, but instead, insert this processor between your LLM - and TTS in the pipeline. +- Introduced `LLMTextProcessor`: A new processor meant to allow customization + for how LLMTextFrames should be aggregated and considered. It's purpose is to + turn `LLMTextFrame`s into `AggregatedTextFrame`s. By default, a TTSService + will still aggregate `LLMTextFrame`s by sentence for the service to + consume. However, if you wish to override how the llm text is aggregated, you + should no longer override the TTS's internal text_aggregator, but instead, + insert this processor between your LLM and TTS in the pipeline. - New `bot-output` RTVI message to represent what the bot actually "says". - - The `RTVIObserver` now emits `bot-output` messages based off the new `AggregatedTextFrame`s - (`bot-tts-text` and `bot-llm-text` are still supported and generated, but `bot-transcript` is - now deprecated in lieu of this new, more thorough, message). + + - The `RTVIObserver` now emits `bot-output` messages based off the new + `AggregatedTextFrame`s (`bot-tts-text` and `bot-llm-text` are still + supported and generated, but `bot-transcript` is now deprecated in lieu of + this new, more thorough, message). + - The new `RTVIBotOutputMessage` includes the fields: + - `spoken`: A boolean indicating whether the text was spoken by TTS - - `aggregated_by`: A string representing how the text was aggregated ("sentence", "word", - "my custom aggregation") - - Introduced new fields to `RTVIObserver` to support the new `bot-output` messaging: - - `bot_output_enabled`: Defaults to True. Set to false to disable bot-output messages. - - `skip_aggregator_types`: Defaults to `None`. Set to a list of strings that match - aggregation types that should not be included in bot-output messages. (Ex. `credit_card`) - - Introduced new methods, `add_text_transformer()` and `remove_text_transformer()`, to - `RTVIObserver` to support providing (and subsequently removing) callbacks for various types of - aggregations (or all aggregations with `*`) that can modify the text before being sent as a - `bot-output` or `tts-text` message. (Think obscuring the credit card or inserting extra detail - the client might want that the context doesn't need.) + + - `aggregated_by`: A string representing how the text was aggregated + ("sentence", "word", "my custom aggregation") + + - Introduced new fields to `RTVIObserver` to support the new `bot-output` + messaging: + + - `bot_output_enabled`: Defaults to True. Set to false to disable bot-output + messages. + + - `skip_aggregator_types`: Defaults to `None`. Set to a list of strings that + match aggregation types that should not be included in bot-output + messages. (Ex. `credit_card`) + + - Introduced new methods, `add_text_transformer()` and + `remove_text_transformer()`, to `RTVIObserver` to support providing (and + subsequently removing) callbacks for various types of aggregations (or all + aggregations with `*`) that can modify the text before being sent as a + `bot-output` or `tts-text` message. (Think obscuring the credit card or + inserting extra detail the client might want that the context doesn't need.) + +- In `MiniMaxHttpTTSService`: + + - Added support for speech-2.6-hd and speech-2.6-turbo models + + - Added languages: Afrikaans, Bulgarian, Catalan, Danish, Persian, Filipino, + Hebrew, Croatian, Hungarian, Malay, Norwegian, Nynorsk, Slovak, Slovenian, + Swedish, and Tamil + + - Added new emotions: calm and fluent ### Changed - Updated `daily-python` to 0.22.0. - `BaseTextAggregator` changes: - Modified the BaseTextAggregator type so that when text gets aggregated, metadata can - be associated with it. Currently, that just means a `type`, so that the aggregation - can be classified or described. Changes made to support this: - - ⚠️ IMPORTANT: Aggregators are now expected to strip leading/trailing white space - characters before returning their aggregation from `aggregation()` or `.text`. This - way all aggregators have a consistent contract allowing downstream use to know how - to stitch aggregations back together. - - Introduced a new `Aggregation` dataclass to represent both the aggregated `text` and - a string identifying the `type` of aggregation (ex. "sentence", "word", "my custom - aggregation") - - ⚠️ Breaking change: `BaseTextAggregator.text` now returns an `Aggregation` (instead of `str`). - To update: `aggregated_text = myAggregator.text` -> `aggregated_text = myAggregator.text.text` - - ⚠️ Breaking change: `BaseTextAggregator.aggregate()` now returns `Optional[Aggregation]` - (instead of `Optional[str]`). To update: - ``` - aggregation = myAggregator.aggregate(text) - if (aggregation): - print(f"successfully aggregated text: {aggregation.text}") // instead of {aggregation} - ``` - - `SimpleTextAggregator`, `SkipTagsAggregator`, `PatternPairAggregator` updated to - produce/consume `Aggregation` objects. + + Modified the BaseTextAggregator type so that when text gets aggregated, + metadata can be associated with it. Currently, that just means a `type`, so + that the aggregation can be classified or described. Changes made to support + this: + + - ⚠️ IMPORTANT: Aggregators are now expected to strip leading/trailing white + space characters before returning their aggregation from `aggregation()` or + `.text`. This way all aggregators have a consistent contract allowing + downstream use to know how to stitch aggregations back together. + + - Introduced a new `Aggregation` dataclass to represent both the aggregated + `text` and a string identifying the `type` of aggregation (ex. "sentence", + "word", "my custom aggregation") + + - ⚠️ Breaking change: `BaseTextAggregator.text` now returns an `Aggregation` + (instead of `str`). + + Before: + + ```python + aggregated_text = myAggregator.text + ``` + + Now: + + ```python + aggregated_text = myAggregator.text.text + ``` + + - ⚠️ Breaking change: `BaseTextAggregator.aggregate()` now returns + `Optional[Aggregation]` (instead of `Optional[str]`). + + Before: + + ```python + aggregation = myAggregator.aggregate(text) + print(f"successfully aggregated text: {aggregation}") + ``` + + Now: + + ```python + aggregation = myAggregator.aggregate(text) + if aggregation: + print(f"successfully aggregated text: {aggregation.text}") + ``` + + - `SimpleTextAggregator`, `SkipTagsAggregator`, `PatternPairAggregator` + updated to produce/consume `Aggregation` objects. + - All uses of the above Aggregators have been updated accordingly. -- Augmented the `PatternPairAggregator` so that matched patterns can be treated as their own - aggregation, taking advantage of the new. To that end: - - Introduced a new, preferred version of `add_pattern` to support a new option for treating a - match as a separate aggregation returned from `aggregate()`. This replaces the now - deprecated `add_pattern_pair` method and you provide a `MatchAction` in lieu of the `remove_match` field. - - `MatchAction` enum: `REMOVE`, `KEEP`, `AGGREGATE`, allowing customization for how - a match should be handled. - - `REMOVE`: The text along with its delimiters will be removed from the streaming text. - Sentence aggregation will continue on as if this text did not exist. - - `KEEP`: The delimiters will be removed, but the content between them will be kept. - Sentence aggregation will continue on with the internal text included. - - `AGGREGATE`: The delimiters will be removed and the content between will be treated - as a separate aggregation. Any text before the start of the pattern will be - returned early, whether or not a complete sentence was found. Then the pattern - will be returned. Then the aggregation will continue on sentence matching after - the closing delimiter is found. The content between the delimiters is not - aggregated by sentence. It is aggregated as one single block of text. - - `PatternMatch` now extends `Aggregation` and provides richer info to handlers. - - ⚠️ Breaking change: The `PatternMatch` type returned to handlers registered via `on_pattern_match` - has been updated to subclass from the new `Aggregation` type, which means that `content` - has been replaced with `text` and `pattern_id` has been replaced with `type`: - ``` - async dev on_match_tag(match: PatternMatch): - pattern = match.type # instead of match.pattern_id - text = match.text # instead of match.content - ``` +- Augmented the `PatternPairAggregator` so that matched patterns can be treated + as their own aggregation, taking advantage of the new. To that end: -- `TextFrame` now includes the field `append_to_context` to support setting whether or not the - encompassing text should be added to the LLM context (by the LLM assistant aggregator). It - defaults to `True`. + - Introduced a new, preferred version of `add_pattern` to support a new option + for treating a match as a separate aggregation returned from + `aggregate()`. This replaces the now deprecated `add_pattern_pair` method + and you provide a `MatchAction` in lieu of the `remove_match` field. + + - `MatchAction` enum: `REMOVE`, `KEEP`, `AGGREGATE`, allowing customization + for how a match should be handled. + + - `REMOVE`: The text along with its delimiters will be removed from the + streaming text. Sentence aggregation will continue on as if this text + did not exist. + + - `KEEP`: The delimiters will be removed, but the content between them + will be kept. Sentence aggregation will continue on with the internal + text included. + + - `AGGREGATE`: The delimiters will be removed and the content between will + be treated as a separate aggregation. Any text before the start of the + pattern will be returned early, whether or not a complete sentence was + found. Then the pattern will be returned. Then the aggregation will + continue on sentence matching after the closing delimiter is found. The + content between the delimiters is not aggregated by sentence. It is + aggregated as one single block of text. + + - `PatternMatch` now extends `Aggregation` and provides richer info to + handlers. + + - ⚠️ Breaking change: The `PatternMatch` type returned to handlers registered + via `on_pattern_match` has been updated to subclass from the new + `Aggregation` type, which means that `content` has been replaced with + `text` and `pattern_id` has been replaced with `type`: + + ```python + async dev on_match_tag(match: PatternMatch): + pattern = match.type # instead of match.pattern_id + text = match.text # instead of match.content + ``` + +- `TextFrame` now includes the field `append_to_context` to support setting + whether or not the encompassing text should be added to the LLM context (by + the LLM assistant aggregator). It defaults to `True`. - `TTSService` base class updates: - - `TTSService`s now accept a new `skip_aggregator_types` to avoid speaking certain aggregation - types (now determined/returned by the aggregator) - - Introduced the ability to do a just-in-time transform of text before it gets sent to the - TTS service via callbacks you can set up via a new init field, `text_transforms` or a new - method `add_text_transformer()`. This makes it possible to do things like introduce - TTS-specific tags for spelling or emotion or change the pronunciation of something on the - fly. `remove_text_transformer` has also been added to support removing a registered - transform callback. - - TTS services push `AggregatedTextFrame` in addition to `TTSTextFrame`s when either an - aggregation occurs that should not be spoken or when the TTS service supports word-by-word - timestamping. In the latter case, the `TTSService` preliminarily generates an - `AggregatedTextFrame`, aggregated by sentence to generate the full sentence content as early - as possible. + + - `TTSService`s now accept a new `skip_aggregator_types` to avoid speaking + certain aggregation types (now determined/returned by the aggregator) + + - Introduced the ability to do a just-in-time transform of text before it gets + sent to the TTS service via callbacks you can set up via a new init field, + `text_transforms` or a new method `add_text_transformer()`. This makes it + possible to do things like introduce TTS-specific tags for spelling or + emotion or change the pronunciation of something on the + fly. `remove_text_transformer` has also been added to support removing a + registered transform callback. + + - TTS services push `AggregatedTextFrame` in addition to `TTSTextFrame`s when + either an aggregation occurs that should not be spoken or when the TTS + service supports word-by-word timestamping. In the latter case, the + `TTSService` preliminarily generates an `AggregatedTextFrame`, aggregated by + sentence to generate the full sentence content as early as possible. - Updated `CartesiaTTSService`: - - Modified use of custom default text_aggregator to avoid deprecation warnings and push users - towards use of transformers or the `LLMTextProcessor` - - Added convenience methods for taking advantage of Cartesia's SSML tags: spell, emotion, - pauses, volume, and speed. + + - Modified use of custom default text_aggregator to avoid deprecation warnings + and push users towards use of transformers or the `LLMTextProcessor` + + - Added convenience methods for taking advantage of Cartesia's SSML tags: + spell, emotion, pauses, volume, and speed. - Updated `RimeTTSService`: - - Modified use of custom default text_aggregator to avoid deprecation warnings and push users - towards use of transformers or the `LLMTextProcessor` - - Added convenience methods for taking advantage of Rime's customization options: spell, - pauses, pronunciations, and inline speed control. + + - Modified use of custom default text_aggregator to avoid deprecation warnings + and push users towards use of transformers or the `LLMTextProcessor` + + - Added convenience methods for taking advantage of Rime's customization + options: spell, pauses, pronunciations, and inline speed control. ### Deprecated - The TTS constructor field, `text_aggregator` is deprecated in favor of the new - `LLMTextProcessor`. TTSServices still have an internal aggregator for support of default - behavior, but if you want to override the aggregation behavior, you should use the new - processor. + `LLMTextProcessor`. TTSServices still have an internal aggregator for support + of default behavior, but if you want to override the aggregation behavior, you + should use the new processor. -- The RTVI `bot-transcription` event is deprecated in favor of the new `bot-output` - message which is the canonical representation of bot output (spoken or not). The code - still emits a transcription message for backwards compatibility while transition occurs. +- The RTVI `bot-transcription` event is deprecated in favor of the new + `bot-output` message which is the canonical representation of bot output + (spoken or not). The code still emits a transcription message for backwards + compatibility while transition occurs. -- Deprecated `add_pattern_pair` in the `PatternPairAggregator` which takes a `pattern_id` - and `remove_match` field in favor of the new `add_pattern` method which takes a `type` and an - `action` +- Deprecated `add_pattern_pair` in the `PatternPairAggregator` which takes a + `pattern_id` and `remove_match` field in favor of the new `add_pattern` method + which takes a `type` and an `action` +- `english_normalization` input parameter for `MiniMaxHttpTTSService` is + deprecated, use `test_normalization` instead. ### Fixed +- Fixed an issue in `ElevenLabsRealtimeSTTService` where dynamic language + updates were not working. + - Fixed `InworldTTSService` audio config payload to use camelCase keys expected by the Inworld API. @@ -218,20 +297,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Updated language mappings for the Google and Gemini TTS services to match official documentation. -- In `MiniMaxHttpTTSService`: --- Added support for speech-2.6-hd and speech-2.6-turbo models --- Added languages: Afrikaans, Bulgarian, Catalan, Danish, Persian, Filipino, Hebrew, -Croatian, Hungarian, Malay, Norwegian, Nynorsk, Slovak, Slovenian, Swedish, and Tamil --- Added new emotions: calm and fluent - ### Deprecated - The `api_key` parameter in `GeminiTTSService` is deprecated. Use `credentials` or `credentials_path` instead for Google Cloud authentication. -- `english_normalization` input parameter for `MiniMaxHttpTTSService` is deprecated, -use `test_normalization` instead. - ### Fixed - Fixed a `SimliVideoService` connection issue.