Compare commits
23 Commits
hush/ttsBu
...
bot-output
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e8640d84ae | ||
|
|
23e4e29999 | ||
|
|
713b488bb6 | ||
|
|
71b87fd420 | ||
|
|
3f269f9834 | ||
|
|
4c698777f3 | ||
|
|
5ca04ad741 | ||
|
|
9a3902a82c | ||
|
|
8ab0c92681 | ||
|
|
124f147a37 | ||
|
|
ed808a9246 | ||
|
|
e9de9daf8c | ||
|
|
82b9c4f0b6 | ||
|
|
5dfe20be91 | ||
|
|
0d2c5286fa | ||
|
|
29417ba44d | ||
|
|
bc6a9cac26 | ||
|
|
8a90decbc0 | ||
|
|
ccca6e8d81 | ||
|
|
e6dc1a510d | ||
|
|
69945c5e0d | ||
|
|
5c8635570d | ||
|
|
fe9aa3383e |
444
CHANGELOG.md
444
CHANGELOG.md
@@ -5,340 +5,98 @@ All notable changes to **Pipecat** will be documented in this file.
|
|||||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
||||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||||
|
|
||||||
## [0.0.96] - 2025-11-26 🦃 "Happy Thanksgiving!" 🦃
|
## [Unreleased]
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|
||||||
- Enhanced error handling across the framework:
|
|
||||||
|
|
||||||
- Added `on_error` callback to `FrameProcessor` for centralized error
|
|
||||||
handling.
|
|
||||||
|
|
||||||
- Renamed `push_error(error: ErrorFrame)` to `push_error_frame(error: ErrorFrame)`
|
|
||||||
for clarity.
|
|
||||||
|
|
||||||
- Added new `push_error` method for simplified error reporting:
|
|
||||||
|
|
||||||
```python
|
|
||||||
async def push_error(error_msg: str,
|
|
||||||
exception: Optional[Exception] = None,
|
|
||||||
fatal: bool = False)
|
|
||||||
```
|
|
||||||
|
|
||||||
- Standardized error logging by replacing `logger.exception` calls with
|
|
||||||
`logger.error` throughout the codebase.
|
|
||||||
|
|
||||||
- Added `cache_read_input_tokens`, `cache_creation_input_tokens` and
|
|
||||||
`reasoning_tokens` to OTel spans for LLM call
|
|
||||||
|
|
||||||
- Added `LiveKitRESTHelper` utility class for managing LiveKit rooms via REST API.
|
|
||||||
|
|
||||||
- Added `DeepgramSageMakerSTTService` which connects to a SageMaker hosted
|
|
||||||
Deepgram STT model. Added `07c-interruptible-deepgram-sagemaker.py`
|
|
||||||
foundational example.
|
|
||||||
|
|
||||||
- Added `SageMakerBidiClient` to connect to SageMaker hosted BiDi compatible
|
|
||||||
services.
|
|
||||||
|
|
||||||
- Added support for `include_timestamps` and `enable_logging` in
|
|
||||||
`ElevenLabsRealtimeSTTService`. When `include_timestamps` is enabled,
|
|
||||||
timestamp data is included in the `TranscriptionFrame`'s `result`
|
|
||||||
parameter.
|
|
||||||
|
|
||||||
- Added optional speaking rate control to `InworldTTSService`.
|
|
||||||
|
|
||||||
- Introduced a new `AggregatedTextFrame` type to support passing text along with
|
|
||||||
an `aggregated_by` field to describe the type of text
|
|
||||||
included. `TTSTextFrame`s now inherit from `AggregatedTextFrame`. With this
|
|
||||||
inheritance, an observer can watch for `AggregatedTextFrame`s to accumlate the
|
|
||||||
perceived output and determine whether or not the text was spoken based on if
|
|
||||||
that frame is also a `TTSTextFrame`.
|
|
||||||
|
|
||||||
With this frame, the llm token stream can be transformed into custom
|
|
||||||
composable chunks, allowing for aggregation outside the TTS service. This
|
|
||||||
makes it possible to listen for or handle those aggregations and sets the
|
|
||||||
stage for doing things like composing a best effort of the perceived llm
|
|
||||||
output in a more digestable form and to do so whether or not it is processed
|
|
||||||
by a TTS or if even a TTS exists.
|
|
||||||
|
|
||||||
- Introduced `LLMTextProcessor`: A new processor meant to allow customization
|
|
||||||
for how LLMTextFrames should be aggregated and considered. It's purpose is to
|
|
||||||
turn `LLMTextFrame`s into `AggregatedTextFrame`s. By default, a TTSService
|
|
||||||
will still aggregate `LLMTextFrame`s by sentence for the service to
|
|
||||||
consume. However, if you wish to override how the llm text is aggregated, you
|
|
||||||
should no longer override the TTS's internal text_aggregator, but instead,
|
|
||||||
insert this processor between your LLM and TTS in the pipeline.
|
|
||||||
|
|
||||||
- New `bot-output` RTVI message to represent what the bot actually "says".
|
|
||||||
|
|
||||||
- The `RTVIObserver` now emits `bot-output` messages based off the new
|
|
||||||
`AggregatedTextFrame`s (`bot-tts-text` and `bot-llm-text` are still
|
|
||||||
supported and generated, but `bot-transcript` is now deprecated in lieu of
|
|
||||||
this new, more thorough, message).
|
|
||||||
|
|
||||||
- The new `RTVIBotOutputMessage` includes the fields:
|
|
||||||
|
|
||||||
- `spoken`: A boolean indicating whether the text was spoken by TTS
|
|
||||||
|
|
||||||
- `aggregated_by`: A string representing how the text was aggregated
|
|
||||||
("sentence", "word", "my custom aggregation")
|
|
||||||
|
|
||||||
- Introduced new fields to `RTVIObserver` to support the new `bot-output`
|
|
||||||
messaging:
|
|
||||||
|
|
||||||
- `bot_output_enabled`: Defaults to True. Set to false to disable bot-output
|
|
||||||
messages.
|
|
||||||
|
|
||||||
- `skip_aggregator_types`: Defaults to `None`. Set to a list of strings that
|
|
||||||
match aggregation types that should not be included in bot-output
|
|
||||||
messages. (Ex. `credit_card`)
|
|
||||||
|
|
||||||
- Introduced new methods, `add_text_transformer()` and
|
|
||||||
`remove_text_transformer()`, to `RTVIObserver` to support providing (and
|
|
||||||
subsequently removing) callbacks for various types of aggregations (or all
|
|
||||||
aggregations with `*`) that can modify the text before being sent as a
|
|
||||||
`bot-output` or `tts-text` message. (Think obscuring the credit card or
|
|
||||||
inserting extra detail the client might want that the context doesn't need.)
|
|
||||||
|
|
||||||
- In `MiniMaxHttpTTSService`:
|
|
||||||
|
|
||||||
- Added support for speech-2.6-hd and speech-2.6-turbo models
|
|
||||||
|
|
||||||
- Added languages: Afrikaans, Bulgarian, Catalan, Danish, Persian, Filipino,
|
|
||||||
Hebrew, Croatian, Hungarian, Malay, Norwegian, Nynorsk, Slovak, Slovenian,
|
|
||||||
Swedish, and Tamil
|
|
||||||
|
|
||||||
- Added new emotions: calm and fluent
|
|
||||||
|
|
||||||
- Added `enable_logging` to `SimliVideoService` input parameters. It's disabled
|
|
||||||
by default.
|
|
||||||
|
|
||||||
### Changed
|
|
||||||
|
|
||||||
- Updated `FishAudioTTSService` default model to `s1`.
|
|
||||||
|
|
||||||
- Updated `DeepgramTTSService` to use Deepgram's TTS websocket API. ⚠️ This is
|
|
||||||
a potential breaking change, which only affects you if you're self-hosting
|
|
||||||
`DeepgramTTSService`. The new service uses Websockets and improves TTFB
|
|
||||||
latency.
|
|
||||||
|
|
||||||
- Updated `daily-python` to 0.22.0.
|
|
||||||
|
|
||||||
- `BaseTextAggregator` changes:
|
|
||||||
|
|
||||||
Modified the BaseTextAggregator type so that when text gets aggregated,
|
|
||||||
metadata can be associated with it. Currently, that just means a `type`, so
|
|
||||||
that the aggregation can be classified or described. Changes made to support
|
|
||||||
this:
|
|
||||||
|
|
||||||
- ⚠️ IMPORTANT: Aggregators are now expected to strip leading/trailing white
|
|
||||||
space characters before returning their aggregation from `aggregation()` or
|
|
||||||
`.text`. This way all aggregators have a consistent contract allowing
|
|
||||||
downstream use to know how to stitch aggregations back together.
|
|
||||||
|
|
||||||
- Introduced a new `Aggregation` dataclass to represent both the aggregated
|
|
||||||
`text` and a string identifying the `type` of aggregation (ex. "sentence",
|
|
||||||
"word", "my custom aggregation")
|
|
||||||
|
|
||||||
- ⚠️ Breaking change: `BaseTextAggregator.text` now returns an `Aggregation`
|
|
||||||
(instead of `str`).
|
|
||||||
|
|
||||||
Before:
|
|
||||||
|
|
||||||
```python
|
|
||||||
aggregated_text = myAggregator.text
|
|
||||||
```
|
|
||||||
|
|
||||||
Now:
|
|
||||||
|
|
||||||
```python
|
|
||||||
aggregated_text = myAggregator.text.text
|
|
||||||
```
|
|
||||||
|
|
||||||
- ⚠️ Breaking change: `BaseTextAggregator.aggregate()` now returns
|
|
||||||
`Optional[Aggregation]` (instead of `Optional[str]`).
|
|
||||||
|
|
||||||
Before:
|
|
||||||
|
|
||||||
```python
|
|
||||||
aggregation = myAggregator.aggregate(text)
|
|
||||||
print(f"successfully aggregated text: {aggregation}")
|
|
||||||
```
|
|
||||||
|
|
||||||
Now:
|
|
||||||
|
|
||||||
```python
|
|
||||||
aggregation = myAggregator.aggregate(text)
|
|
||||||
if aggregation:
|
|
||||||
print(f"successfully aggregated text: {aggregation.text}")
|
|
||||||
```
|
|
||||||
|
|
||||||
- `SimpleTextAggregator`, `SkipTagsAggregator`, `PatternPairAggregator`
|
|
||||||
updated to produce/consume `Aggregation` objects.
|
|
||||||
|
|
||||||
- All uses of the above Aggregators have been updated accordingly.
|
|
||||||
|
|
||||||
- Augmented the `PatternPairAggregator` so that matched patterns can be treated
|
|
||||||
as their own aggregation, taking advantage of the new. To that end:
|
|
||||||
|
|
||||||
- Introduced a new, preferred version of `add_pattern` to support a new option
|
|
||||||
for treating a match as a separate aggregation returned from
|
|
||||||
`aggregate()`. This replaces the now deprecated `add_pattern_pair` method
|
|
||||||
and you provide a `MatchAction` in lieu of the `remove_match` field.
|
|
||||||
|
|
||||||
- `MatchAction` enum: `REMOVE`, `KEEP`, `AGGREGATE`, allowing customization
|
|
||||||
for how a match should be handled.
|
|
||||||
|
|
||||||
- `REMOVE`: The text along with its delimiters will be removed from the
|
|
||||||
streaming text. Sentence aggregation will continue on as if this text
|
|
||||||
did not exist.
|
|
||||||
|
|
||||||
- `KEEP`: The delimiters will be removed, but the content between them
|
|
||||||
will be kept. Sentence aggregation will continue on with the internal
|
|
||||||
text included.
|
|
||||||
|
|
||||||
- `AGGREGATE`: The delimiters will be removed and the content between will
|
|
||||||
be treated as a separate aggregation. Any text before the start of the
|
|
||||||
pattern will be returned early, whether or not a complete sentence was
|
|
||||||
found. Then the pattern will be returned. Then the aggregation will
|
|
||||||
continue on sentence matching after the closing delimiter is found. The
|
|
||||||
content between the delimiters is not aggregated by sentence. It is
|
|
||||||
aggregated as one single block of text.
|
|
||||||
|
|
||||||
- `PatternMatch` now extends `Aggregation` and provides richer info to
|
|
||||||
handlers.
|
|
||||||
|
|
||||||
- ⚠️ Breaking change: The `PatternMatch` type returned to handlers registered
|
|
||||||
via `on_pattern_match` has been updated to subclass from the new
|
|
||||||
`Aggregation` type, which means that `content` has been replaced with
|
|
||||||
`text` and `pattern_id` has been replaced with `type`:
|
|
||||||
|
|
||||||
```python
|
|
||||||
async dev on_match_tag(match: PatternMatch):
|
|
||||||
pattern = match.type # instead of match.pattern_id
|
|
||||||
text = match.text # instead of match.content
|
|
||||||
```
|
|
||||||
|
|
||||||
- `TextFrame` now includes the field `append_to_context` to support setting
|
|
||||||
whether or not the encompassing text should be added to the LLM context (by
|
|
||||||
the LLM assistant aggregator). It defaults to `True`.
|
|
||||||
|
|
||||||
- `TTSService` base class updates:
|
|
||||||
|
|
||||||
- `TTSService`s now accept a new `skip_aggregator_types` to avoid speaking
|
|
||||||
certain aggregation types (now determined/returned by the aggregator)
|
|
||||||
|
|
||||||
- Introduced the ability to do a just-in-time transform of text before it gets
|
|
||||||
sent to the TTS service via callbacks you can set up via a new init field,
|
|
||||||
`text_transforms` or a new method `add_text_transformer()`. This makes it
|
|
||||||
possible to do things like introduce TTS-specific tags for spelling or
|
|
||||||
emotion or change the pronunciation of something on the
|
|
||||||
fly. `remove_text_transformer` has also been added to support removing a
|
|
||||||
registered transform callback.
|
|
||||||
|
|
||||||
- TTS services push `AggregatedTextFrame` in addition to `TTSTextFrame`s when
|
|
||||||
either an aggregation occurs that should not be spoken or when the TTS
|
|
||||||
service supports word-by-word timestamping. In the latter case, the
|
|
||||||
`TTSService` preliminarily generates an `AggregatedTextFrame`, aggregated by
|
|
||||||
sentence to generate the full sentence content as early as possible.
|
|
||||||
|
|
||||||
- Updated `CartesiaTTSService`:
|
|
||||||
|
|
||||||
- Modified use of custom default text_aggregator to avoid deprecation warnings
|
|
||||||
and push users towards use of transformers or the `LLMTextProcessor`
|
|
||||||
|
|
||||||
- Added convenience methods for taking advantage of Cartesia's SSML tags:
|
|
||||||
spell, emotion, pauses, volume, and speed.
|
|
||||||
|
|
||||||
- Updated `RimeTTSService`:
|
|
||||||
|
|
||||||
- Modified use of custom default text_aggregator to avoid deprecation warnings
|
|
||||||
and push users towards use of transformers or the `LLMTextProcessor`
|
|
||||||
|
|
||||||
- Added convenience methods for taking advantage of Rime's customization
|
|
||||||
options: spell, pauses, pronunciations, and inline speed control.
|
|
||||||
|
|
||||||
### Deprecated
|
|
||||||
|
|
||||||
- The TTS constructor field, `text_aggregator` is deprecated in favor of the new
|
|
||||||
`LLMTextProcessor`. TTSServices still have an internal aggregator for support
|
|
||||||
of default behavior, but if you want to override the aggregation behavior, you
|
|
||||||
should use the new processor.
|
|
||||||
|
|
||||||
- The RTVI `bot-transcription` event is deprecated in favor of the new
|
|
||||||
`bot-output` message which is the canonical representation of bot output
|
|
||||||
(spoken or not). The code still emits a transcription message for backwards
|
|
||||||
compatibility while transition occurs.
|
|
||||||
|
|
||||||
- Deprecated `add_pattern_pair` in the `PatternPairAggregator` which takes a
|
|
||||||
`pattern_id` and `remove_match` field in favor of the new `add_pattern` method
|
|
||||||
which takes a `type` and an `action`
|
|
||||||
|
|
||||||
- `english_normalization` input parameter for `MiniMaxHttpTTSService` is
|
|
||||||
deprecated, use `test_normalization` instead.
|
|
||||||
|
|
||||||
### Fixed
|
|
||||||
|
|
||||||
- Fixed an issue in `AWSBedrockLLMService` where the `aws_region` arg was
|
|
||||||
always set to `us-east-1`.
|
|
||||||
|
|
||||||
- Fixed an issue with `DeepgramFluxSTTService` where it sometimes failed to reconnect.
|
|
||||||
|
|
||||||
- Fixed an issue in `ElevenLabsRealtimeSTTService` where dynamic language
|
|
||||||
updates were not working.
|
|
||||||
|
|
||||||
- Fixed an issue in `ElevenLabsRealtimeSTTService` where setting the sample
|
|
||||||
rate would result in transcripts failing.
|
|
||||||
|
|
||||||
- Fixed `InworldTTSService` audio config payload to use camelCase keys expected
|
|
||||||
by the Inworld API.
|
|
||||||
|
|
||||||
## [0.0.95] - 2025-11-18
|
|
||||||
|
|
||||||
### Added
|
|
||||||
|
|
||||||
- Added ai-coustics integrated VAD (`AICVADAnalyzer`) with `AICFilter` factory and
|
|
||||||
example wiring; leverages the enhancement model for robust detection with no
|
|
||||||
ONNX dependency or added processing complexity.
|
|
||||||
|
|
||||||
- Added a watchdog to `DeepgramFluxSTTService` to prevent dangling tasks in case the
|
|
||||||
user was speaking and we stop receiving audio.
|
|
||||||
|
|
||||||
- Introduced a minimum confidence parameter in `DeepgramFluxSTTService` to avoid
|
|
||||||
generating transcriptions below a defined threshold.
|
|
||||||
|
|
||||||
- Added `ElevenLabsRealtimeSTTService` which implements the Realtime STT
|
- Added `ElevenLabsRealtimeSTTService` which implements the Realtime STT
|
||||||
service from ElevenLabs.
|
service from ElevenLabs.
|
||||||
|
|
||||||
- Added word-level timestamps support to Hume TTS service
|
- Added a `TTSService.includes_inter_frame_spaces` property getter, so that TTS
|
||||||
|
services that subclass `TTSService` can indicate whether the text in the
|
||||||
|
`TTSTextFrame`s they push already contain any necessary inter-frame spaces.
|
||||||
|
|
||||||
|
- Introduced new `AggregatedTextFrame` type to support representing a best effort of
|
||||||
|
the perceived llm output whether or not it is processed by the TTS. This new frame
|
||||||
|
type includes the field `aggregated_by` to represent the conceptual format by which
|
||||||
|
the given text is aggregated. `TTSTextFrame`s now inherit from `AggregatedTextFrame`.
|
||||||
|
With this inheritance, an observer can watch for `AggregatedTextFrame`s to accumlate
|
||||||
|
the perceived output and determine whether or not the text was spoken based on if that
|
||||||
|
frame is also a `TTSTextFrame`. (See bullet below on new `bot-output` which takes
|
||||||
|
advantage of this)
|
||||||
|
|
||||||
|
- Introduced `LLMTextProcessor`: A new processor meant to allow customization for how
|
||||||
|
LLMTextFrames should be aggregated and considered. It's purpose is to turn
|
||||||
|
`LLMTextFrame`s into `AggregatedTextFrame`s. By default, a TTSService will still
|
||||||
|
aggregate `LLMTextFrame`s by sentence for the service to consume. However, if you
|
||||||
|
wish to override how the llm text is aggregated, you should no longer override the
|
||||||
|
TTS's internal aggregator, but instead, insert this processor between your LLM and
|
||||||
|
TTS in the pipeline.
|
||||||
|
|
||||||
|
- New `bot-output` RTVI message to represent what the bot actually "says".
|
||||||
|
- The `RTVIObserver` now emits `bot-output` messages based off the new `AggregatedTextFrame`s
|
||||||
|
(`bot-tts-text` and `bot-llm-text` are still supported and generated, but `bot-transcript` is
|
||||||
|
now deprecated in lieu of this new, more thorough, message).
|
||||||
|
- The new `RTVIBotOutputMessage` includes the fields:
|
||||||
|
- `spoken`: A boolean indicating whether the text was spoken by TTS
|
||||||
|
- `aggregated_by`: A string representing how the text was aggregated ("sentence", "word",
|
||||||
|
"my custom aggregation")
|
||||||
|
- Introduced new fields to `RTVIObserver` to support the new `bot-output` messaging:
|
||||||
|
- `bot_output_enabled`: Defaults to True. Set to false to disable bot-output messages.
|
||||||
|
- `skip_aggregator_types`: Defaults to `None`. Set to a list of strings that match
|
||||||
|
aggregation types that should not be included in bot-output messages. (Ex. `credit_card`)
|
||||||
|
- Introduced new methods, `add_text_transformer()` and `remove_text_transformer()`, to `RTVIObserver` to support providing (and subsequently removing)
|
||||||
|
callbacks for various types of aggregations (or all aggregations with `*`) that can modify the
|
||||||
|
text before being sent as a `bot-output` or `tts-text` message. (Think obscuring the credit card
|
||||||
|
or inserting extra detail the client might want that the context doesn't need.)
|
||||||
|
|
||||||
|
- Updated the base aggregator type:
|
||||||
|
- Introduced a new `Aggregation` dataclass to represent both the aggregated `text` and
|
||||||
|
a string identifying the `type` of aggregation (ex. "sentence", "word", "my custom
|
||||||
|
aggregation")
|
||||||
|
- **BREAKING**: `BaseTextAggregator.text` now returns an `Aggregation` (instead of `str`).
|
||||||
|
To update: `aggregated_text = myAggregator.text` -> `aggregated_text = myAggregator.text.text`
|
||||||
|
- **BREAKING**: `BaseTextAggregator.aggregate()` now returns `Optional[Aggregation]`
|
||||||
|
(instead of `Optional[str]`). To update:
|
||||||
|
```
|
||||||
|
aggregation = myAggregator.aggregate(text)
|
||||||
|
if (aggregation):
|
||||||
|
print(f"successfully aggregated text: {aggregation.text}") // instead of {aggregation}
|
||||||
|
```
|
||||||
|
- `SimpleTextAggregator`, `SkipTagsAggregator`, `PatternPairAggregator` updated to
|
||||||
|
produce/consume `Aggregation` objects.
|
||||||
|
|
||||||
|
- Augmented the `PatternPairAggregator`:
|
||||||
|
- Introduced a new, preferred version of `add_pattern` to support a new option for treating a
|
||||||
|
match as a separate aggregation returned from `aggregate()`. This replaces the now
|
||||||
|
deprecated `add_pattern_pair` method and you provide a `MatchAction` in lieu of the `remove_match` field.
|
||||||
|
- `MatchAction` enum: `REMOVE`, `KEEP`, `AGGREGATE`, allowing customization for how
|
||||||
|
a match should be handled.
|
||||||
|
- `REMOVE`: The text along with its delimiters will be removed from the streaming text.
|
||||||
|
Sentence aggregation will continue on as if this text did not exist.
|
||||||
|
- `KEEP`: The delimiters will be removed, but the content between them will be kept.
|
||||||
|
Sentence aggregation will continue on with the internal text included.
|
||||||
|
- `AGGREGATE`: The delimiters will be removed and the content between will be treated
|
||||||
|
as a separate aggregation. Any text before the start of the pattern will be
|
||||||
|
returned early, whether or not a complete sentence was found. Then the pattern
|
||||||
|
will be returned. Then the aggregation will continue on sentence matching after
|
||||||
|
the closing delimiter is found. The content between the delimiters is not
|
||||||
|
aggregated by sentence. It is aggregated as one single block of text.
|
||||||
|
- `PatternMatch` now extends `Aggregation` and provides richer info to handlers.
|
||||||
|
- **BREAKING**: The `PatternMatch` type returned to handlers registered via `on_pattern_match`
|
||||||
|
has been updated to subclass from the new `Aggregation` type, which means that `content`
|
||||||
|
has been replaced with `text` and `pattern_id` has been replaced with `type`:
|
||||||
|
```
|
||||||
|
async dev on_match_tag(match: PatternMatch):
|
||||||
|
pattern = match.type # instead of match.pattern_id
|
||||||
|
text = match.text # instead of match.content
|
||||||
|
```
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
|
|
||||||
- ⚠️ Breaking change: `LLMContext.create_image_message()`,
|
|
||||||
`LLMContext.create_audio_message()`, `LLMContext.add_image_frame_message()`
|
|
||||||
and `LLMContext.add_audio_frames_message()` are now async methods. This fixes
|
|
||||||
an issue where the asyncio event loop would be blocked while encoding audio or
|
|
||||||
images.
|
|
||||||
|
|
||||||
- `ConsumerProcessor` now queues frames from the producer internally instead of
|
|
||||||
pushing them directly. This allows us to subclass consumer processors and
|
|
||||||
manipulate frames before they are pushed.
|
|
||||||
|
|
||||||
- `BaseTextFilter` only require subclasses to implement the `filter()` method.
|
|
||||||
|
|
||||||
- Extracted the logic for retrying connections, and create a new `send_with_retry`
|
|
||||||
method inside `WebSocketService`.
|
|
||||||
|
|
||||||
- Refactored `DeepgramFluxSTTService` to automatically reconnect if sending a
|
|
||||||
message fails.
|
|
||||||
|
|
||||||
- Updated all STT and TTS services to use consistent error handling pattern with
|
- Updated all STT and TTS services to use consistent error handling pattern with
|
||||||
`push_error()` method for better pipeline error event integration.
|
`push_error()` method for better pipeline error event integration.
|
||||||
|
|
||||||
- Added support for `maybe_capture_participant_camera()` and
|
|
||||||
`maybe_capture_participant_screen()` for `SmallWebRTCTransport` in the runner
|
|
||||||
utils.
|
|
||||||
|
|
||||||
- Added Hindi support for Rime TTS services.
|
- Added Hindi support for Rime TTS services.
|
||||||
|
|
||||||
- Updated `GeminiTTSService` to use Google Cloud Text-to-Speech streaming API
|
- Updated `GeminiTTSService` to use Google Cloud Text-to-Speech streaming API
|
||||||
@@ -351,18 +109,44 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
- Updated language mappings for the Google and Gemini TTS services to match
|
- Updated language mappings for the Google and Gemini TTS services to match
|
||||||
official documentation.
|
official documentation.
|
||||||
|
|
||||||
|
- `TextFrame` new field `append_to_context` used to indicate if the encompassing
|
||||||
|
text should be added to the LLM context (by the LLM assistant aggregator). It
|
||||||
|
defaults to `True`.
|
||||||
|
|
||||||
|
- TTS flow respects aggregation metadata
|
||||||
|
- `TTSService` accepts a new `skip_aggregator_types` to avoid speaking certain aggregation types
|
||||||
|
(now determined/returned by the aggregator)
|
||||||
|
- TTS services push `AggregatedTextFrame` in addition to `TTSTextFrame`s when either an
|
||||||
|
aggregation occurs that should not be spoken or when the TTS service supports word-by-word
|
||||||
|
timestamping. In the latter case, the `TTSService` preliminarily generates an
|
||||||
|
`AggregatedTextFrame`, aggregated by sentence to generate the full sentence content as early
|
||||||
|
as possible.
|
||||||
|
- Introduced a new methods, `add_text_transformer()` and `remove_text_transformer()`:
|
||||||
|
These functions introduce the ability to provide (and subsequently remove) callbacks to the TTS to transform text based on
|
||||||
|
its aggregated type prior to sending the text to the underlying TTS service. This makes it
|
||||||
|
possible to do things like introduce TTS-specific tags for spelling or emotion or change the
|
||||||
|
pronunciation of something on the fly.
|
||||||
|
|
||||||
### Deprecated
|
### Deprecated
|
||||||
|
|
||||||
- The `api_key` parameter in `GeminiTTSService` is deprecated. Use
|
- The `api_key` parameter in `GeminiTTSService` is deprecated. Use
|
||||||
`credentials` or `credentials_path` instead for Google Cloud authentication.
|
`credentials` or `credentials_path` instead for Google Cloud authentication.
|
||||||
|
|
||||||
|
- The RTVI `bot-transcription` event is deprecated in favor of the new `bot-output`
|
||||||
|
message which is the canonical representation of bot output (spoken or not). The code
|
||||||
|
still emits a transcription message for backwards compatibility while transition occurs.
|
||||||
|
|
||||||
|
- The TTS constructor field, `text_aggregator` is deprecated in favor of the new
|
||||||
|
`LLMTextProcessor`. TTSServices still have an internal aggregator for support of default
|
||||||
|
behavior, but if you want to override the aggregation behavior, you should use the new
|
||||||
|
processor.
|
||||||
|
|
||||||
|
- Deprecated `add_pattern_pair` in the `PatternPairAggregator` which takes a `pattern_id`
|
||||||
|
and `remove_match` field in favor of the new `add_pattern` method which takes a `type` and an
|
||||||
|
`action`
|
||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
|
|
||||||
- Fixed a `SimliVideoService` connection issue.
|
|
||||||
|
|
||||||
- Fixed an issue in the `Runner` where, when using `SmallWebRTCTransport`, the
|
|
||||||
`request_data` was not being passed to the `SmallWebRTCRunnerArguments` body.
|
|
||||||
|
|
||||||
- Fixed subtle issue of assistant context messages ending up with double spaces
|
- Fixed subtle issue of assistant context messages ending up with double spaces
|
||||||
between words or sentences.
|
between words or sentences.
|
||||||
|
|
||||||
@@ -377,6 +161,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
- Prevented `HeyGenVideoService` from automatically disconnecting after 5 minutes.
|
- Prevented `HeyGenVideoService` from automatically disconnecting after 5 minutes.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- Added ai-coustics integrated VAD (`AICVADAnalyzer`) with `AICFilter` factory and
|
||||||
|
example wiring; leverages the enhancement model for robust detection with no
|
||||||
|
ONNX dependency or added processing complexity.
|
||||||
|
|
||||||
## [0.0.94] - 2025-11-10
|
## [0.0.94] - 2025-11-10
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
|
|||||||
@@ -44,7 +44,6 @@ DAILY_SAMPLE_ROOM_URL=https://...
|
|||||||
|
|
||||||
# Deepgram
|
# Deepgram
|
||||||
DEEPGRAM_API_KEY=...
|
DEEPGRAM_API_KEY=...
|
||||||
SAGEMAKER_ENDPOINT_NAME=...
|
|
||||||
|
|
||||||
# DeepSeek
|
# DeepSeek
|
||||||
DEEPSEEK_API_KEY=...
|
DEEPSEEK_API_KEY=...
|
||||||
|
|||||||
@@ -13,29 +13,24 @@ from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams
|
|||||||
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
|
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
|
||||||
from pipecat.audio.vad.silero import SileroVADAnalyzer
|
from pipecat.audio.vad.silero import SileroVADAnalyzer
|
||||||
from pipecat.audio.vad.vad_analyzer import VADParams
|
from pipecat.audio.vad.vad_analyzer import VADParams
|
||||||
from pipecat.frames.frames import LLMRunFrame, TTSTextFrame
|
from pipecat.frames.frames import LLMRunFrame
|
||||||
from pipecat.observers.loggers.debug_log_observer import DebugLogObserver, FrameEndpoint
|
|
||||||
from pipecat.pipeline.pipeline import Pipeline
|
from pipecat.pipeline.pipeline import Pipeline
|
||||||
from pipecat.pipeline.runner import PipelineRunner
|
from pipecat.pipeline.runner import PipelineRunner
|
||||||
from pipecat.pipeline.task import PipelineParams, PipelineTask
|
from pipecat.pipeline.task import PipelineParams, PipelineTask
|
||||||
from pipecat.processors.aggregators.llm_context import LLMContext
|
from pipecat.processors.aggregators.llm_context import LLMContext
|
||||||
from pipecat.processors.aggregators.llm_response_universal import (
|
from pipecat.processors.aggregators.llm_response_universal import LLMContextAggregatorPair
|
||||||
LLMContextAggregatorPair,
|
|
||||||
)
|
|
||||||
from pipecat.processors.frameworks.rtvi import RTVIConfig, RTVIObserver, RTVIProcessor
|
from pipecat.processors.frameworks.rtvi import RTVIConfig, RTVIObserver, RTVIProcessor
|
||||||
from pipecat.runner.types import RunnerArguments
|
from pipecat.runner.types import RunnerArguments
|
||||||
from pipecat.runner.utils import create_transport
|
from pipecat.runner.utils import create_transport
|
||||||
from pipecat.services.deepgram.stt import DeepgramSTTService
|
from pipecat.services.deepgram.stt import DeepgramSTTService
|
||||||
from pipecat.services.hume.tts import HUME_SAMPLE_RATE, HumeTTSService
|
from pipecat.services.hume.tts import HUME_SAMPLE_RATE, HumeTTSService
|
||||||
from pipecat.services.openai.llm import OpenAILLMService
|
from pipecat.services.openai.llm import OpenAILLMService
|
||||||
from pipecat.transports.base_output import BaseOutputTransport
|
|
||||||
from pipecat.transports.base_transport import BaseTransport, TransportParams
|
from pipecat.transports.base_transport import BaseTransport, TransportParams
|
||||||
from pipecat.transports.daily.transport import DailyParams
|
from pipecat.transports.daily.transport import DailyParams
|
||||||
from pipecat.transports.websocket.fastapi import FastAPIWebsocketParams
|
from pipecat.transports.websocket.fastapi import FastAPIWebsocketParams
|
||||||
|
|
||||||
load_dotenv(override=True)
|
load_dotenv(override=True)
|
||||||
|
|
||||||
|
|
||||||
# We store functions so objects (e.g. SileroVADAnalyzer) don't get
|
# We store functions so objects (e.g. SileroVADAnalyzer) don't get
|
||||||
# instantiated. The function will be called when the desired transport gets
|
# instantiated. The function will be called when the desired transport gets
|
||||||
# selected.
|
# selected.
|
||||||
@@ -93,7 +88,7 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
|||||||
stt,
|
stt,
|
||||||
context_aggregator.user(), # User responses
|
context_aggregator.user(), # User responses
|
||||||
llm, # LLM
|
llm, # LLM
|
||||||
tts, # TTS (HumeTTSService with word timestamps)
|
tts, # TTS
|
||||||
transport.output(), # Transport bot output
|
transport.output(), # Transport bot output
|
||||||
context_aggregator.assistant(), # Assistant spoken responses
|
context_aggregator.assistant(), # Assistant spoken responses
|
||||||
]
|
]
|
||||||
@@ -107,14 +102,7 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
|||||||
audio_out_sample_rate=HUME_SAMPLE_RATE,
|
audio_out_sample_rate=HUME_SAMPLE_RATE,
|
||||||
),
|
),
|
||||||
idle_timeout_secs=runner_args.pipeline_idle_timeout_secs,
|
idle_timeout_secs=runner_args.pipeline_idle_timeout_secs,
|
||||||
observers=[
|
observers=[RTVIObserver(rtvi)],
|
||||||
RTVIObserver(rtvi),
|
|
||||||
DebugLogObserver(
|
|
||||||
frame_types={
|
|
||||||
TTSTextFrame: (BaseOutputTransport, FrameEndpoint.SOURCE),
|
|
||||||
}
|
|
||||||
),
|
|
||||||
],
|
|
||||||
)
|
)
|
||||||
|
|
||||||
@rtvi.event_handler("on_client_ready")
|
@rtvi.event_handler("on_client_ready")
|
||||||
@@ -124,9 +112,6 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
|||||||
@transport.event_handler("on_client_connected")
|
@transport.event_handler("on_client_connected")
|
||||||
async def on_client_connected(transport, client):
|
async def on_client_connected(transport, client):
|
||||||
logger.info(f"Client connected")
|
logger.info(f"Client connected")
|
||||||
logger.info(
|
|
||||||
"💡 Word timestamps are enabled! Watch the console for TTSTextFrame logs showing each word with its PTS."
|
|
||||||
)
|
|
||||||
# Kick off the conversation.
|
# Kick off the conversation.
|
||||||
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
|
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
|
||||||
await task.queue_frames([LLMRunFrame()])
|
await task.queue_frames([LLMRunFrame()])
|
||||||
|
|||||||
@@ -52,10 +52,7 @@ transport_params = {
|
|||||||
async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
||||||
logger.info(f"Starting bot")
|
logger.info(f"Starting bot")
|
||||||
|
|
||||||
stt = DeepgramFluxSTTService(
|
stt = DeepgramFluxSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
|
||||||
api_key=os.getenv("DEEPGRAM_API_KEY"),
|
|
||||||
params=DeepgramFluxSTTService.InputParams(min_confidence=0.3),
|
|
||||||
)
|
|
||||||
|
|
||||||
tts = DeepgramTTSService(api_key=os.getenv("DEEPGRAM_API_KEY"), voice="aura-2-andromeda-en")
|
tts = DeepgramTTSService(api_key=os.getenv("DEEPGRAM_API_KEY"), voice="aura-2-andromeda-en")
|
||||||
|
|
||||||
|
|||||||
@@ -110,7 +110,7 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
|||||||
|
|
||||||
# Kick off the conversation.
|
# Kick off the conversation.
|
||||||
image = Image.open(image_path)
|
image = Image.open(image_path)
|
||||||
message = await LLMContext.create_image_message(
|
message = LLMContext.create_image_message(
|
||||||
image=image.tobytes(),
|
image=image.tobytes(),
|
||||||
format="RGB",
|
format="RGB",
|
||||||
size=image.size,
|
size=image.size,
|
||||||
|
|||||||
@@ -110,7 +110,7 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
|||||||
|
|
||||||
# Kick off the conversation.
|
# Kick off the conversation.
|
||||||
image = Image.open(image_path)
|
image = Image.open(image_path)
|
||||||
message = await LLMContext.create_image_message(
|
message = LLMContext.create_image_message(
|
||||||
image=image.tobytes(),
|
image=image.tobytes(),
|
||||||
format="RGB",
|
format="RGB",
|
||||||
size=image.size,
|
size=image.size,
|
||||||
|
|||||||
@@ -117,7 +117,7 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
|||||||
|
|
||||||
# Kick off the conversation.
|
# Kick off the conversation.
|
||||||
image = Image.open(image_path)
|
image = Image.open(image_path)
|
||||||
message = await LLMContext.create_image_message(
|
message = LLMContext.create_image_message(
|
||||||
image=image.tobytes(),
|
image=image.tobytes(),
|
||||||
format="RGB",
|
format="RGB",
|
||||||
size=image.size,
|
size=image.size,
|
||||||
|
|||||||
@@ -110,7 +110,7 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
|||||||
|
|
||||||
# Kick off the conversation.
|
# Kick off the conversation.
|
||||||
image = Image.open(image_path)
|
image = Image.open(image_path)
|
||||||
message = await LLMContext.create_image_message(
|
message = LLMContext.create_image_message(
|
||||||
image=image.tobytes(),
|
image=image.tobytes(),
|
||||||
format="RGB",
|
format="RGB",
|
||||||
size=image.size,
|
size=image.size,
|
||||||
|
|||||||
@@ -15,21 +15,14 @@ from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams
|
|||||||
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
|
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
|
||||||
from pipecat.audio.vad.silero import SileroVADAnalyzer
|
from pipecat.audio.vad.silero import SileroVADAnalyzer
|
||||||
from pipecat.audio.vad.vad_analyzer import VADParams
|
from pipecat.audio.vad.vad_analyzer import VADParams
|
||||||
from pipecat.frames.frames import (
|
from pipecat.frames.frames import LLMRunFrame, UserImageRequestFrame
|
||||||
Frame,
|
|
||||||
LLMFullResponseEndFrame,
|
|
||||||
LLMFullResponseStartFrame,
|
|
||||||
LLMRunFrame,
|
|
||||||
TextFrame,
|
|
||||||
UserImageRequestFrame,
|
|
||||||
)
|
|
||||||
from pipecat.pipeline.parallel_pipeline import ParallelPipeline
|
from pipecat.pipeline.parallel_pipeline import ParallelPipeline
|
||||||
from pipecat.pipeline.pipeline import Pipeline
|
from pipecat.pipeline.pipeline import Pipeline
|
||||||
from pipecat.pipeline.runner import PipelineRunner
|
from pipecat.pipeline.runner import PipelineRunner
|
||||||
from pipecat.pipeline.task import PipelineTask
|
from pipecat.pipeline.task import PipelineTask
|
||||||
from pipecat.processors.aggregators.llm_context import LLMContext
|
from pipecat.processors.aggregators.llm_context import LLMContext
|
||||||
from pipecat.processors.aggregators.llm_response_universal import LLMContextAggregatorPair
|
from pipecat.processors.aggregators.llm_response_universal import LLMContextAggregatorPair
|
||||||
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
|
from pipecat.processors.frame_processor import FrameDirection
|
||||||
from pipecat.runner.types import RunnerArguments
|
from pipecat.runner.types import RunnerArguments
|
||||||
from pipecat.runner.utils import (
|
from pipecat.runner.utils import (
|
||||||
create_transport,
|
create_transport,
|
||||||
@@ -73,27 +66,6 @@ async def fetch_user_image(params: FunctionCallParams):
|
|||||||
# await params.result_callback({"result": "Image is being captured."})
|
# await params.result_callback({"result": "Image is being captured."})
|
||||||
|
|
||||||
|
|
||||||
class MoondreamTextFrameWrapper(FrameProcessor):
|
|
||||||
"""Wraps Moondream-provided TextFrames with LLM response start/end frames.
|
|
||||||
|
|
||||||
This processor detects TextFrames and automatically wraps them with
|
|
||||||
LLMFullResponseStartFrame and LLMFullResponseEndFrame to provide proper
|
|
||||||
response boundaries for downstream processors.
|
|
||||||
"""
|
|
||||||
|
|
||||||
async def process_frame(self, frame: Frame, direction: FrameDirection):
|
|
||||||
await super().process_frame(frame, direction)
|
|
||||||
|
|
||||||
# If we receive a TextFrame, wrap it with response start/end frames
|
|
||||||
if isinstance(frame, TextFrame):
|
|
||||||
await self.push_frame(LLMFullResponseStartFrame(), direction)
|
|
||||||
await self.push_frame(frame, direction)
|
|
||||||
await self.push_frame(LLMFullResponseEndFrame(), direction)
|
|
||||||
else:
|
|
||||||
# For all other frames, just pass them through
|
|
||||||
await self.push_frame(frame, direction)
|
|
||||||
|
|
||||||
|
|
||||||
# We store functions so objects (e.g. SileroVADAnalyzer) don't get
|
# We store functions so objects (e.g. SileroVADAnalyzer) don't get
|
||||||
# instantiated. The function will be called when the desired transport gets
|
# instantiated. The function will be called when the desired transport gets
|
||||||
# selected.
|
# selected.
|
||||||
@@ -158,12 +130,6 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
|||||||
# If you run into weird description, try with use_cpu=True
|
# If you run into weird description, try with use_cpu=True
|
||||||
moondream = MoondreamService()
|
moondream = MoondreamService()
|
||||||
|
|
||||||
# Wrap TextFrames with LLM response start/end frames, which makes Moondream
|
|
||||||
# output be treated like LLM responses for the purpose of context
|
|
||||||
# aggregation. Without this, the assistant context aggregator would ignore
|
|
||||||
# Moondream output (if the TTS service is disabled).
|
|
||||||
moondream_text_wrapper = MoondreamTextFrameWrapper()
|
|
||||||
|
|
||||||
pipeline = Pipeline(
|
pipeline = Pipeline(
|
||||||
[
|
[
|
||||||
transport.input(), # Transport user input
|
transport.input(), # Transport user input
|
||||||
@@ -171,7 +137,7 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
|||||||
context_aggregator.user(), # User responses
|
context_aggregator.user(), # User responses
|
||||||
ParallelPipeline(
|
ParallelPipeline(
|
||||||
[llm], # LLM
|
[llm], # LLM
|
||||||
[moondream, moondream_text_wrapper],
|
[moondream],
|
||||||
),
|
),
|
||||||
tts, # TTS
|
tts, # TTS
|
||||||
transport.output(), # Transport bot output
|
transport.output(), # Transport bot output
|
||||||
|
|||||||
@@ -391,7 +391,7 @@ class AudioAccumulator(FrameProcessor):
|
|||||||
)
|
)
|
||||||
self._user_speaking = False
|
self._user_speaking = False
|
||||||
context = LLMContext()
|
context = LLMContext()
|
||||||
await context.add_audio_frames_message(audio_frames=self._audio_frames)
|
context.add_audio_frames_message(audio_frames=self._audio_frames)
|
||||||
await self.push_frame(LLMContextFrame(context=context))
|
await self.push_frame(LLMContextFrame(context=context))
|
||||||
elif isinstance(frame, InputAudioRawFrame):
|
elif isinstance(frame, InputAudioRawFrame):
|
||||||
# Append the audio frame to our buffer. Treat the buffer as a ring buffer, dropping the oldest
|
# Append the audio frame to our buffer. Treat the buffer as a ring buffer, dropping the oldest
|
||||||
|
|||||||
@@ -150,7 +150,7 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
|||||||
LLMLogObserver(),
|
LLMLogObserver(),
|
||||||
DebugLogObserver(
|
DebugLogObserver(
|
||||||
frame_types={
|
frame_types={
|
||||||
TTSTextFrame: (BaseOutputTransport, FrameEndpoint.SOURCE),
|
TTSTextFrame: (BaseOutputTransport, FrameEndpoint.DESTINATION),
|
||||||
UserStartedSpeakingFrame: (BaseInputTransport, FrameEndpoint.SOURCE),
|
UserStartedSpeakingFrame: (BaseInputTransport, FrameEndpoint.SOURCE),
|
||||||
EndFrame: None,
|
EndFrame: None,
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -155,7 +155,7 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
|||||||
You are a helpful LLM in a WebRTC call.
|
You are a helpful LLM in a WebRTC call.
|
||||||
Your goal is to demonstrate your capabilities in a succinct way.
|
Your goal is to demonstrate your capabilities in a succinct way.
|
||||||
You have access to tools to search the Rijksmuseum collection.
|
You have access to tools to search the Rijksmuseum collection.
|
||||||
Offer, for example, to show a floral still life, use the `search_artwork` tool.
|
Offer, for example, to show the earliest Rembrandt work from the museum. Use the `search_artwork` tool.
|
||||||
The tool may respond with a JSON object with an `artworks` array. Choose the art from that array.
|
The tool may respond with a JSON object with an `artworks` array. Choose the art from that array.
|
||||||
Once the tool has responded, tell the user the title and use the `open_image_in_browser` tool.
|
Once the tool has responded, tell the user the title and use the `open_image_in_browser` tool.
|
||||||
Your output will be spoken aloud, so avoid special characters that can't easily be spoken, such as emojis or bullet points.
|
Your output will be spoken aloud, so avoid special characters that can't easily be spoken, such as emojis or bullet points.
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ import os
|
|||||||
|
|
||||||
from dotenv import load_dotenv
|
from dotenv import load_dotenv
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
from mcp.client.session_group import SseServerParameters
|
||||||
|
|
||||||
from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams
|
from pipecat.audio.turn.smart_turn.base_smart_turn import SmartTurnParams
|
||||||
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
|
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
|
||||||
@@ -22,16 +23,16 @@ from pipecat.processors.aggregators.llm_context import LLMContext
|
|||||||
from pipecat.processors.aggregators.llm_response_universal import LLMContextAggregatorPair
|
from pipecat.processors.aggregators.llm_response_universal import LLMContextAggregatorPair
|
||||||
from pipecat.runner.types import RunnerArguments
|
from pipecat.runner.types import RunnerArguments
|
||||||
from pipecat.runner.utils import create_transport
|
from pipecat.runner.utils import create_transport
|
||||||
from pipecat.services.aws.llm import AWSBedrockLLMService
|
from pipecat.services.anthropic.llm import AnthropicLLMService
|
||||||
from pipecat.services.deepgram.stt_sagemaker import DeepgramSageMakerSTTService
|
from pipecat.services.cartesia.tts import CartesiaTTSService
|
||||||
from pipecat.services.deepgram.tts import DeepgramTTSService
|
from pipecat.services.deepgram.stt import DeepgramSTTService
|
||||||
|
from pipecat.services.mcp_service import MCPClient
|
||||||
from pipecat.transports.base_transport import BaseTransport, TransportParams
|
from pipecat.transports.base_transport import BaseTransport, TransportParams
|
||||||
from pipecat.transports.daily.transport import DailyParams
|
from pipecat.transports.daily.transport import DailyParams
|
||||||
from pipecat.transports.websocket.fastapi import FastAPIWebsocketParams
|
from pipecat.transports.websocket.fastapi import FastAPIWebsocketParams
|
||||||
|
|
||||||
load_dotenv(override=True)
|
load_dotenv(override=True)
|
||||||
|
|
||||||
|
|
||||||
# We store functions so objects (e.g. SileroVADAnalyzer) don't get
|
# We store functions so objects (e.g. SileroVADAnalyzer) don't get
|
||||||
# instantiated. The function will be called when the desired transport gets
|
# instantiated. The function will be called when the desired transport gets
|
||||||
# selected.
|
# selected.
|
||||||
@@ -60,42 +61,56 @@ transport_params = {
|
|||||||
async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
||||||
logger.info(f"Starting bot")
|
logger.info(f"Starting bot")
|
||||||
|
|
||||||
# Initialize Deepgram SageMaker STT Service
|
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
|
||||||
# This requires:
|
|
||||||
# - AWS credentials configured (via environment variables or AWS CLI)
|
tts = CartesiaTTSService(
|
||||||
# - A deployed SageMaker endpoint with Deepgram model
|
api_key=os.getenv("CARTESIA_API_KEY"),
|
||||||
stt = DeepgramSageMakerSTTService(
|
voice_id="71a7ad14-091c-4e8e-a314-022ece01c121", # British Reading Lady
|
||||||
endpoint_name=os.getenv("SAGEMAKER_ENDPOINT_NAME"),
|
|
||||||
region=os.getenv("AWS_REGION"),
|
|
||||||
)
|
)
|
||||||
|
|
||||||
tts = DeepgramTTSService(api_key=os.getenv("DEEPGRAM_API_KEY"), voice="aura-2-andromeda-en")
|
llm = AnthropicLLMService(
|
||||||
|
api_key=os.getenv("ANTHROPIC_API_KEY"), model="claude-3-7-sonnet-latest"
|
||||||
llm = AWSBedrockLLMService(
|
|
||||||
aws_region=os.getenv("AWS_REGION"),
|
|
||||||
model="us.amazon.nova-pro-v1:0",
|
|
||||||
params=AWSBedrockLLMService.InputParams(temperature=0.8),
|
|
||||||
)
|
)
|
||||||
|
|
||||||
messages = [
|
try:
|
||||||
{
|
# https://docs.mcp.run/integrating/tutorials/mcp-run-sse-openai-agents/
|
||||||
"role": "system",
|
mcp = MCPClient(server_params=SseServerParameters(url=os.getenv("MCP_RUN_SSE_URL")))
|
||||||
"content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be spoken aloud, so avoid special characters that can't easily be spoken, such as emojis or bullet points. Respond to what the user said in a creative and helpful way.",
|
except Exception as e:
|
||||||
},
|
logger.error(f"error setting up mcp")
|
||||||
]
|
logger.exception("error trace:")
|
||||||
|
|
||||||
context = LLMContext(messages)
|
tools = {}
|
||||||
|
try:
|
||||||
|
tools = await mcp.register_tools(llm)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"error registering tools")
|
||||||
|
logger.exception("error trace:")
|
||||||
|
|
||||||
|
system = f"""
|
||||||
|
You are a helpful LLM in a WebRTC call.
|
||||||
|
Your goal is to demonstrate your capabilities in a succinct way.
|
||||||
|
You have access to a number of tools provided by mcp.run. Use any and all tools to help users.
|
||||||
|
Your output will be spoken aloud, so avoid special characters that can't easily be spoken, such as emojis or bullet points.
|
||||||
|
Respond to what the user said in a creative and helpful way.
|
||||||
|
When asked for today's date, use 'https://www.datetoday.net/'.
|
||||||
|
Don't overexplain what you are doing.
|
||||||
|
Just respond with short sentences when you are carrying out tool calls.
|
||||||
|
"""
|
||||||
|
|
||||||
|
messages = [{"role": "system", "content": system}]
|
||||||
|
|
||||||
|
context = LLMContext(messages, tools)
|
||||||
context_aggregator = LLMContextAggregatorPair(context)
|
context_aggregator = LLMContextAggregatorPair(context)
|
||||||
|
|
||||||
pipeline = Pipeline(
|
pipeline = Pipeline(
|
||||||
[
|
[
|
||||||
transport.input(), # Transport user input
|
transport.input(), # Transport user input
|
||||||
stt, # STT
|
stt,
|
||||||
context_aggregator.user(), # User responses
|
context_aggregator.user(), # User spoken responses
|
||||||
llm, # LLM
|
llm, # LLM
|
||||||
tts, # TTS
|
tts, # TTS
|
||||||
transport.output(), # Transport bot output
|
transport.output(), # Transport bot output
|
||||||
context_aggregator.assistant(), # Assistant spoken responses
|
context_aggregator.assistant(), # Assistant spoken responses and tool context
|
||||||
]
|
]
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -110,9 +125,8 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
|||||||
|
|
||||||
@transport.event_handler("on_client_connected")
|
@transport.event_handler("on_client_connected")
|
||||||
async def on_client_connected(transport, client):
|
async def on_client_connected(transport, client):
|
||||||
logger.info(f"Client connected")
|
logger.info(f"Client connected: {client}")
|
||||||
# Kick off the conversation.
|
# Kick off the conversation.
|
||||||
messages.append({"role": "system", "content": "Please introduce yourself to the user."})
|
|
||||||
await task.queue_frames([LLMRunFrame()])
|
await task.queue_frames([LLMRunFrame()])
|
||||||
|
|
||||||
@transport.event_handler("on_client_disconnected")
|
@transport.event_handler("on_client_disconnected")
|
||||||
@@ -132,6 +146,14 @@ async def bot(runner_args: RunnerArguments):
|
|||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
if not os.getenv("MCP_RUN_SSE_URL"):
|
||||||
|
logger.error(
|
||||||
|
f"Please set MCP_RUN_SSE_URL environment variable for this example. See https://mcp.run"
|
||||||
|
)
|
||||||
|
import sys
|
||||||
|
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
from pipecat.runner.run import main
|
from pipecat.runner.run import main
|
||||||
|
|
||||||
main()
|
main()
|
||||||
@@ -7,7 +7,6 @@
|
|||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import io
|
import io
|
||||||
import json
|
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
import shutil
|
import shutil
|
||||||
@@ -16,7 +15,7 @@ import aiohttp
|
|||||||
from dotenv import load_dotenv
|
from dotenv import load_dotenv
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
from mcp import StdioServerParameters
|
from mcp import StdioServerParameters
|
||||||
from mcp.client.session_group import StreamableHttpParameters
|
from mcp.client.session_group import SseServerParameters
|
||||||
from PIL import Image
|
from PIL import Image
|
||||||
|
|
||||||
from pipecat.adapters.schemas.tools_schema import ToolsSchema
|
from pipecat.adapters.schemas.tools_schema import ToolsSchema
|
||||||
@@ -67,12 +66,10 @@ class UrlToImageProcessor(FrameProcessor):
|
|||||||
await self.push_frame(frame, direction)
|
await self.push_frame(frame, direction)
|
||||||
|
|
||||||
def extract_url(self, text: str):
|
def extract_url(self, text: str):
|
||||||
data = json.loads(text)
|
pattern = r"!\[[^\]]*\]\((https?://[^)]+\.(png|jpg|jpeg|PNG|JPG|JPEG|gif))\)"
|
||||||
if "artObject" in data:
|
match = re.search(pattern, text)
|
||||||
return data["artObject"]["webImage"]["url"]
|
if match:
|
||||||
if "artworks" in data and len(data["artworks"]):
|
return match.group(1)
|
||||||
return data["artworks"][0]["webImage"]["url"]
|
|
||||||
|
|
||||||
return None
|
return None
|
||||||
|
|
||||||
async def run_image_process(self, image_url: str):
|
async def run_image_process(self, image_url: str):
|
||||||
@@ -135,11 +132,10 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
|||||||
system = f"""
|
system = f"""
|
||||||
You are a helpful LLM in a WebRTC call.
|
You are a helpful LLM in a WebRTC call.
|
||||||
Your goal is to demonstrate your capabilities in a succinct way.
|
Your goal is to demonstrate your capabilities in a succinct way.
|
||||||
You have access to tools to search the Rijksmuseum collection and the user's GitHub repositories and account.
|
You have access to tools to search the Rijksmuseum collection.
|
||||||
Offer, for example, to show a floral still life, use the `search_artwork` tool.
|
Offer, for example, to show the earliest Rembrandt work from the museum. Use the `search_artwork` tool.
|
||||||
The tool may respond with a JSON object with an `artworks` array. Choose the art from that array.
|
The tool may respond with a JSON object with an `artworks` array. Choose the art from that array.
|
||||||
Once the tool has responded, tell the user the title and use the `open_image_in_browser` tool.
|
Once the tool has responded, tell the user the title and use the `open_image_in_browser` tool.
|
||||||
You can also offer to answer users questions about their GitHub repositories and account.
|
|
||||||
Your output will be spoken aloud, so avoid special characters that can't easily be spoken, such as emojis or bullet points.
|
Your output will be spoken aloud, so avoid special characters that can't easily be spoken, such as emojis or bullet points.
|
||||||
Respond to what the user said in a creative and helpful way.
|
Respond to what the user said in a creative and helpful way.
|
||||||
Don't overexplain what you are doing.
|
Don't overexplain what you are doing.
|
||||||
@@ -149,11 +145,11 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
|||||||
messages = [{"role": "system", "content": system}]
|
messages = [{"role": "system", "content": system}]
|
||||||
|
|
||||||
try:
|
try:
|
||||||
rijksmuseum_mcp = MCPClient(
|
mcp = MCPClient(
|
||||||
server_params=StdioServerParameters(
|
server_params=StdioServerParameters(
|
||||||
command=shutil.which("npx"),
|
command=shutil.which("npx"),
|
||||||
# https://github.com/r-huijts/rijksmuseum-mcp
|
# https://github.com/r-huijts/rijksmuseum-mcp
|
||||||
args=["-y", "mcp-server-rijksmuseum"],
|
args=["-y", "mcp-server-error setting up mcp"],
|
||||||
env={"RIJKSMUSEUM_API_KEY": os.getenv("RIJKSMUSEUM_API_KEY")},
|
env={"RIJKSMUSEUM_API_KEY": os.getenv("RIJKSMUSEUM_API_KEY")},
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
@@ -161,32 +157,24 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
|||||||
logger.error(f"error setting up rijksmuseum mcp")
|
logger.error(f"error setting up rijksmuseum mcp")
|
||||||
logger.exception("error trace:")
|
logger.exception("error trace:")
|
||||||
try:
|
try:
|
||||||
# Github MCP docs: https://github.com/github/github-mcp-server
|
# https://docs.mcp.run/integrating/tutorials/mcp-run-sse-openai-agents/
|
||||||
# Enable Github Copilot on your GitHub account. Free tier is ok. (https://github.com/settings/copilot)
|
# ie. "https://www.mcp.run/api/mcp/sse?..."
|
||||||
# Generate a personal access token. It must be a Fine-grained token, classic tokens are not supported. (https://github.com/settings/personal-access-tokens)
|
# ensure the profile has a tool or few installed
|
||||||
# Set permissions you want to use (eg. "all repositories", "profile: read/write", etc)
|
mcp_run = MCPClient(server_params=SseServerParameters(url=os.getenv("MCP_RUN_SSE_URL")))
|
||||||
github_mcp = MCPClient(
|
|
||||||
server_params=StreamableHttpParameters(
|
|
||||||
url="https://api.githubcopilot.com/mcp/",
|
|
||||||
headers={
|
|
||||||
"Authorization": f"Bearer {os.getenv('GITHUB_PERSONAL_ACCESS_TOKEN')}"
|
|
||||||
},
|
|
||||||
)
|
|
||||||
)
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"error setting up mcp.run")
|
logger.error(f"error setting up mcp.run")
|
||||||
logger.exception("error trace:")
|
logger.exception("error trace:")
|
||||||
|
|
||||||
rijksmuseum_tools = {}
|
tools = {}
|
||||||
github_tools = {}
|
run_tools = {}
|
||||||
try:
|
try:
|
||||||
rijksmuseum_tools = await rijksmuseum_mcp.register_tools(llm)
|
tools = await mcp.register_tools(llm)
|
||||||
github_tools = await github_mcp.register_tools(llm)
|
run_tools = await mcp_run.register_tools(llm)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"error registering tools")
|
logger.error(f"error registering tools")
|
||||||
logger.exception("error trace:")
|
logger.exception("error trace:")
|
||||||
|
|
||||||
all_standard_tools = rijksmuseum_tools.standard_tools + github_tools.standard_tools
|
all_standard_tools = run_tools.standard_tools + tools.standard_tools
|
||||||
all_tools = ToolsSchema(standard_tools=all_standard_tools)
|
all_tools = ToolsSchema(standard_tools=all_standard_tools)
|
||||||
|
|
||||||
context = LLMContext(messages, all_tools)
|
context = LLMContext(messages, all_tools)
|
||||||
@@ -238,9 +226,9 @@ async def bot(runner_args: RunnerArguments):
|
|||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
if not os.getenv("RIJKSMUSEUM_API_KEY") or not os.getenv("GITHUB_PERSONAL_ACCESS_TOKEN"):
|
if not os.getenv("RIJKSMUSEUM_API_KEY") or not os.getenv("MCP_RUN_SSE_URL"):
|
||||||
logger.error(
|
logger.error(
|
||||||
f"Please set `RIJKSMUSEUM_API_KEY` and `GITHUB_PERSONAL_ACCESS_TOKEN` environment variables. See https://github.com/r-huijts/rijksmuseum-mcp."
|
f"Please set RIJKSMUSEUM_API_KEY and MCP_RUN_SSE_URL environment variables. See https://github.com/r-huijts/rijksmuseum-mcp and https://mcp.run"
|
||||||
)
|
)
|
||||||
import sys
|
import sys
|
||||||
|
|
||||||
@@ -49,14 +49,14 @@ aic = [ "aic-sdk~=1.1.0" ]
|
|||||||
anthropic = [ "anthropic~=0.49.0" ]
|
anthropic = [ "anthropic~=0.49.0" ]
|
||||||
assemblyai = [ "pipecat-ai[websockets-base]" ]
|
assemblyai = [ "pipecat-ai[websockets-base]" ]
|
||||||
asyncai = [ "pipecat-ai[websockets-base]" ]
|
asyncai = [ "pipecat-ai[websockets-base]" ]
|
||||||
aws = [ "aioboto3~=15.5.0", "pipecat-ai[websockets-base]" ]
|
aws = [ "aioboto3~=15.0.0", "pipecat-ai[websockets-base]" ]
|
||||||
aws-nova-sonic = [ "aws_sdk_bedrock_runtime~=0.2.0; python_version>='3.12'" ]
|
aws-nova-sonic = [ "aws_sdk_bedrock_runtime~=0.1.1; python_version>='3.12'" ]
|
||||||
azure = [ "azure-cognitiveservices-speech~=1.42.0"]
|
azure = [ "azure-cognitiveservices-speech~=1.42.0"]
|
||||||
cartesia = [ "cartesia~=2.0.3", "pipecat-ai[websockets-base]" ]
|
cartesia = [ "cartesia~=2.0.3", "pipecat-ai[websockets-base]" ]
|
||||||
cerebras = []
|
cerebras = []
|
||||||
daily = [ "daily-python~=0.22.0" ]
|
|
||||||
deepgram = [ "deepgram-sdk~=4.7.0", "pipecat-ai[websockets-base]" ]
|
|
||||||
deepseek = []
|
deepseek = []
|
||||||
|
daily = [ "daily-python~=0.21.0" ]
|
||||||
|
deepgram = [ "deepgram-sdk~=4.7.0" ]
|
||||||
elevenlabs = [ "pipecat-ai[websockets-base]" ]
|
elevenlabs = [ "pipecat-ai[websockets-base]" ]
|
||||||
fal = [ "fal-client~=0.5.9" ]
|
fal = [ "fal-client~=0.5.9" ]
|
||||||
fireworks = []
|
fireworks = []
|
||||||
@@ -69,21 +69,19 @@ gstreamer = [ "pygobject~=3.50.0" ]
|
|||||||
heygen = [ "livekit>=1.0.13", "pipecat-ai[websockets-base]" ]
|
heygen = [ "livekit>=1.0.13", "pipecat-ai[websockets-base]" ]
|
||||||
hume = [ "hume>=0.11.2" ]
|
hume = [ "hume>=0.11.2" ]
|
||||||
inworld = []
|
inworld = []
|
||||||
koala = [ "pvkoala~=2.0.3" ]
|
|
||||||
krisp = [ "pipecat-ai-krisp~=0.4.0" ]
|
krisp = [ "pipecat-ai-krisp~=0.4.0" ]
|
||||||
|
koala = [ "pvkoala~=2.0.3" ]
|
||||||
langchain = [ "langchain~=0.3.20", "langchain-community~=0.3.20", "langchain-openai~=0.3.9" ]
|
langchain = [ "langchain~=0.3.20", "langchain-community~=0.3.20", "langchain-openai~=0.3.9" ]
|
||||||
livekit = [ "livekit~=1.0.13", "livekit-api~=1.0.5", "tenacity>=8.2.3,<10.0.0", "pyjwt>=2.10.1" ]
|
livekit = [ "livekit~=1.0.13", "livekit-api~=1.0.5", "tenacity>=8.2.3,<10.0.0" ]
|
||||||
lmnt = [ "pipecat-ai[websockets-base]" ]
|
lmnt = [ "pipecat-ai[websockets-base]" ]
|
||||||
local = [ "pyaudio~=0.2.14" ]
|
local = [ "pyaudio~=0.2.14" ]
|
||||||
local-smart-turn = [ "coremltools>=8.0", "transformers", "torch>=2.5.0,<3", "torchaudio>=2.5.0,<3" ]
|
|
||||||
local-smart-turn-v3 = [ "transformers", "onnxruntime>=1.20.1,<2" ]
|
|
||||||
mcp = [ "mcp[cli]>=1.11.0,<2" ]
|
mcp = [ "mcp[cli]>=1.11.0,<2" ]
|
||||||
mem0 = [ "mem0ai~=0.1.94" ]
|
mem0 = [ "mem0ai~=0.1.94" ]
|
||||||
mistral = []
|
mistral = []
|
||||||
mlx-whisper = [ "mlx-whisper~=0.4.2" ]
|
mlx-whisper = [ "mlx-whisper~=0.4.2" ]
|
||||||
moondream = [ "accelerate~=1.10.0", "einops~=0.8.0", "pyvips[binary]~=3.0.0", "timm~=1.0.13", "transformers>=4.48.0" ]
|
moondream = [ "accelerate~=1.10.0", "einops~=0.8.0", "pyvips[binary]~=3.0.0", "timm~=1.0.13", "transformers>=4.48.0" ]
|
||||||
neuphonic = [ "pipecat-ai[websockets-base]" ]
|
|
||||||
nim = []
|
nim = []
|
||||||
|
neuphonic = [ "pipecat-ai[websockets-base]" ]
|
||||||
noisereduce = [ "noisereduce~=3.0.3" ]
|
noisereduce = [ "noisereduce~=3.0.3" ]
|
||||||
openai = [ "pipecat-ai[websockets-base]" ]
|
openai = [ "pipecat-ai[websockets-base]" ]
|
||||||
openpipe = [ "openpipe>=4.50.0,<6" ]
|
openpipe = [ "openpipe>=4.50.0,<6" ]
|
||||||
@@ -91,16 +89,17 @@ openrouter = []
|
|||||||
perplexity = []
|
perplexity = []
|
||||||
playht = [ "pipecat-ai[websockets-base]" ]
|
playht = [ "pipecat-ai[websockets-base]" ]
|
||||||
qwen = []
|
qwen = []
|
||||||
remote-smart-turn = []
|
|
||||||
rime = [ "pipecat-ai[websockets-base]" ]
|
rime = [ "pipecat-ai[websockets-base]" ]
|
||||||
riva = [ "nvidia-riva-client~=2.21.1" ]
|
riva = [ "nvidia-riva-client~=2.21.1" ]
|
||||||
runner = [ "python-dotenv>=1.0.0,<2.0.0", "uvicorn>=0.32.0,<1.0.0", "fastapi>=0.115.6,<0.122.0", "pipecat-ai-small-webrtc-prebuilt>=1.0.0"]
|
runner = [ "python-dotenv>=1.0.0,<2.0.0", "uvicorn>=0.32.0,<1.0.0", "fastapi>=0.115.6,<0.122.0", "pipecat-ai-small-webrtc-prebuilt>=1.0.0"]
|
||||||
sagemaker = ["aws_sdk_sagemaker_runtime_http2; python_version>='3.12'"]
|
|
||||||
sambanova = []
|
sambanova = []
|
||||||
sarvam = [ "sarvamai==0.1.21", "pipecat-ai[websockets-base]" ]
|
sarvam = [ "sarvamai==0.1.21", "pipecat-ai[websockets-base]" ]
|
||||||
sentry = [ "sentry-sdk>=2.28.0,<3" ]
|
sentry = [ "sentry-sdk>=2.28.0,<3" ]
|
||||||
|
local-smart-turn = [ "coremltools>=8.0", "transformers", "torch>=2.5.0,<3", "torchaudio>=2.5.0,<3" ]
|
||||||
|
local-smart-turn-v3 = [ "transformers", "onnxruntime>=1.20.1,<2" ]
|
||||||
|
remote-smart-turn = []
|
||||||
silero = [ "onnxruntime>=1.20.1,<2" ]
|
silero = [ "onnxruntime>=1.20.1,<2" ]
|
||||||
simli = [ "simli-ai~=1.0.3"]
|
simli = [ "simli-ai~=0.1.25"]
|
||||||
soniox = [ "pipecat-ai[websockets-base]" ]
|
soniox = [ "pipecat-ai[websockets-base]" ]
|
||||||
soundfile = [ "soundfile~=0.13.1" ]
|
soundfile = [ "soundfile~=0.13.1" ]
|
||||||
speechmatics = [ "speechmatics-rt>=0.5.0" ]
|
speechmatics = [ "speechmatics-rt>=0.5.0" ]
|
||||||
|
|||||||
@@ -30,8 +30,8 @@ EVAL_SIMPLE_MATH = EvalConfig(
|
|||||||
)
|
)
|
||||||
|
|
||||||
EVAL_WEATHER = EvalConfig(
|
EVAL_WEATHER = EvalConfig(
|
||||||
prompt="What's the weather in San Francisco (in farhenheit or celsius)?",
|
prompt="What's the weather in San Francisco?",
|
||||||
eval="The user says something specific about the current weather in San Francisco, including the degrees (in farhenheit or celsius).",
|
eval="The user says something specific about the current weather in San Francisco, including the degrees.",
|
||||||
)
|
)
|
||||||
|
|
||||||
EVAL_ONLINE_SEARCH = EvalConfig(
|
EVAL_ONLINE_SEARCH = EvalConfig(
|
||||||
@@ -70,7 +70,7 @@ EVAL_VOICEMAIL = EvalConfig(
|
|||||||
|
|
||||||
EVAL_CONVERSATION = EvalConfig(
|
EVAL_CONVERSATION = EvalConfig(
|
||||||
prompt="Hello, this is Mark.",
|
prompt="Hello, this is Mark.",
|
||||||
eval="The user acknowledges the greeting.",
|
eval="The user replies with a greeting.",
|
||||||
eval_speaks_first=True,
|
eval_speaks_first=True,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -152,7 +152,7 @@ class IVRProcessor(FrameProcessor):
|
|||||||
result = await self._aggregator.aggregate(frame.text)
|
result = await self._aggregator.aggregate(frame.text)
|
||||||
if result:
|
if result:
|
||||||
# Push aggregated text that doesn't contain XML patterns
|
# Push aggregated text that doesn't contain XML patterns
|
||||||
await self.push_frame(LLMTextFrame(result.text), direction)
|
await self.push_frame(LLMTextFrame(result), direction)
|
||||||
|
|
||||||
else:
|
else:
|
||||||
await self.push_frame(frame, direction)
|
await self.push_frame(frame, direction)
|
||||||
|
|||||||
@@ -327,19 +327,23 @@ class TextFrame(DataFrame):
|
|||||||
|
|
||||||
Parameters:
|
Parameters:
|
||||||
text: The text content.
|
text: The text content.
|
||||||
skip_tts: Whether this text should skip TTS processing.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
text: str
|
text: str
|
||||||
skip_tts: bool = field(default=False, kw_only=True)
|
skip_tts: bool = field(init=False)
|
||||||
# Whether any necessary inter-frame (leading/trailing) spaces are already
|
# Whether any necessary inter-frame (leading/trailing) spaces are already
|
||||||
# included in the text.
|
# included in the text.
|
||||||
|
# NOTE: Ideally this would be available at init time with a default value,
|
||||||
|
# but that would impact how subclasses can be initialized (it would require
|
||||||
|
# mandatory fields of theirs to have defaults to preserve
|
||||||
|
# non-default-before-default argument order)
|
||||||
includes_inter_frame_spaces: bool = field(init=False)
|
includes_inter_frame_spaces: bool = field(init=False)
|
||||||
# Whether this text frame should be appended to the LLM context.
|
# Whether this text frame should be appended to the LLM context.
|
||||||
append_to_context: bool = field(init=False)
|
append_to_context: bool = field(init=False)
|
||||||
|
|
||||||
def __post_init__(self):
|
def __post_init__(self):
|
||||||
super().__post_init__()
|
super().__post_init__()
|
||||||
|
self.skip_tts = False
|
||||||
self.includes_inter_frame_spaces = False
|
self.includes_inter_frame_spaces = False
|
||||||
self.append_to_context = True
|
self.append_to_context = True
|
||||||
|
|
||||||
@@ -352,10 +356,7 @@ class TextFrame(DataFrame):
|
|||||||
class LLMTextFrame(TextFrame):
|
class LLMTextFrame(TextFrame):
|
||||||
"""Text frame generated by LLM services."""
|
"""Text frame generated by LLM services."""
|
||||||
|
|
||||||
def __post_init__(self):
|
pass
|
||||||
super().__post_init__()
|
|
||||||
# LLM services send text frames with all necessary spaces included
|
|
||||||
self.includes_inter_frame_spaces = True
|
|
||||||
|
|
||||||
|
|
||||||
class AggregationType(str, Enum):
|
class AggregationType(str, Enum):
|
||||||
@@ -831,13 +832,11 @@ class ErrorFrame(SystemFrame):
|
|||||||
error: Description of the error that occurred.
|
error: Description of the error that occurred.
|
||||||
fatal: Whether the error is fatal and requires bot shutdown.
|
fatal: Whether the error is fatal and requires bot shutdown.
|
||||||
processor: The frame processor that generated the error.
|
processor: The frame processor that generated the error.
|
||||||
exception: The exception that occurred.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
error: str
|
error: str
|
||||||
fatal: bool = False
|
fatal: bool = False
|
||||||
processor: Optional["FrameProcessor"] = None
|
processor: Optional["FrameProcessor"] = None
|
||||||
exception: Optional[Exception] = None
|
|
||||||
|
|
||||||
def __str__(self):
|
def __str__(self):
|
||||||
return f"{self.name}(error: {self.error}, fatal: {self.fatal})"
|
return f"{self.name}(error: {self.error}, fatal: {self.fatal})"
|
||||||
@@ -1626,23 +1625,24 @@ class LLMFullResponseStartFrame(ControlFrame):
|
|||||||
|
|
||||||
Used to indicate the beginning of an LLM response. Followed by one or
|
Used to indicate the beginning of an LLM response. Followed by one or
|
||||||
more TextFrames and a final LLMFullResponseEndFrame.
|
more TextFrames and a final LLMFullResponseEndFrame.
|
||||||
|
|
||||||
Parameters:
|
|
||||||
skip_tts: Whether LLM output should skip TTS processing.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
skip_tts: bool = field(default=False, kw_only=True)
|
skip_tts: bool = field(init=False)
|
||||||
|
|
||||||
|
def __post_init__(self):
|
||||||
|
super().__post_init__()
|
||||||
|
self.skip_tts = False
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class LLMFullResponseEndFrame(ControlFrame):
|
class LLMFullResponseEndFrame(ControlFrame):
|
||||||
"""Frame indicating the end of an LLM response.
|
"""Frame indicating the end of an LLM response."""
|
||||||
|
|
||||||
Parameters:
|
skip_tts: bool = field(init=False)
|
||||||
skip_tts: Whether LLM output should skip TTS processing.
|
|
||||||
"""
|
|
||||||
|
|
||||||
skip_tts: bool = field(default=False, kw_only=True)
|
def __post_init__(self):
|
||||||
|
super().__post_init__()
|
||||||
|
self.skip_tts = False
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
|
|||||||
@@ -14,7 +14,6 @@ translation from this universal context into whatever format it needs, using a
|
|||||||
service-specific adapter.
|
service-specific adapter.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import base64
|
import base64
|
||||||
import io
|
import io
|
||||||
import wave
|
import wave
|
||||||
@@ -138,7 +137,7 @@ class LLMContext:
|
|||||||
return {"role": role, "content": content}
|
return {"role": role, "content": content}
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
async def create_image_message(
|
def create_image_message(
|
||||||
*,
|
*,
|
||||||
role: str = "user",
|
role: str = "user",
|
||||||
format: str,
|
format: str,
|
||||||
@@ -155,21 +154,15 @@ class LLMContext:
|
|||||||
image: Raw image bytes.
|
image: Raw image bytes.
|
||||||
text: Optional text to include with the image.
|
text: Optional text to include with the image.
|
||||||
"""
|
"""
|
||||||
|
buffer = io.BytesIO()
|
||||||
def encode_image():
|
Image.frombytes(format, size, image).save(buffer, format="JPEG")
|
||||||
buffer = io.BytesIO()
|
encoded_image = base64.b64encode(buffer.getvalue()).decode("utf-8")
|
||||||
Image.frombytes(format, size, image).save(buffer, format="JPEG")
|
|
||||||
encoded_image = base64.b64encode(buffer.getvalue()).decode("utf-8")
|
|
||||||
return encoded_image
|
|
||||||
|
|
||||||
encoded_image = await asyncio.to_thread(encode_image)
|
|
||||||
|
|
||||||
url = f"data:image/jpeg;base64,{encoded_image}"
|
url = f"data:image/jpeg;base64,{encoded_image}"
|
||||||
|
|
||||||
return LLMContext.create_image_url_message(role=role, url=url, text=text)
|
return LLMContext.create_image_url_message(role=role, url=url, text=text)
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
async def create_audio_message(
|
def create_audio_message(
|
||||||
*, role: str = "user", audio_frames: list[AudioRawFrame], text: str = "Audio follows"
|
*, role: str = "user", audio_frames: list[AudioRawFrame], text: str = "Audio follows"
|
||||||
) -> LLMContextMessage:
|
) -> LLMContextMessage:
|
||||||
"""Create a context message containing audio.
|
"""Create a context message containing audio.
|
||||||
@@ -179,26 +172,21 @@ class LLMContext:
|
|||||||
audio_frames: List of audio frame objects to include.
|
audio_frames: List of audio frame objects to include.
|
||||||
text: Optional text to include with the audio.
|
text: Optional text to include with the audio.
|
||||||
"""
|
"""
|
||||||
|
sample_rate = audio_frames[0].sample_rate
|
||||||
|
num_channels = audio_frames[0].num_channels
|
||||||
|
|
||||||
async def encode_audio():
|
content = []
|
||||||
sample_rate = audio_frames[0].sample_rate
|
content.append({"type": "text", "text": text})
|
||||||
num_channels = audio_frames[0].num_channels
|
data = b"".join(frame.audio for frame in audio_frames)
|
||||||
|
|
||||||
content = []
|
with io.BytesIO() as buffer:
|
||||||
content.append({"type": "text", "text": text})
|
with wave.open(buffer, "wb") as wf:
|
||||||
data = b"".join(frame.audio for frame in audio_frames)
|
wf.setsampwidth(2)
|
||||||
|
wf.setnchannels(num_channels)
|
||||||
|
wf.setframerate(sample_rate)
|
||||||
|
wf.writeframes(data)
|
||||||
|
|
||||||
with io.BytesIO() as buffer:
|
encoded_audio = base64.b64encode(buffer.getvalue()).decode("utf-8")
|
||||||
with wave.open(buffer, "wb") as wf:
|
|
||||||
wf.setsampwidth(2)
|
|
||||||
wf.setnchannels(num_channels)
|
|
||||||
wf.setframerate(sample_rate)
|
|
||||||
wf.writeframes(data)
|
|
||||||
|
|
||||||
encoded_audio = base64.b64encode(buffer.getvalue()).decode("utf-8")
|
|
||||||
return encoded_audio
|
|
||||||
|
|
||||||
encoded_audio = await asyncio.to_thread(encode_audio)
|
|
||||||
|
|
||||||
content.append(
|
content.append(
|
||||||
{
|
{
|
||||||
@@ -333,7 +321,7 @@ class LLMContext:
|
|||||||
"""
|
"""
|
||||||
self._tool_choice = tool_choice
|
self._tool_choice = tool_choice
|
||||||
|
|
||||||
async def add_image_frame_message(
|
def add_image_frame_message(
|
||||||
self, *, format: str, size: tuple[int, int], image: bytes, text: Optional[str] = None
|
self, *, format: str, size: tuple[int, int], image: bytes, text: Optional[str] = None
|
||||||
):
|
):
|
||||||
"""Add a message containing an image frame.
|
"""Add a message containing an image frame.
|
||||||
@@ -344,12 +332,10 @@ class LLMContext:
|
|||||||
image: Raw image bytes.
|
image: Raw image bytes.
|
||||||
text: Optional text to include with the image.
|
text: Optional text to include with the image.
|
||||||
"""
|
"""
|
||||||
message = await LLMContext.create_image_message(
|
message = LLMContext.create_image_message(format=format, size=size, image=image, text=text)
|
||||||
format=format, size=size, image=image, text=text
|
|
||||||
)
|
|
||||||
self.add_message(message)
|
self.add_message(message)
|
||||||
|
|
||||||
async def add_audio_frames_message(
|
def add_audio_frames_message(
|
||||||
self, *, audio_frames: list[AudioRawFrame], text: str = "Audio follows"
|
self, *, audio_frames: list[AudioRawFrame], text: str = "Audio follows"
|
||||||
):
|
):
|
||||||
"""Add a message containing audio frames.
|
"""Add a message containing audio frames.
|
||||||
@@ -358,7 +344,7 @@ class LLMContext:
|
|||||||
audio_frames: List of audio frame objects to include.
|
audio_frames: List of audio frame objects to include.
|
||||||
text: Optional text to include with the audio.
|
text: Optional text to include with the audio.
|
||||||
"""
|
"""
|
||||||
message = await LLMContext.create_audio_message(audio_frames=audio_frames, text=text)
|
message = LLMContext.create_audio_message(audio_frames=audio_frames, text=text)
|
||||||
self.add_message(message)
|
self.add_message(message)
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
|
|||||||
@@ -66,7 +66,7 @@ from pipecat.processors.aggregators.llm_response import (
|
|||||||
LLMUserAggregatorParams,
|
LLMUserAggregatorParams,
|
||||||
)
|
)
|
||||||
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
|
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
|
||||||
from pipecat.utils.string import TextPartForConcatenation, concatenate_aggregated_text
|
from pipecat.utils.string import concatenate_aggregated_text
|
||||||
from pipecat.utils.time import time_now_iso8601
|
from pipecat.utils.time import time_now_iso8601
|
||||||
|
|
||||||
|
|
||||||
@@ -90,7 +90,15 @@ class LLMContextAggregator(FrameProcessor):
|
|||||||
self._context = context
|
self._context = context
|
||||||
self._role = role
|
self._role = role
|
||||||
|
|
||||||
self._aggregation: List[TextPartForConcatenation] = []
|
self._aggregation: List[str] = []
|
||||||
|
|
||||||
|
# Whether to add spaces between text parts.
|
||||||
|
# (Currently only used by LLMAssistantAggregator, but could be expanded
|
||||||
|
# to LLMUserAggregator in the future if needed; that would require
|
||||||
|
# additional work since LLMUserAggregator currently trims spaces from
|
||||||
|
# incoming frames before determining whether it "really" received any
|
||||||
|
# text).
|
||||||
|
self._add_spaces = True
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def messages(self) -> List[LLMContextMessage]:
|
def messages(self) -> List[LLMContextMessage]:
|
||||||
@@ -183,7 +191,7 @@ class LLMContextAggregator(FrameProcessor):
|
|||||||
Returns:
|
Returns:
|
||||||
The concatenated aggregation string.
|
The concatenated aggregation string.
|
||||||
"""
|
"""
|
||||||
return concatenate_aggregated_text(self._aggregation)
|
return concatenate_aggregated_text(self._aggregation, self._add_spaces)
|
||||||
|
|
||||||
|
|
||||||
class LLMUserAggregator(LLMContextAggregator):
|
class LLMUserAggregator(LLMContextAggregator):
|
||||||
@@ -433,12 +441,7 @@ class LLMUserAggregator(LLMContextAggregator):
|
|||||||
if not text.strip():
|
if not text.strip():
|
||||||
return
|
return
|
||||||
|
|
||||||
# Transcriptions never include inter-part spaces (so far).
|
self._aggregation.append(text)
|
||||||
self._aggregation.append(
|
|
||||||
TextPartForConcatenation(
|
|
||||||
text, includes_inter_part_spaces=frame.includes_inter_frame_spaces
|
|
||||||
)
|
|
||||||
)
|
|
||||||
# We just got a final result, so let's reset interim results.
|
# We just got a final result, so let's reset interim results.
|
||||||
self._seen_interim_results = False
|
self._seen_interim_results = False
|
||||||
# Reset aggregation timer.
|
# Reset aggregation timer.
|
||||||
@@ -793,7 +796,7 @@ class LLMAssistantAggregator(LLMContextAggregator):
|
|||||||
|
|
||||||
logger.debug(f"{self} Appending UserImageRawFrame to LLM context (size: {frame.size})")
|
logger.debug(f"{self} Appending UserImageRawFrame to LLM context (size: {frame.size})")
|
||||||
|
|
||||||
await self._context.add_image_frame_message(
|
self._context.add_image_frame_message(
|
||||||
format=frame.format,
|
format=frame.format,
|
||||||
size=frame.size,
|
size=frame.size,
|
||||||
image=frame.image,
|
image=frame.image,
|
||||||
@@ -818,11 +821,11 @@ class LLMAssistantAggregator(LLMContextAggregator):
|
|||||||
if len(frame.text) == 0:
|
if len(frame.text) == 0:
|
||||||
return
|
return
|
||||||
|
|
||||||
self._aggregation.append(
|
# Track whether we need to add spaces between text parts
|
||||||
TextPartForConcatenation(
|
# Assumption: we can just keep track of the latest frame's value
|
||||||
frame.text, includes_inter_part_spaces=frame.includes_inter_frame_spaces
|
self._add_spaces = not frame.includes_inter_frame_spaces
|
||||||
)
|
|
||||||
)
|
self._aggregation.append(frame.text)
|
||||||
|
|
||||||
def _context_updated_task_finished(self, task: asyncio.Task):
|
def _context_updated_task_finished(self, task: asyncio.Task):
|
||||||
self._context_updated_tasks.discard(task)
|
self._context_updated_tasks.discard(task)
|
||||||
|
|||||||
@@ -83,4 +83,4 @@ class ConsumerProcessor(FrameProcessor):
|
|||||||
while True:
|
while True:
|
||||||
frame = await self._queue.get()
|
frame = await self._queue.get()
|
||||||
new_frame = await self._transformer(frame)
|
new_frame = await self._transformer(frame)
|
||||||
await self.queue_frame(new_frame, self._direction)
|
await self.push_frame(new_frame, self._direction)
|
||||||
|
|||||||
@@ -126,4 +126,6 @@ class WakeCheckFilter(FrameProcessor):
|
|||||||
else:
|
else:
|
||||||
await self.push_frame(frame, direction)
|
await self.push_frame(frame, direction)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error in wake word filter: {e}", exception=e)
|
error_msg = f"Error in wake word filter: {e}"
|
||||||
|
logger.exception(error_msg)
|
||||||
|
await self.push_error(ErrorFrame(error_msg))
|
||||||
|
|||||||
@@ -142,7 +142,6 @@ class FrameProcessor(BaseObject):
|
|||||||
- on_after_process_frame: Called after a frame is processed
|
- on_after_process_frame: Called after a frame is processed
|
||||||
- on_before_push_frame: Called before a frame is pushed
|
- on_before_push_frame: Called before a frame is pushed
|
||||||
- on_after_push_frame: Called after a frame is pushed
|
- on_after_push_frame: Called after a frame is pushed
|
||||||
- on_error: Called when an error is raised in the frame processing.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
@@ -235,7 +234,6 @@ class FrameProcessor(BaseObject):
|
|||||||
self._register_event_handler("on_after_process_frame", sync=True)
|
self._register_event_handler("on_after_process_frame", sync=True)
|
||||||
self._register_event_handler("on_before_push_frame", sync=True)
|
self._register_event_handler("on_before_push_frame", sync=True)
|
||||||
self._register_event_handler("on_after_push_frame", sync=True)
|
self._register_event_handler("on_after_push_frame", sync=True)
|
||||||
self._register_event_handler("on_error", sync=True)
|
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def id(self) -> int:
|
def id(self) -> int:
|
||||||
@@ -632,43 +630,7 @@ class FrameProcessor(BaseObject):
|
|||||||
elif isinstance(frame, (FrameProcessorResumeFrame, FrameProcessorResumeUrgentFrame)):
|
elif isinstance(frame, (FrameProcessorResumeFrame, FrameProcessorResumeUrgentFrame)):
|
||||||
await self.__resume(frame)
|
await self.__resume(frame)
|
||||||
|
|
||||||
async def push_error(
|
async def push_error(self, error: ErrorFrame):
|
||||||
self,
|
|
||||||
error_msg: str,
|
|
||||||
exception: Optional[Exception] = None,
|
|
||||||
fatal: bool = False,
|
|
||||||
):
|
|
||||||
"""Creates and pushes an ErrorFrame upstream.
|
|
||||||
|
|
||||||
Creates and pushes an ErrorFrame upstream to notify other processors in the
|
|
||||||
pipeline about an error condition. The error frame will include context about
|
|
||||||
which processor generated the error.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
error_msg: Descriptive message explaining the error condition.
|
|
||||||
exception: Optional exception object that caused the error, if available.
|
|
||||||
This provides additional context for debugging and error handling.
|
|
||||||
fatal: Whether this error should be considered fatal to the pipeline.
|
|
||||||
Fatal errors typically cause the entire pipeline to stop processing.
|
|
||||||
Defaults to False for non-fatal errors.
|
|
||||||
|
|
||||||
Example::
|
|
||||||
|
|
||||||
```python
|
|
||||||
# Non-fatal error
|
|
||||||
await self.push_error("Failed to process audio chunk, skipping")
|
|
||||||
|
|
||||||
# Fatal error with exception context
|
|
||||||
try:
|
|
||||||
result = some_critical_operation()
|
|
||||||
except Exception as e:
|
|
||||||
await self.push_error("Critical operation failed", exception=e, fatal=True)
|
|
||||||
```
|
|
||||||
"""
|
|
||||||
error_frame = ErrorFrame(error=error_msg, fatal=fatal, exception=exception, processor=self)
|
|
||||||
await self.push_error_frame(error=error_frame)
|
|
||||||
|
|
||||||
async def push_error_frame(self, error: ErrorFrame):
|
|
||||||
"""Push an error frame upstream.
|
"""Push an error frame upstream.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
@@ -676,8 +638,6 @@ class FrameProcessor(BaseObject):
|
|||||||
"""
|
"""
|
||||||
if not error.processor:
|
if not error.processor:
|
||||||
error.processor = self
|
error.processor = self
|
||||||
await self._call_event_handler("on_error", error)
|
|
||||||
logger.error(f"{error.processor} error: {error.error}")
|
|
||||||
await self.push_frame(error, FrameDirection.UPSTREAM)
|
await self.push_frame(error, FrameDirection.UPSTREAM)
|
||||||
|
|
||||||
async def push_frame(self, frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM):
|
async def push_frame(self, frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM):
|
||||||
@@ -799,10 +759,8 @@ class FrameProcessor(BaseObject):
|
|||||||
await self.__cancel_process_task()
|
await self.__cancel_process_task()
|
||||||
self.__create_process_task()
|
self.__create_process_task()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(
|
logger.exception(f"Uncaught exception in {self} when handling _start_interruption: {e}")
|
||||||
error_msg=f"Uncaught exception handling _start_interruption: {e}",
|
await self.push_error(ErrorFrame(str(e)))
|
||||||
exception=e,
|
|
||||||
)
|
|
||||||
|
|
||||||
async def __internal_push_frame(self, frame: Frame, direction: FrameDirection):
|
async def __internal_push_frame(self, frame: Frame, direction: FrameDirection):
|
||||||
"""Internal method to push frames to adjacent processors.
|
"""Internal method to push frames to adjacent processors.
|
||||||
@@ -839,7 +797,8 @@ class FrameProcessor(BaseObject):
|
|||||||
await self._observer.on_push_frame(data)
|
await self._observer.on_push_frame(data)
|
||||||
await self._prev.queue_frame(frame, direction)
|
await self._prev.queue_frame(frame, direction)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Uncaught exception: {e}", exception=e)
|
logger.exception(f"Uncaught exception in {self}: {e}")
|
||||||
|
await self.push_error(ErrorFrame(str(e)))
|
||||||
|
|
||||||
def _check_started(self, frame: Frame):
|
def _check_started(self, frame: Frame):
|
||||||
"""Check if the processor has been started.
|
"""Check if the processor has been started.
|
||||||
@@ -915,7 +874,8 @@ class FrameProcessor(BaseObject):
|
|||||||
|
|
||||||
await self._call_event_handler("on_after_process_frame", frame)
|
await self._call_event_handler("on_after_process_frame", frame)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error processing frame: {e}", exception=e)
|
logger.exception(f"{self}: error processing frame: {e}")
|
||||||
|
await self.push_error(ErrorFrame(str(e)))
|
||||||
|
|
||||||
async def __input_frame_task_handler(self):
|
async def __input_frame_task_handler(self):
|
||||||
"""Handle frames from the input queue.
|
"""Handle frames from the input queue.
|
||||||
|
|||||||
@@ -24,7 +24,7 @@ try:
|
|||||||
from langchain_core.messages import AIMessageChunk
|
from langchain_core.messages import AIMessageChunk
|
||||||
from langchain_core.runnables import Runnable
|
from langchain_core.runnables import Runnable
|
||||||
except ModuleNotFoundError as e:
|
except ModuleNotFoundError as e:
|
||||||
logger.error("In order to use Langchain, you need to `pip install pipecat-ai[langchain]`. ")
|
logger.exception("In order to use Langchain, you need to `pip install pipecat-ai[langchain]`. ")
|
||||||
raise Exception(f"Missing module: {e}")
|
raise Exception(f"Missing module: {e}")
|
||||||
|
|
||||||
|
|
||||||
@@ -113,6 +113,6 @@ class LangchainProcessor(FrameProcessor):
|
|||||||
except GeneratorExit:
|
except GeneratorExit:
|
||||||
logger.warning(f"{self} generator was closed prematurely")
|
logger.warning(f"{self} generator was closed prematurely")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.exception(f"{self} an unknown error occurred: {e}")
|
||||||
finally:
|
finally:
|
||||||
await self.push_frame(LLMFullResponseEndFrame())
|
await self.push_frame(LLMFullResponseEndFrame())
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ try:
|
|||||||
from strands import Agent
|
from strands import Agent
|
||||||
from strands.multiagent.graph import Graph
|
from strands.multiagent.graph import Graph
|
||||||
except ModuleNotFoundError as e:
|
except ModuleNotFoundError as e:
|
||||||
logger.error("In order to use Strands Agents, you need to `pip install strands-agents`.")
|
logger.exception("In order to use Strands Agents, you need to `pip install strands-agents`.")
|
||||||
raise Exception(f"Missing module: {e}")
|
raise Exception(f"Missing module: {e}")
|
||||||
|
|
||||||
|
|
||||||
@@ -143,7 +143,7 @@ class StrandsAgentsProcessor(FrameProcessor):
|
|||||||
except GeneratorExit:
|
except GeneratorExit:
|
||||||
logger.warning(f"{self} generator was closed prematurely")
|
logger.warning(f"{self} generator was closed prematurely")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.exception(f"{self} an unknown error occurred: {e}")
|
||||||
finally:
|
finally:
|
||||||
if ttfb_tracking:
|
if ttfb_tracking:
|
||||||
await self.stop_ttfb_metrics()
|
await self.stop_ttfb_metrics()
|
||||||
|
|||||||
@@ -26,7 +26,7 @@ from pipecat.frames.frames import (
|
|||||||
TTSTextFrame,
|
TTSTextFrame,
|
||||||
)
|
)
|
||||||
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
|
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
|
||||||
from pipecat.utils.string import TextPartForConcatenation, concatenate_aggregated_text
|
from pipecat.utils.string import concatenate_aggregated_text
|
||||||
from pipecat.utils.time import time_now_iso8601
|
from pipecat.utils.time import time_now_iso8601
|
||||||
|
|
||||||
|
|
||||||
@@ -98,9 +98,15 @@ class AssistantTranscriptProcessor(BaseTranscriptProcessor):
|
|||||||
**kwargs: Additional arguments passed to parent class.
|
**kwargs: Additional arguments passed to parent class.
|
||||||
"""
|
"""
|
||||||
super().__init__(**kwargs)
|
super().__init__(**kwargs)
|
||||||
self._current_text_parts: List[TextPartForConcatenation] = []
|
self._current_text_parts: List[str] = []
|
||||||
self._aggregation_start_time: Optional[str] = None
|
self._aggregation_start_time: Optional[str] = None
|
||||||
|
|
||||||
|
# Whether to add spaces between text parts.
|
||||||
|
# (The use of this could be expanded to the UserTranscriptProcessor in
|
||||||
|
# the future if needed; currently the UserTranscriptProcessor assumes
|
||||||
|
# that user transcription frames do not need aggregation).
|
||||||
|
self._add_spaces = True
|
||||||
|
|
||||||
async def _emit_aggregated_text(self):
|
async def _emit_aggregated_text(self):
|
||||||
"""Aggregates and emits text fragments as a transcript message.
|
"""Aggregates and emits text fragments as a transcript message.
|
||||||
|
|
||||||
@@ -141,7 +147,7 @@ class AssistantTranscriptProcessor(BaseTranscriptProcessor):
|
|||||||
Result: "Hello there how are you"
|
Result: "Hello there how are you"
|
||||||
"""
|
"""
|
||||||
if self._current_text_parts and self._aggregation_start_time:
|
if self._current_text_parts and self._aggregation_start_time:
|
||||||
content = concatenate_aggregated_text(self._current_text_parts)
|
content = concatenate_aggregated_text(self._current_text_parts, self._add_spaces)
|
||||||
if content:
|
if content:
|
||||||
logger.trace(f"Emitting aggregated assistant message: {content}")
|
logger.trace(f"Emitting aggregated assistant message: {content}")
|
||||||
message = TranscriptionMessage(
|
message = TranscriptionMessage(
|
||||||
@@ -185,11 +191,11 @@ class AssistantTranscriptProcessor(BaseTranscriptProcessor):
|
|||||||
if not self._aggregation_start_time:
|
if not self._aggregation_start_time:
|
||||||
self._aggregation_start_time = time_now_iso8601()
|
self._aggregation_start_time = time_now_iso8601()
|
||||||
|
|
||||||
self._current_text_parts.append(
|
# Track whether we need to add spaces between text parts
|
||||||
TextPartForConcatenation(
|
# Assumption: we can just keep track of the latest frame's value
|
||||||
frame.text, includes_inter_part_spaces=frame.includes_inter_frame_spaces
|
self._add_spaces = not frame.includes_inter_frame_spaces
|
||||||
)
|
|
||||||
)
|
self._current_text_parts.append(frame.text)
|
||||||
|
|
||||||
# Push frame.
|
# Push frame.
|
||||||
await self.push_frame(frame, direction)
|
await self.push_frame(frame, direction)
|
||||||
|
|||||||
@@ -264,10 +264,7 @@ def _setup_webrtc_routes(
|
|||||||
# Prepare runner arguments with the callback to run your bot
|
# Prepare runner arguments with the callback to run your bot
|
||||||
async def webrtc_connection_callback(connection):
|
async def webrtc_connection_callback(connection):
|
||||||
bot_module = _get_bot_module()
|
bot_module = _get_bot_module()
|
||||||
|
runner_args = SmallWebRTCRunnerArguments(webrtc_connection=connection)
|
||||||
runner_args = SmallWebRTCRunnerArguments(
|
|
||||||
webrtc_connection=connection, body=request.request_data
|
|
||||||
)
|
|
||||||
background_tasks.add_task(bot_module.bot, runner_args)
|
background_tasks.add_task(bot_module.bot, runner_args)
|
||||||
|
|
||||||
# Delegate handling to SmallWebRTCRequestHandler
|
# Delegate handling to SmallWebRTCRequestHandler
|
||||||
@@ -329,8 +326,7 @@ def _setup_webrtc_routes(
|
|||||||
type=request_data["type"],
|
type=request_data["type"],
|
||||||
pc_id=request_data.get("pc_id"),
|
pc_id=request_data.get("pc_id"),
|
||||||
restart_pc=request_data.get("restart_pc"),
|
restart_pc=request_data.get("restart_pc"),
|
||||||
request_data=request_data.get("request_data")
|
request_data=request_data,
|
||||||
or request_data.get("requestData"),
|
|
||||||
)
|
)
|
||||||
return await offer(webrtc_request, background_tasks)
|
return await offer(webrtc_request, background_tasks)
|
||||||
elif request.method == HTTPMethod.PATCH.value:
|
elif request.method == HTTPMethod.PATCH.value:
|
||||||
|
|||||||
@@ -281,14 +281,6 @@ async def maybe_capture_participant_camera(
|
|||||||
except ImportError:
|
except ImportError:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
try:
|
|
||||||
from pipecat.transports.smallwebrtc.transport import SmallWebRTCTransport
|
|
||||||
|
|
||||||
if isinstance(transport, SmallWebRTCTransport):
|
|
||||||
await transport.capture_participant_video(video_source="camera")
|
|
||||||
except ImportError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
|
|
||||||
async def maybe_capture_participant_screen(
|
async def maybe_capture_participant_screen(
|
||||||
transport: BaseTransport, client: Any, framerate: int = 0
|
transport: BaseTransport, client: Any, framerate: int = 0
|
||||||
@@ -311,14 +303,6 @@ async def maybe_capture_participant_screen(
|
|||||||
except ImportError:
|
except ImportError:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
try:
|
|
||||||
from pipecat.transports.smallwebrtc.transport import SmallWebRTCTransport
|
|
||||||
|
|
||||||
if isinstance(transport, SmallWebRTCTransport):
|
|
||||||
await transport.capture_participant_video(video_source="screenVideo")
|
|
||||||
except ImportError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
|
|
||||||
def _smallwebrtc_sdp_cleanup_ice_candidates(text: str, pattern: str) -> str:
|
def _smallwebrtc_sdp_cleanup_ice_candidates(text: str, pattern: str) -> str:
|
||||||
"""Clean up ICE candidates in SDP text for SmallWebRTC.
|
"""Clean up ICE candidates in SDP text for SmallWebRTC.
|
||||||
|
|||||||
@@ -199,7 +199,7 @@ class PlivoFrameSerializer(FrameSerializer):
|
|||||||
)
|
)
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Failed to hang up Plivo call: {e}")
|
logger.exception(f"Failed to hang up Plivo call: {e}")
|
||||||
|
|
||||||
async def deserialize(self, data: str | bytes) -> Frame | None:
|
async def deserialize(self, data: str | bytes) -> Frame | None:
|
||||||
"""Deserializes Plivo WebSocket data to Pipecat frames.
|
"""Deserializes Plivo WebSocket data to Pipecat frames.
|
||||||
|
|||||||
@@ -225,7 +225,7 @@ class TelnyxFrameSerializer(FrameSerializer):
|
|||||||
)
|
)
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Failed to hang up Telnyx call: {e}")
|
logger.exception(f"Failed to hang up Telnyx call: {e}")
|
||||||
|
|
||||||
async def deserialize(self, data: str | bytes) -> Frame | None:
|
async def deserialize(self, data: str | bytes) -> Frame | None:
|
||||||
"""Deserializes Telnyx WebSocket data to Pipecat frames.
|
"""Deserializes Telnyx WebSocket data to Pipecat frames.
|
||||||
|
|||||||
@@ -236,7 +236,7 @@ class TwilioFrameSerializer(FrameSerializer):
|
|||||||
)
|
)
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Failed to hang up Twilio call: {e}")
|
logger.exception(f"Failed to hang up Twilio call: {e}")
|
||||||
|
|
||||||
async def deserialize(self, data: str | bytes) -> Frame | None:
|
async def deserialize(self, data: str | bytes) -> Frame | None:
|
||||||
"""Deserializes Twilio WebSocket data to Pipecat frames.
|
"""Deserializes Twilio WebSocket data to Pipecat frames.
|
||||||
|
|||||||
@@ -166,6 +166,6 @@ class AIService(FrameProcessor):
|
|||||||
async for f in generator:
|
async for f in generator:
|
||||||
if f:
|
if f:
|
||||||
if isinstance(f, ErrorFrame):
|
if isinstance(f, ErrorFrame):
|
||||||
await self.push_error_frame(f)
|
await self.push_error(f)
|
||||||
else:
|
else:
|
||||||
await self.push_frame(f)
|
await self.push_frame(f)
|
||||||
|
|||||||
@@ -327,7 +327,7 @@ class AnthropicLLMService(LLMService):
|
|||||||
cache_read_input_tokens = 0
|
cache_read_input_tokens = 0
|
||||||
|
|
||||||
try:
|
try:
|
||||||
await self.push_frame(LLMFullResponseStartFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseStartFrame())
|
||||||
await self.start_processing_metrics()
|
await self.start_processing_metrics()
|
||||||
|
|
||||||
params_from_context = self._get_llm_invocation_params(context)
|
params_from_context = self._get_llm_invocation_params(context)
|
||||||
@@ -373,9 +373,9 @@ class AnthropicLLMService(LLMService):
|
|||||||
|
|
||||||
if event.type == "content_block_delta":
|
if event.type == "content_block_delta":
|
||||||
if hasattr(event.delta, "text"):
|
if hasattr(event.delta, "text"):
|
||||||
await self.push_frame(
|
frame = LLMTextFrame(event.delta.text)
|
||||||
LLMTextFrame(event.delta.text, skip_tts=self._get_skip_tts())
|
frame.includes_inter_frame_spaces = True
|
||||||
)
|
await self.push_frame(frame)
|
||||||
completion_tokens_estimate += self._estimate_tokens(event.delta.text)
|
completion_tokens_estimate += self._estimate_tokens(event.delta.text)
|
||||||
elif hasattr(event.delta, "partial_json") and tool_use_block:
|
elif hasattr(event.delta, "partial_json") and tool_use_block:
|
||||||
json_accumulator += event.delta.partial_json
|
json_accumulator += event.delta.partial_json
|
||||||
@@ -460,10 +460,11 @@ class AnthropicLLMService(LLMService):
|
|||||||
except httpx.TimeoutException:
|
except httpx.TimeoutException:
|
||||||
await self._call_event_handler("on_completion_timeout")
|
await self._call_event_handler("on_completion_timeout")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.exception(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(f"{e}"))
|
||||||
finally:
|
finally:
|
||||||
await self.stop_processing_metrics()
|
await self.stop_processing_metrics()
|
||||||
await self.push_frame(LLMFullResponseEndFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseEndFrame())
|
||||||
comp_tokens = (
|
comp_tokens = (
|
||||||
completion_tokens
|
completion_tokens
|
||||||
if not use_completion_tokens_estimate
|
if not use_completion_tokens_estimate
|
||||||
|
|||||||
@@ -206,8 +206,9 @@ class AssemblyAISTTService(STTService):
|
|||||||
|
|
||||||
await self._call_event_handler("on_connected")
|
await self._call_event_handler("on_connected")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
|
logger.error(f"{self} exception: {e}")
|
||||||
self._connected = False
|
self._connected = False
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
raise
|
raise
|
||||||
|
|
||||||
async def _disconnect(self):
|
async def _disconnect(self):
|
||||||
@@ -232,7 +233,8 @@ class AssemblyAISTTService(STTService):
|
|||||||
logger.warning("Timed out waiting for termination message from server")
|
logger.warning("Timed out waiting for termination message from server")
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
|
|
||||||
if self._receive_task:
|
if self._receive_task:
|
||||||
await self.cancel_task(self._receive_task)
|
await self.cancel_task(self._receive_task)
|
||||||
@@ -240,7 +242,8 @@ class AssemblyAISTTService(STTService):
|
|||||||
await self._websocket.close()
|
await self._websocket.close()
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
|
|
||||||
finally:
|
finally:
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
@@ -259,11 +262,13 @@ class AssemblyAISTTService(STTService):
|
|||||||
except websockets.exceptions.ConnectionClosedOK:
|
except websockets.exceptions.ConnectionClosedOK:
|
||||||
break
|
break
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
break
|
break
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
|
|
||||||
def _parse_message(self, message: Dict[str, Any]) -> BaseMessage:
|
def _parse_message(self, message: Dict[str, Any]) -> BaseMessage:
|
||||||
"""Parse a raw message into the appropriate message type."""
|
"""Parse a raw message into the appropriate message type."""
|
||||||
@@ -292,7 +297,8 @@ class AssemblyAISTTService(STTService):
|
|||||||
elif isinstance(parsed_message, TerminationMessage):
|
elif isinstance(parsed_message, TerminationMessage):
|
||||||
await self._handle_termination(parsed_message)
|
await self._handle_termination(parsed_message)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
|
|
||||||
async def _handle_termination(self, message: TerminationMessage):
|
async def _handle_termination(self, message: TerminationMessage):
|
||||||
"""Handle termination message."""
|
"""Handle termination message."""
|
||||||
|
|||||||
@@ -146,6 +146,15 @@ class AsyncAITTSService(InterruptibleTTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that AsyncAI TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that AsyncAI's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
def language_to_service_language(self, language: Language) -> Optional[str]:
|
def language_to_service_language(self, language: Language) -> Optional[str]:
|
||||||
"""Convert a Language enum to Async language format.
|
"""Convert a Language enum to Async language format.
|
||||||
|
|
||||||
@@ -228,7 +237,8 @@ class AsyncAITTSService(InterruptibleTTSService):
|
|||||||
|
|
||||||
await self._call_event_handler("on_connected")
|
await self._call_event_handler("on_connected")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
await self._call_event_handler("on_connection_error", f"{e}")
|
await self._call_event_handler("on_connection_error", f"{e}")
|
||||||
|
|
||||||
@@ -240,7 +250,8 @@ class AsyncAITTSService(InterruptibleTTSService):
|
|||||||
logger.debug("Disconnecting from Async")
|
logger.debug("Disconnecting from Async")
|
||||||
await self._websocket.close()
|
await self._websocket.close()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
finally:
|
finally:
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
self._started = False
|
self._started = False
|
||||||
@@ -285,11 +296,12 @@ class AsyncAITTSService(InterruptibleTTSService):
|
|||||||
)
|
)
|
||||||
await self.push_frame(frame)
|
await self.push_frame(frame)
|
||||||
elif msg.get("error_code"):
|
elif msg.get("error_code"):
|
||||||
|
logger.error(f"{self} error: {msg}")
|
||||||
await self.push_frame(TTSStoppedFrame())
|
await self.push_frame(TTSStoppedFrame())
|
||||||
await self.stop_all_metrics()
|
await self.stop_all_metrics()
|
||||||
await self.push_error(error_msg=f"Error: {msg['message']}")
|
await self.push_error(ErrorFrame(error=f"{self} error: {msg['message']}"))
|
||||||
else:
|
else:
|
||||||
await self.push_error(error_msg=f"Unknown message type: {msg}")
|
logger.error(f"{self} error, unknown message type: {msg}")
|
||||||
|
|
||||||
async def _keepalive_task_handler(self):
|
async def _keepalive_task_handler(self):
|
||||||
"""Send periodic keepalive messages to maintain WebSocket connection."""
|
"""Send periodic keepalive messages to maintain WebSocket connection."""
|
||||||
@@ -332,14 +344,16 @@ class AsyncAITTSService(InterruptibleTTSService):
|
|||||||
await self._get_websocket().send(msg)
|
await self._get_websocket().send(msg)
|
||||||
await self.start_tts_usage_metrics(text)
|
await self.start_tts_usage_metrics(text)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
await self._disconnect()
|
await self._disconnect()
|
||||||
await self._connect()
|
await self._connect()
|
||||||
return
|
return
|
||||||
yield None
|
yield None
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
|
|
||||||
|
|
||||||
class AsyncAIHttpTTSService(TTSService):
|
class AsyncAIHttpTTSService(TTSService):
|
||||||
@@ -419,6 +433,15 @@ class AsyncAIHttpTTSService(TTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that AsyncAI TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that AsyncAI's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
def language_to_service_language(self, language: Language) -> Optional[str]:
|
def language_to_service_language(self, language: Language) -> Optional[str]:
|
||||||
"""Convert a Language enum to Async language format.
|
"""Convert a Language enum to Async language format.
|
||||||
|
|
||||||
@@ -472,7 +495,8 @@ class AsyncAIHttpTTSService(TTSService):
|
|||||||
async with self._session.post(url, json=payload, headers=headers) as response:
|
async with self._session.post(url, json=payload, headers=headers) as response:
|
||||||
if response.status != 200:
|
if response.status != 200:
|
||||||
error_text = await response.text()
|
error_text = await response.text()
|
||||||
await self.push_error(error_msg=f"Async API error: {error_text}")
|
logger.error(f"Async API error: {error_text}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"Async API error: {error_text}"))
|
||||||
raise Exception(f"Async API returned status {response.status}: {error_text}")
|
raise Exception(f"Async API returned status {response.status}: {error_text}")
|
||||||
|
|
||||||
audio_data = await response.read()
|
audio_data = await response.read()
|
||||||
@@ -488,7 +512,8 @@ class AsyncAIHttpTTSService(TTSService):
|
|||||||
yield frame
|
yield frame
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
finally:
|
finally:
|
||||||
await self.stop_ttfb_metrics()
|
await self.stop_ttfb_metrics()
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
|
|||||||
@@ -8,10 +8,8 @@ import sys
|
|||||||
|
|
||||||
from pipecat.services import DeprecatedModuleProxy
|
from pipecat.services import DeprecatedModuleProxy
|
||||||
|
|
||||||
from .agent_core import *
|
|
||||||
from .llm import *
|
from .llm import *
|
||||||
from .nova_sonic import *
|
from .nova_sonic import *
|
||||||
from .sagemaker import *
|
|
||||||
from .stt import *
|
from .stt import *
|
||||||
from .tts import *
|
from .tts import *
|
||||||
|
|
||||||
|
|||||||
@@ -1,258 +0,0 @@
|
|||||||
#
|
|
||||||
# Copyright (c) 2025, Daily
|
|
||||||
#
|
|
||||||
# SPDX-License-Identifier: BSD 2-Clause License
|
|
||||||
#
|
|
||||||
|
|
||||||
"""AWS AgentCore Processor Module.
|
|
||||||
|
|
||||||
This module defines the AWSAgentCoreProcessor, which invokes agents hosted on
|
|
||||||
Amazon Bedrock AgentCore Runtime and streams their responses as LLMTextFrames.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
from typing import Callable, Optional
|
|
||||||
|
|
||||||
import aioboto3
|
|
||||||
from loguru import logger
|
|
||||||
|
|
||||||
from pipecat.frames.frames import (
|
|
||||||
Frame,
|
|
||||||
LLMContextFrame,
|
|
||||||
LLMFullResponseEndFrame,
|
|
||||||
LLMFullResponseStartFrame,
|
|
||||||
LLMTextFrame,
|
|
||||||
)
|
|
||||||
from pipecat.processors.aggregators.llm_context import LLMContext, LLMSpecificMessage
|
|
||||||
from pipecat.processors.aggregators.openai_llm_context import (
|
|
||||||
OpenAILLMContext,
|
|
||||||
OpenAILLMContextFrame,
|
|
||||||
)
|
|
||||||
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
|
|
||||||
|
|
||||||
|
|
||||||
def default_context_to_payload_transformer(
|
|
||||||
context: LLMContext | OpenAILLMContext,
|
|
||||||
) -> Optional[str]:
|
|
||||||
"""Default transformer to create AgentCore payload from LLM context.
|
|
||||||
|
|
||||||
Extracts the latest user or system message text and wraps it in {"prompt": "<text>"}.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
context: The LLM context containing conversation messages.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
A JSON string payload for AgentCore, or None if no valid message found.
|
|
||||||
"""
|
|
||||||
messages = context.messages
|
|
||||||
|
|
||||||
if not messages:
|
|
||||||
return None
|
|
||||||
|
|
||||||
last_message = messages[-1]
|
|
||||||
if isinstance(last_message, LLMSpecificMessage) or last_message.get("role") not in (
|
|
||||||
"user",
|
|
||||||
"system",
|
|
||||||
):
|
|
||||||
return None
|
|
||||||
|
|
||||||
content = last_message.get("content")
|
|
||||||
if not content:
|
|
||||||
return None
|
|
||||||
|
|
||||||
if isinstance(content, str):
|
|
||||||
prompt = content
|
|
||||||
elif isinstance(content, list):
|
|
||||||
prompt = " ".join([part.get("text", "") for part in content])
|
|
||||||
else:
|
|
||||||
return None
|
|
||||||
|
|
||||||
return json.dumps({"prompt": prompt})
|
|
||||||
|
|
||||||
|
|
||||||
def default_response_to_output_transformer(response_line: str) -> Optional[str]:
|
|
||||||
"""Default transformer to extract output text from AgentCore response.
|
|
||||||
|
|
||||||
Expects responses with {"response": "<text>"} format.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
response_line: The raw response line from AgentCore (without "data: " prefix).
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
The extracted output text, or None if no text found.
|
|
||||||
"""
|
|
||||||
response_json = json.loads(response_line)
|
|
||||||
return response_json.get("response")
|
|
||||||
|
|
||||||
|
|
||||||
class AWSAgentCoreProcessor(FrameProcessor):
|
|
||||||
"""Processor that runs an Amazon Bedrock AgentCore agent.
|
|
||||||
|
|
||||||
Input:
|
|
||||||
- LLMContextFrame: Supplies a context used to invoke the agent.
|
|
||||||
|
|
||||||
Output:
|
|
||||||
- LLMTextFrame: The agent's text response(s).
|
|
||||||
A single agent invocation may result in multiple text frames.
|
|
||||||
|
|
||||||
This processor transforms the input context to a payload for the AgentCore
|
|
||||||
agent, and transforms the agent's response(s) into output text frame(s). Both
|
|
||||||
mappings are configurable via transformers. Below is the default behavior.
|
|
||||||
|
|
||||||
Input transformer (context_to_payload_transformer):
|
|
||||||
- Grabs the latest user or system message (if it's the latest message)
|
|
||||||
- Extracts its text content
|
|
||||||
- Constructs a payload that looks like {"prompt": "<text>"}
|
|
||||||
|
|
||||||
Output transformer (response_to_output_transformer):
|
|
||||||
- Expects responses that look like {"response": "<text>"}
|
|
||||||
- Extracts the text for use in the LLMTextFrame(s)
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
agentArn: str,
|
|
||||||
aws_access_key: Optional[str] = None,
|
|
||||||
aws_secret_key: Optional[str] = None,
|
|
||||||
aws_session_token: Optional[str] = None,
|
|
||||||
aws_region: Optional[str] = None,
|
|
||||||
context_to_payload_transformer: Optional[
|
|
||||||
Callable[[LLMContext | OpenAILLMContext], Optional[str]]
|
|
||||||
] = None,
|
|
||||||
response_to_output_transformer: Optional[Callable[[str], Optional[str]]] = None,
|
|
||||||
**kwargs,
|
|
||||||
):
|
|
||||||
"""Initialize the AWS AgentCore processor.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
agentArn: The Amazon Web Services Resource Name (ARN) of the agent.
|
|
||||||
aws_access_key: AWS access key ID. If None, uses default credentials.
|
|
||||||
aws_secret_key: AWS secret access key. If None, uses default credentials.
|
|
||||||
aws_session_token: AWS session token for temporary credentials.
|
|
||||||
aws_region: AWS region.
|
|
||||||
context_to_payload_transformer: Optional callable to transform
|
|
||||||
LLMContext into AgentCore payload string. If None, uses
|
|
||||||
default_context_to_payload_transformer.
|
|
||||||
response_to_output_transformer: Optional callable to extract output text
|
|
||||||
from AgentCore response. If None, uses
|
|
||||||
default_response_to_output_transformer.
|
|
||||||
**kwargs: Additional arguments passed to parent FrameProcessor.
|
|
||||||
"""
|
|
||||||
super().__init__(**kwargs)
|
|
||||||
|
|
||||||
self._agentArn = agentArn
|
|
||||||
self._aws_session = aioboto3.Session()
|
|
||||||
|
|
||||||
# Store AWS session parameters for creating client in async context
|
|
||||||
self._aws_params = {
|
|
||||||
"aws_access_key_id": aws_access_key or os.getenv("AWS_ACCESS_KEY_ID"),
|
|
||||||
"aws_secret_access_key": aws_secret_key or os.getenv("AWS_SECRET_ACCESS_KEY"),
|
|
||||||
"aws_session_token": aws_session_token or os.getenv("AWS_SESSION_TOKEN"),
|
|
||||||
"region_name": aws_region or os.getenv("AWS_REGION", "us-east-1"),
|
|
||||||
}
|
|
||||||
|
|
||||||
# Set transformers with defaults
|
|
||||||
self._context_to_payload_transformer = (
|
|
||||||
context_to_payload_transformer or default_context_to_payload_transformer
|
|
||||||
)
|
|
||||||
self._response_to_output_transformer = (
|
|
||||||
response_to_output_transformer or default_response_to_output_transformer
|
|
||||||
)
|
|
||||||
|
|
||||||
# State for managing output response bookends
|
|
||||||
self._output_response_open = False
|
|
||||||
self._last_text_frame_time: Optional[float] = None
|
|
||||||
self._close_task: Optional[asyncio.Task] = None
|
|
||||||
self._output_response_timeout = 1.0 # seconds
|
|
||||||
|
|
||||||
async def _close_output_response_after_timeout(self):
|
|
||||||
"""Close the output response after timeout if no new text frames arrive."""
|
|
||||||
await asyncio.sleep(self._output_response_timeout)
|
|
||||||
if self._output_response_open:
|
|
||||||
self._output_response_open = False
|
|
||||||
await self.push_frame(LLMFullResponseEndFrame(skip_tts=self._get_skip_tts()))
|
|
||||||
|
|
||||||
async def _push_text_frame(self, text: str):
|
|
||||||
"""Push a text frame, managing output response bookends."""
|
|
||||||
# Cancel any pending close task
|
|
||||||
if self._close_task and not self._close_task.done():
|
|
||||||
await self.cancel_task(self._close_task)
|
|
||||||
|
|
||||||
# Open output response if needed
|
|
||||||
if not self._output_response_open:
|
|
||||||
await self.push_frame(LLMFullResponseStartFrame(skip_tts=self._get_skip_tts()))
|
|
||||||
self._output_response_open = True
|
|
||||||
|
|
||||||
# Push the text frame
|
|
||||||
await self.push_frame(LLMTextFrame(text, skip_tts=self._get_skip_tts()))
|
|
||||||
self._last_text_frame_time = asyncio.get_event_loop().time()
|
|
||||||
|
|
||||||
# Schedule closing the output response after timeout
|
|
||||||
self._close_task = self.create_task(self._close_output_response_after_timeout())
|
|
||||||
|
|
||||||
async def process_frame(self, frame: Frame, direction: FrameDirection):
|
|
||||||
"""Process incoming frames and handle LLM message frames.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
frame: The incoming frame to process.
|
|
||||||
direction: The direction of frame flow in the pipeline.
|
|
||||||
"""
|
|
||||||
await super().process_frame(frame, direction)
|
|
||||||
if isinstance(frame, (LLMContextFrame, OpenAILLMContextFrame)):
|
|
||||||
# Create payload to invoke AgentCore agent
|
|
||||||
payload = self._context_to_payload_transformer(frame.context)
|
|
||||||
|
|
||||||
if not payload:
|
|
||||||
return
|
|
||||||
|
|
||||||
async with self._aws_session.client("bedrock-agentcore", **self._aws_params) as client:
|
|
||||||
# Invoke the AgentCore agent
|
|
||||||
response = await client.invoke_agent_runtime(
|
|
||||||
agentRuntimeArn=self._agentArn, payload=payload.encode()
|
|
||||||
)
|
|
||||||
|
|
||||||
# Determine if this is a streamed multi-part response, which
|
|
||||||
# will affect our parsing
|
|
||||||
is_multi_part_response = "text/event-stream" in response.get("contentType", "")
|
|
||||||
|
|
||||||
# Handle each response part (there may be one, for single
|
|
||||||
# responses, or multiple, for streamed multi-part responses)
|
|
||||||
async for part in response.get("response", []):
|
|
||||||
part_string = part.decode("utf-8")
|
|
||||||
|
|
||||||
# In streamed multi-part responses, each part might have
|
|
||||||
# one or more lines, each of which starts with "data: ".
|
|
||||||
# Treat each line as a response.
|
|
||||||
if is_multi_part_response:
|
|
||||||
for line in part_string.split("\n"):
|
|
||||||
# Get response text from this line
|
|
||||||
if not line:
|
|
||||||
continue
|
|
||||||
if not line.startswith("data: "):
|
|
||||||
logger.warning(f"Expected line to start with 'data: ', got: {line}")
|
|
||||||
continue
|
|
||||||
line = line[6:] # omit "data: "
|
|
||||||
|
|
||||||
# Transform response line to output text
|
|
||||||
text = self._response_to_output_transformer(line)
|
|
||||||
if text:
|
|
||||||
await self._push_text_frame(text)
|
|
||||||
|
|
||||||
# In single-part responses, the whole part is one response
|
|
||||||
# and there's no "data: " prefix
|
|
||||||
else:
|
|
||||||
# Transform response part string to output text
|
|
||||||
text = self._response_to_output_transformer(part_string)
|
|
||||||
if text:
|
|
||||||
await self._push_text_frame(text)
|
|
||||||
|
|
||||||
# Final close if output response is still open after all parts processed
|
|
||||||
if self._output_response_open:
|
|
||||||
if self._close_task and not self._close_task.done():
|
|
||||||
await self.cancel_task(self._close_task)
|
|
||||||
self._output_response_open = False
|
|
||||||
await self.push_frame(LLMFullResponseEndFrame(skip_tts=self._get_skip_tts()))
|
|
||||||
else:
|
|
||||||
await self.push_frame(frame, direction)
|
|
||||||
@@ -734,7 +734,7 @@ class AWSBedrockLLMService(LLMService):
|
|||||||
aws_access_key: Optional[str] = None,
|
aws_access_key: Optional[str] = None,
|
||||||
aws_secret_key: Optional[str] = None,
|
aws_secret_key: Optional[str] = None,
|
||||||
aws_session_token: Optional[str] = None,
|
aws_session_token: Optional[str] = None,
|
||||||
aws_region: Optional[str] = None,
|
aws_region: str = "us-east-1",
|
||||||
params: Optional[InputParams] = None,
|
params: Optional[InputParams] = None,
|
||||||
client_config: Optional[Config] = None,
|
client_config: Optional[Config] = None,
|
||||||
retry_timeout_secs: Optional[float] = 5.0,
|
retry_timeout_secs: Optional[float] = 5.0,
|
||||||
@@ -981,7 +981,7 @@ class AWSBedrockLLMService(LLMService):
|
|||||||
using_noop_tool = False
|
using_noop_tool = False
|
||||||
|
|
||||||
try:
|
try:
|
||||||
await self.push_frame(LLMFullResponseStartFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseStartFrame())
|
||||||
await self.start_processing_metrics()
|
await self.start_processing_metrics()
|
||||||
|
|
||||||
await self.start_ttfb_metrics()
|
await self.start_ttfb_metrics()
|
||||||
@@ -1078,9 +1078,9 @@ class AWSBedrockLLMService(LLMService):
|
|||||||
if "contentBlockDelta" in event:
|
if "contentBlockDelta" in event:
|
||||||
delta = event["contentBlockDelta"]["delta"]
|
delta = event["contentBlockDelta"]["delta"]
|
||||||
if "text" in delta:
|
if "text" in delta:
|
||||||
await self.push_frame(
|
frame = LLMTextFrame(delta["text"])
|
||||||
LLMTextFrame(delta["text"], skip_tts=self._get_skip_tts())
|
frame.includes_inter_frame_spaces = True
|
||||||
)
|
await self.push_frame(frame)
|
||||||
completion_tokens_estimate += self._estimate_tokens(delta["text"])
|
completion_tokens_estimate += self._estimate_tokens(delta["text"])
|
||||||
elif "toolUse" in delta and "input" in delta["toolUse"]:
|
elif "toolUse" in delta and "input" in delta["toolUse"]:
|
||||||
# Handle partial JSON for tool use
|
# Handle partial JSON for tool use
|
||||||
@@ -1138,10 +1138,10 @@ class AWSBedrockLLMService(LLMService):
|
|||||||
except (ReadTimeoutError, asyncio.TimeoutError):
|
except (ReadTimeoutError, asyncio.TimeoutError):
|
||||||
await self._call_event_handler("on_completion_timeout")
|
await self._call_event_handler("on_completion_timeout")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.exception(f"{self} exception: {e}")
|
||||||
finally:
|
finally:
|
||||||
await self.stop_processing_metrics()
|
await self.stop_processing_metrics()
|
||||||
await self.push_frame(LLMFullResponseEndFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseEndFrame())
|
||||||
comp_tokens = (
|
comp_tokens = (
|
||||||
completion_tokens
|
completion_tokens
|
||||||
if not use_completion_tokens_estimate
|
if not use_completion_tokens_estimate
|
||||||
|
|||||||
@@ -453,7 +453,7 @@ class AWSNovaSonicLLMService(LLMService):
|
|||||||
self._ready_to_send_context = True
|
self._ready_to_send_context = True
|
||||||
await self._finish_connecting_if_context_available()
|
await self._finish_connecting_if_context_available()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Initialization error: {e}", exception=e)
|
logger.error(f"{self} initialization error: {e}")
|
||||||
await self._disconnect()
|
await self._disconnect()
|
||||||
|
|
||||||
async def _process_completed_function_calls(self, send_new_results: bool):
|
async def _process_completed_function_calls(self, send_new_results: bool):
|
||||||
@@ -577,7 +577,7 @@ class AWSNovaSonicLLMService(LLMService):
|
|||||||
|
|
||||||
logger.info("Finished disconnecting")
|
logger.info("Finished disconnecting")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error disconnecting: {e}", exception=e)
|
logger.error(f"{self} error disconnecting: {e}")
|
||||||
|
|
||||||
def _create_client(self) -> BedrockRuntimeClient:
|
def _create_client(self) -> BedrockRuntimeClient:
|
||||||
config = Config(
|
config = Config(
|
||||||
@@ -885,7 +885,7 @@ class AWSNovaSonicLLMService(LLMService):
|
|||||||
# Errors are kind of expected while disconnecting, so just
|
# Errors are kind of expected while disconnecting, so just
|
||||||
# ignore them and do nothing
|
# ignore them and do nothing
|
||||||
return
|
return
|
||||||
await self.push_error(error_msg=f"Error processing responses: {e}", exception=e)
|
logger.error(f"{self} error processing responses: {e}")
|
||||||
if self._wants_connection:
|
if self._wants_connection:
|
||||||
await self.reset_conversation()
|
await self.reset_conversation()
|
||||||
|
|
||||||
@@ -1016,7 +1016,7 @@ class AWSNovaSonicLLMService(LLMService):
|
|||||||
logger.debug("Assistant response started")
|
logger.debug("Assistant response started")
|
||||||
|
|
||||||
# Report the start of the assistant response.
|
# Report the start of the assistant response.
|
||||||
await self.push_frame(LLMFullResponseStartFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseStartFrame())
|
||||||
|
|
||||||
# Report that equivalent of TTS (this is a speech-to-speech model) started
|
# Report that equivalent of TTS (this is a speech-to-speech model) started
|
||||||
await self.push_frame(TTSStartedFrame())
|
await self.push_frame(TTSStartedFrame())
|
||||||
@@ -1062,7 +1062,7 @@ class AWSNovaSonicLLMService(LLMService):
|
|||||||
# We also need to re-push the LLMFullResponseStartFrame since the
|
# We also need to re-push the LLMFullResponseStartFrame since the
|
||||||
# TTSTextFrame would be ignored otherwise (the interruption frame
|
# TTSTextFrame would be ignored otherwise (the interruption frame
|
||||||
# would have cleared the assistant aggregator state).
|
# would have cleared the assistant aggregator state).
|
||||||
await self.push_frame(LLMFullResponseStartFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseStartFrame())
|
||||||
frame = TTSTextFrame(
|
frame = TTSTextFrame(
|
||||||
self._assistant_text_buffer, aggregated_by=AggregationType.SENTENCE
|
self._assistant_text_buffer, aggregated_by=AggregationType.SENTENCE
|
||||||
)
|
)
|
||||||
@@ -1071,7 +1071,7 @@ class AWSNovaSonicLLMService(LLMService):
|
|||||||
self._may_need_repush_assistant_text = False
|
self._may_need_repush_assistant_text = False
|
||||||
|
|
||||||
# Report the end of the assistant response.
|
# Report the end of the assistant response.
|
||||||
await self.push_frame(LLMFullResponseEndFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseEndFrame())
|
||||||
|
|
||||||
# Report that equivalent of TTS (this is a speech-to-speech model) stopped.
|
# Report that equivalent of TTS (this is a speech-to-speech model) stopped.
|
||||||
await self.push_frame(TTSStoppedFrame())
|
await self.push_frame(TTSStoppedFrame())
|
||||||
|
|||||||
@@ -1,283 +0,0 @@
|
|||||||
#
|
|
||||||
# Copyright (c) 2024–2025, Daily
|
|
||||||
#
|
|
||||||
# SPDX-License-Identifier: BSD 2-Clause License
|
|
||||||
#
|
|
||||||
|
|
||||||
"""AWS SageMaker bidirectional streaming client.
|
|
||||||
|
|
||||||
This module provides a client for streaming bidirectional communication with
|
|
||||||
SageMaker endpoints using the HTTP/2 protocol. Supports sending audio, text,
|
|
||||||
and JSON data to SageMaker model endpoints and receiving streaming responses.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import os
|
|
||||||
from typing import Optional
|
|
||||||
|
|
||||||
from loguru import logger
|
|
||||||
|
|
||||||
try:
|
|
||||||
from aws_sdk_sagemaker_runtime_http2.client import SageMakerRuntimeHTTP2Client
|
|
||||||
from aws_sdk_sagemaker_runtime_http2.config import Config, HTTPAuthSchemeResolver
|
|
||||||
from aws_sdk_sagemaker_runtime_http2.models import (
|
|
||||||
InvokeEndpointWithBidirectionalStreamInput,
|
|
||||||
RequestPayloadPart,
|
|
||||||
RequestStreamEventPayloadPart,
|
|
||||||
ResponseStreamEvent,
|
|
||||||
)
|
|
||||||
from smithy_aws_core.auth.sigv4 import SigV4AuthScheme
|
|
||||||
from smithy_aws_core.identity import EnvironmentCredentialsResolver
|
|
||||||
from smithy_core.aio.eventstream import DuplexEventStream
|
|
||||||
except ModuleNotFoundError as e:
|
|
||||||
logger.error(f"Exception: {e}")
|
|
||||||
logger.error(
|
|
||||||
"In order to use SageMaker BiDi client, you need to `pip install pipecat-ai[sagemaker]`."
|
|
||||||
)
|
|
||||||
raise Exception(f"Missing module: {e}")
|
|
||||||
|
|
||||||
|
|
||||||
class SageMakerBidiClient:
|
|
||||||
"""Client for bidirectional streaming with AWS SageMaker endpoints.
|
|
||||||
|
|
||||||
Handles low-level HTTP/2 bidirectional streaming protocol for communicating
|
|
||||||
with SageMaker model endpoints. Provides methods for sending various data
|
|
||||||
types (audio, text, JSON) and receiving streaming responses.
|
|
||||||
|
|
||||||
This client uses AWS SigV4 authentication and supports credential resolution
|
|
||||||
from environment variables, AWS CLI configuration, and instance metadata.
|
|
||||||
|
|
||||||
Example::
|
|
||||||
|
|
||||||
client = SageMakerBidiClient(
|
|
||||||
endpoint_name="my-deepgram-endpoint",
|
|
||||||
region="us-east-2",
|
|
||||||
model_invocation_path="v1/listen",
|
|
||||||
model_query_string="model=nova-3&language=en"
|
|
||||||
)
|
|
||||||
await client.start_session()
|
|
||||||
await client.send_audio_chunk(audio_bytes)
|
|
||||||
response = await client.receive_response()
|
|
||||||
await client.close_session()
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
endpoint_name: str,
|
|
||||||
region: str,
|
|
||||||
model_invocation_path: str = "",
|
|
||||||
model_query_string: str = "",
|
|
||||||
):
|
|
||||||
"""Initialize the SageMaker BiDi client.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
endpoint_name: Name of the SageMaker endpoint to connect to.
|
|
||||||
region: AWS region where the endpoint is deployed.
|
|
||||||
model_invocation_path: API path for the model invocation (e.g., "v1/listen").
|
|
||||||
model_query_string: Query string parameters for the model (e.g., "model=nova-3").
|
|
||||||
"""
|
|
||||||
self.endpoint_name = endpoint_name
|
|
||||||
self.region = region
|
|
||||||
self.model_invocation_path = model_invocation_path
|
|
||||||
self.model_query_string = model_query_string
|
|
||||||
self.bidi_endpoint = f"https://runtime.sagemaker.{region}.amazonaws.com:8443"
|
|
||||||
self._client: Optional[SageMakerRuntimeHTTP2Client] = None
|
|
||||||
self._stream: Optional[
|
|
||||||
DuplexEventStream[RequestStreamEventPayloadPart, ResponseStreamEvent, any]
|
|
||||||
] = None
|
|
||||||
self._output_stream = None
|
|
||||||
self._is_active = False
|
|
||||||
|
|
||||||
def _initialize_client(self):
|
|
||||||
"""Initialize the SageMaker Runtime HTTP2 client with AWS credentials.
|
|
||||||
|
|
||||||
Creates and configures the SageMaker Runtime HTTP2 client with SigV4
|
|
||||||
authentication. Attempts to resolve AWS credentials from environment
|
|
||||||
variables, AWS CLI configuration, or instance metadata.
|
|
||||||
"""
|
|
||||||
logger.debug(f"Initializing SageMaker BiDi client for region: {self.region}")
|
|
||||||
logger.debug(f"Using endpoint URI: {self.bidi_endpoint}")
|
|
||||||
|
|
||||||
# Check for AWS credentials
|
|
||||||
has_env_creds = bool(os.getenv("AWS_ACCESS_KEY_ID") and os.getenv("AWS_SECRET_ACCESS_KEY"))
|
|
||||||
|
|
||||||
if not has_env_creds:
|
|
||||||
logger.warning(
|
|
||||||
"AWS credentials not found in environment variables. "
|
|
||||||
"Attempting to use EnvironmentCredentialsResolver which will check "
|
|
||||||
"AWS CLI configuration and instance metadata."
|
|
||||||
)
|
|
||||||
|
|
||||||
config = Config(
|
|
||||||
endpoint_uri=self.bidi_endpoint,
|
|
||||||
region=self.region,
|
|
||||||
aws_credentials_identity_resolver=EnvironmentCredentialsResolver(),
|
|
||||||
auth_scheme_resolver=HTTPAuthSchemeResolver(),
|
|
||||||
auth_schemes={"aws.auth#sigv4": SigV4AuthScheme(service="sagemaker")},
|
|
||||||
)
|
|
||||||
self._client = SageMakerRuntimeHTTP2Client(config=config)
|
|
||||||
|
|
||||||
async def start_session(self):
|
|
||||||
"""Start a bidirectional streaming session with the SageMaker endpoint.
|
|
||||||
|
|
||||||
Initializes the client if needed, creates the bidirectional stream, and
|
|
||||||
establishes the connection to the SageMaker endpoint. Must be called
|
|
||||||
before sending or receiving data.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
The output stream for receiving responses.
|
|
||||||
|
|
||||||
Raises:
|
|
||||||
RuntimeError: If client initialization or connection fails.
|
|
||||||
"""
|
|
||||||
if not self._client:
|
|
||||||
self._initialize_client()
|
|
||||||
|
|
||||||
logger.debug(f"Starting BiDi session with endpoint: {self.endpoint_name}")
|
|
||||||
logger.debug(f"Model invocation path: {self.model_invocation_path}")
|
|
||||||
logger.debug(f"Model query string: {self.model_query_string}")
|
|
||||||
|
|
||||||
# Create the bidirectional stream
|
|
||||||
stream_input = InvokeEndpointWithBidirectionalStreamInput(
|
|
||||||
endpoint_name=self.endpoint_name,
|
|
||||||
model_invocation_path=self.model_invocation_path,
|
|
||||||
model_query_string=self.model_query_string,
|
|
||||||
)
|
|
||||||
|
|
||||||
try:
|
|
||||||
self._stream = await self._client.invoke_endpoint_with_bidirectional_stream(
|
|
||||||
stream_input
|
|
||||||
)
|
|
||||||
self._is_active = True
|
|
||||||
|
|
||||||
# Get output stream
|
|
||||||
output = await self._stream.await_output()
|
|
||||||
self._output_stream = output[1]
|
|
||||||
|
|
||||||
logger.debug("BiDi session started successfully")
|
|
||||||
return self._output_stream
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"Failed to start BiDi session: {e}")
|
|
||||||
self._is_active = False
|
|
||||||
raise RuntimeError(f"Failed to start SageMaker BiDi session: {e}")
|
|
||||||
|
|
||||||
async def send_data(self, data_bytes: bytes, data_type: Optional[str] = None):
|
|
||||||
"""Send a chunk of data to the stream.
|
|
||||||
|
|
||||||
Generic method for sending any type of data to the SageMaker endpoint.
|
|
||||||
Use the convenience methods (send_audio_chunk, send_text, send_json)
|
|
||||||
for common data types.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
data_bytes: Raw bytes to send.
|
|
||||||
data_type: Optional data type header. Common values are "BINARY" for
|
|
||||||
audio/binary data and "UTF8" for text/JSON data.
|
|
||||||
|
|
||||||
Raises:
|
|
||||||
RuntimeError: If session is not active or send fails.
|
|
||||||
"""
|
|
||||||
if not self._is_active or not self._stream:
|
|
||||||
raise RuntimeError("BiDi session not active")
|
|
||||||
|
|
||||||
try:
|
|
||||||
payload = RequestPayloadPart(bytes_=data_bytes, data_type=data_type)
|
|
||||||
event = RequestStreamEventPayloadPart(value=payload)
|
|
||||||
await self._stream.input_stream.send(event)
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"Failed to send data: {e}")
|
|
||||||
raise
|
|
||||||
|
|
||||||
async def send_audio_chunk(self, audio_bytes: bytes):
|
|
||||||
"""Send a chunk of audio data to the stream.
|
|
||||||
|
|
||||||
Convenience method for sending audio data. Automatically sets the data
|
|
||||||
type to "BINARY".
|
|
||||||
|
|
||||||
Args:
|
|
||||||
audio_bytes: Raw audio bytes to send (e.g., PCM audio data).
|
|
||||||
|
|
||||||
Raises:
|
|
||||||
RuntimeError: If session is not active or send fails.
|
|
||||||
"""
|
|
||||||
await self.send_data(audio_bytes, data_type="BINARY")
|
|
||||||
|
|
||||||
async def send_text(self, text: str):
|
|
||||||
"""Send text data to the stream.
|
|
||||||
|
|
||||||
Convenience method for sending text data. Automatically encodes the text
|
|
||||||
as UTF-8 and sets the data type to "UTF8".
|
|
||||||
|
|
||||||
Args:
|
|
||||||
text: Text string to send.
|
|
||||||
|
|
||||||
Raises:
|
|
||||||
RuntimeError: If session is not active or send fails.
|
|
||||||
"""
|
|
||||||
await self.send_data(text.encode("utf-8"), data_type="UTF8")
|
|
||||||
|
|
||||||
async def send_json(self, data: dict):
|
|
||||||
"""Send JSON data to the stream.
|
|
||||||
|
|
||||||
Convenience method for sending JSON-encoded messages. Useful for control
|
|
||||||
messages like KeepAlive or CloseStream. Automatically serializes the
|
|
||||||
dictionary to JSON, encodes as UTF-8, and sets the data type to "UTF8".
|
|
||||||
|
|
||||||
Args:
|
|
||||||
data: Dictionary to send as JSON (e.g., {"type": "KeepAlive"}).
|
|
||||||
|
|
||||||
Raises:
|
|
||||||
RuntimeError: If session is not active or send fails.
|
|
||||||
"""
|
|
||||||
import json
|
|
||||||
|
|
||||||
await self.send_data(json.dumps(data).encode("utf-8"), data_type="UTF8")
|
|
||||||
|
|
||||||
async def receive_response(self) -> Optional[ResponseStreamEvent]:
|
|
||||||
"""Receive a response from the stream.
|
|
||||||
|
|
||||||
Blocks until a response is available from the SageMaker endpoint. Returns
|
|
||||||
None when the stream is closed.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
The response event containing payload data, or None if stream is closed.
|
|
||||||
|
|
||||||
Raises:
|
|
||||||
RuntimeError: If session is not active.
|
|
||||||
"""
|
|
||||||
if not self._is_active or not self._output_stream:
|
|
||||||
raise RuntimeError("BiDi session not active")
|
|
||||||
|
|
||||||
try:
|
|
||||||
result = await self._output_stream.receive()
|
|
||||||
return result
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"Failed to receive response: {e}")
|
|
||||||
raise
|
|
||||||
|
|
||||||
async def close_session(self):
|
|
||||||
"""Close the bidirectional streaming session.
|
|
||||||
|
|
||||||
Gracefully closes the input stream and marks the session as inactive.
|
|
||||||
Safe to call multiple times.
|
|
||||||
"""
|
|
||||||
if not self._is_active:
|
|
||||||
return
|
|
||||||
|
|
||||||
logger.debug("Closing BiDi session...")
|
|
||||||
self._is_active = False
|
|
||||||
|
|
||||||
try:
|
|
||||||
if self._stream:
|
|
||||||
await self._stream.input_stream.close()
|
|
||||||
logger.debug("BiDi session closed successfully")
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning(f"Error closing BiDi session: {e}")
|
|
||||||
|
|
||||||
@property
|
|
||||||
def is_active(self) -> bool:
|
|
||||||
"""Check if the session is currently active.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
True if session is active, False otherwise.
|
|
||||||
"""
|
|
||||||
return self._is_active
|
|
||||||
@@ -140,7 +140,8 @@ class AWSTranscribeSTTService(STTService):
|
|||||||
return
|
return
|
||||||
logger.warning("WebSocket connection not established after connect")
|
logger.warning("WebSocket connection not established after connect")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
retry_count += 1
|
retry_count += 1
|
||||||
if retry_count < max_retries:
|
if retry_count < max_retries:
|
||||||
await asyncio.sleep(1) # Wait before retrying
|
await asyncio.sleep(1) # Wait before retrying
|
||||||
@@ -181,7 +182,8 @@ class AWSTranscribeSTTService(STTService):
|
|||||||
try:
|
try:
|
||||||
await self._connect()
|
await self._connect()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
return
|
return
|
||||||
|
|
||||||
# Format the audio data according to AWS event stream format
|
# Format the audio data according to AWS event stream format
|
||||||
@@ -198,11 +200,13 @@ class AWSTranscribeSTTService(STTService):
|
|||||||
await self._disconnect()
|
await self._disconnect()
|
||||||
# Don't yield error here - we'll retry on next frame
|
# Don't yield error here - we'll retry on next frame
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
await self._disconnect()
|
await self._disconnect()
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
await self._disconnect()
|
await self._disconnect()
|
||||||
|
|
||||||
async def _connect(self):
|
async def _connect(self):
|
||||||
@@ -285,7 +289,8 @@ class AWSTranscribeSTTService(STTService):
|
|||||||
|
|
||||||
await self._call_event_handler("on_connected")
|
await self._call_event_handler("on_connected")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
await self._disconnect()
|
await self._disconnect()
|
||||||
raise
|
raise
|
||||||
|
|
||||||
@@ -305,7 +310,8 @@ class AWSTranscribeSTTService(STTService):
|
|||||||
await self._ws_client.send(json.dumps(end_stream))
|
await self._ws_client.send(json.dumps(end_stream))
|
||||||
await self._ws_client.close()
|
await self._ws_client.close()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
finally:
|
finally:
|
||||||
self._ws_client = None
|
self._ws_client = None
|
||||||
await self._call_event_handler("on_disconnected")
|
await self._call_event_handler("on_disconnected")
|
||||||
@@ -523,15 +529,15 @@ class AWSTranscribeSTTService(STTService):
|
|||||||
)
|
)
|
||||||
elif headers.get(":message-type") == "exception":
|
elif headers.get(":message-type") == "exception":
|
||||||
error_msg = payload.get("Message", "Unknown error")
|
error_msg = payload.get("Message", "Unknown error")
|
||||||
await self.push_error(error_msg=f"AWS Transcribe error: {error_msg}")
|
logger.error(f"{self} Exception from AWS: {error_msg}")
|
||||||
|
await self.push_frame(ErrorFrame(f"AWS Transcribe error: {error_msg}"))
|
||||||
else:
|
else:
|
||||||
logger.debug(f"{self} Other message type received: {headers}")
|
logger.debug(f"{self} Other message type received: {headers}")
|
||||||
logger.debug(f"{self} Payload: {payload}")
|
logger.debug(f"{self} Payload: {payload}")
|
||||||
except websockets.exceptions.ConnectionClosed as e:
|
except websockets.exceptions.ConnectionClosed as e:
|
||||||
await self.push_error(
|
logger.error(f"{self} WebSocket connection closed in receive loop: {e}")
|
||||||
error_msg=f"WebSocket connection closed in receive loop", exception=e
|
|
||||||
)
|
|
||||||
break
|
break
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
break
|
break
|
||||||
|
|||||||
@@ -209,6 +209,15 @@ class AWSPollyTTSService(TTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that AWS TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that AWS's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
def language_to_service_language(self, language: Language) -> Optional[str]:
|
def language_to_service_language(self, language: Language) -> Optional[str]:
|
||||||
"""Convert a Language enum to AWS Polly language format.
|
"""Convert a Language enum to AWS Polly language format.
|
||||||
|
|
||||||
@@ -312,6 +321,7 @@ class AWSPollyTTSService(TTSService):
|
|||||||
|
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
except (BotoCoreError, ClientError) as error:
|
except (BotoCoreError, ClientError) as error:
|
||||||
|
logger.exception(f"{self} error generating TTS: {error}")
|
||||||
error_message = f"AWS Polly TTS error: {str(error)}"
|
error_message = f"AWS Polly TTS error: {str(error)}"
|
||||||
yield ErrorFrame(error=error_message)
|
yield ErrorFrame(error=error_message)
|
||||||
|
|
||||||
|
|||||||
@@ -91,6 +91,7 @@ class AzureImageGenServiceREST(ImageGenService):
|
|||||||
while status != "succeeded":
|
while status != "succeeded":
|
||||||
attempts_left -= 1
|
attempts_left -= 1
|
||||||
if attempts_left == 0:
|
if attempts_left == 0:
|
||||||
|
logger.error(f"{self} error: image generation timed out")
|
||||||
yield ErrorFrame("Image generation timed out")
|
yield ErrorFrame("Image generation timed out")
|
||||||
return
|
return
|
||||||
|
|
||||||
@@ -103,6 +104,7 @@ class AzureImageGenServiceREST(ImageGenService):
|
|||||||
|
|
||||||
image_url = json_response["result"]["data"][0]["url"] if json_response else None
|
image_url = json_response["result"]["data"][0]["url"] if json_response else None
|
||||||
if not image_url:
|
if not image_url:
|
||||||
|
logger.error(f"{self} error: image generation failed")
|
||||||
yield ErrorFrame("Image generation failed")
|
yield ErrorFrame("Image generation failed")
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|||||||
@@ -61,5 +61,5 @@ class AzureRealtimeLLMService(OpenAIRealtimeLLMService):
|
|||||||
)
|
)
|
||||||
self._receive_task = self.create_task(self._receive_task_handler())
|
self._receive_task = self.create_task(self._receive_task_handler())
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"initialization error: {e}", exception=e)
|
logger.error(f"{self} initialization error: {e}")
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
|
|||||||
@@ -121,7 +121,8 @@ class AzureSTTService(STTService):
|
|||||||
self._audio_stream.write(audio)
|
self._audio_stream.write(audio)
|
||||||
yield None
|
yield None
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
|
|
||||||
async def start(self, frame: StartFrame):
|
async def start(self, frame: StartFrame):
|
||||||
"""Start the speech recognition service.
|
"""Start the speech recognition service.
|
||||||
@@ -150,9 +151,8 @@ class AzureSTTService(STTService):
|
|||||||
self._speech_recognizer.recognized.connect(self._on_handle_recognized)
|
self._speech_recognizer.recognized.connect(self._on_handle_recognized)
|
||||||
self._speech_recognizer.start_continuous_recognition_async()
|
self._speech_recognizer.start_continuous_recognition_async()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(
|
logger.error(f"{self} exception during initialization: {e}")
|
||||||
error_msg=f"Uncaught exception during initialization: {e}", exception=e
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
)
|
|
||||||
|
|
||||||
async def stop(self, frame: EndFrame):
|
async def stop(self, frame: EndFrame):
|
||||||
"""Stop the speech recognition service.
|
"""Stop the speech recognition service.
|
||||||
|
|||||||
@@ -151,6 +151,15 @@ class AzureBaseTTSService(TTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that Azure TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that Azure's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
def language_to_service_language(self, language: Language) -> Optional[str]:
|
def language_to_service_language(self, language: Language) -> Optional[str]:
|
||||||
"""Convert a Language enum to Azure language format.
|
"""Convert a Language enum to Azure language format.
|
||||||
|
|
||||||
@@ -327,6 +336,7 @@ class AzureTTSService(AzureBaseTTSService):
|
|||||||
try:
|
try:
|
||||||
if self._speech_synthesizer is None:
|
if self._speech_synthesizer is None:
|
||||||
error_msg = "Speech synthesizer not initialized."
|
error_msg = "Speech synthesizer not initialized."
|
||||||
|
logger.error(error_msg)
|
||||||
yield ErrorFrame(error=error_msg)
|
yield ErrorFrame(error=error_msg)
|
||||||
return
|
return
|
||||||
|
|
||||||
@@ -354,13 +364,15 @@ class AzureTTSService(AzureBaseTTSService):
|
|||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
# Could add reconnection logic here if needed
|
# Could add reconnection logic here if needed
|
||||||
return
|
return
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
|
|
||||||
|
|
||||||
class AzureHttpTTSService(AzureBaseTTSService):
|
class AzureHttpTTSService(AzureBaseTTSService):
|
||||||
@@ -437,6 +449,5 @@ class AzureHttpTTSService(AzureBaseTTSService):
|
|||||||
cancellation_details = result.cancellation_details
|
cancellation_details = result.cancellation_details
|
||||||
logger.warning(f"Speech synthesis canceled: {cancellation_details.reason}")
|
logger.warning(f"Speech synthesis canceled: {cancellation_details.reason}")
|
||||||
if cancellation_details.reason == CancellationReason.Error:
|
if cancellation_details.reason == CancellationReason.Error:
|
||||||
yield ErrorFrame(
|
logger.error(f"{self} error: {cancellation_details.error_details}")
|
||||||
error=f"Unknown error occurred: {cancellation_details.error_details}"
|
yield ErrorFrame(error=f"{self} error: {cancellation_details.error_details}")
|
||||||
)
|
|
||||||
|
|||||||
@@ -276,7 +276,8 @@ class CartesiaSTTService(WebsocketSTTService):
|
|||||||
self._websocket = await websocket_connect(ws_url, additional_headers=headers)
|
self._websocket = await websocket_connect(ws_url, additional_headers=headers)
|
||||||
await self._call_event_handler("on_connected")
|
await self._call_event_handler("on_connected")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
|
|
||||||
async def _disconnect_websocket(self):
|
async def _disconnect_websocket(self):
|
||||||
try:
|
try:
|
||||||
@@ -284,7 +285,8 @@ class CartesiaSTTService(WebsocketSTTService):
|
|||||||
logger.debug("Disconnecting from Cartesia STT")
|
logger.debug("Disconnecting from Cartesia STT")
|
||||||
await self._websocket.close()
|
await self._websocket.close()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error closing websocket: {e}", exception=e)
|
logger.error(f"{self} error closing websocket: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
finally:
|
finally:
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
await self._call_event_handler("on_disconnected")
|
await self._call_event_handler("on_disconnected")
|
||||||
@@ -317,7 +319,8 @@ class CartesiaSTTService(WebsocketSTTService):
|
|||||||
|
|
||||||
elif data["type"] == "error":
|
elif data["type"] == "error":
|
||||||
error_msg = data.get("message", "Unknown error")
|
error_msg = data.get("message", "Unknown error")
|
||||||
await self.push_error(error_msg=error_msg)
|
logger.error(f"Cartesia error: {error_msg}")
|
||||||
|
await self.push_error(ErrorFrame(error=error_msg))
|
||||||
|
|
||||||
@traced_stt
|
@traced_stt
|
||||||
async def _handle_transcription(
|
async def _handle_transcription(
|
||||||
|
|||||||
@@ -497,7 +497,8 @@ class CartesiaTTSService(AudioContextWordTTSService):
|
|||||||
)
|
)
|
||||||
await self._call_event_handler("on_connected")
|
await self._call_event_handler("on_connected")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
await self._call_event_handler("on_connection_error", f"{e}")
|
await self._call_event_handler("on_connection_error", f"{e}")
|
||||||
|
|
||||||
@@ -509,7 +510,8 @@ class CartesiaTTSService(AudioContextWordTTSService):
|
|||||||
logger.debug("Disconnecting from Cartesia")
|
logger.debug("Disconnecting from Cartesia")
|
||||||
await self._websocket.close()
|
await self._websocket.close()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
finally:
|
finally:
|
||||||
self._context_id = None
|
self._context_id = None
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
@@ -562,12 +564,13 @@ class CartesiaTTSService(AudioContextWordTTSService):
|
|||||||
)
|
)
|
||||||
await self.append_to_audio_context(msg["context_id"], frame)
|
await self.append_to_audio_context(msg["context_id"], frame)
|
||||||
elif msg["type"] == "error":
|
elif msg["type"] == "error":
|
||||||
|
logger.error(f"{self} error: {msg}")
|
||||||
await self.push_frame(TTSStoppedFrame())
|
await self.push_frame(TTSStoppedFrame())
|
||||||
await self.stop_all_metrics()
|
await self.stop_all_metrics()
|
||||||
await self.push_error(error_msg=f"Error: {msg}")
|
await self.push_error(ErrorFrame(error=f"{self} error: {msg['error']}"))
|
||||||
self._context_id = None
|
self._context_id = None
|
||||||
else:
|
else:
|
||||||
await self.push_error(error_msg=f"Error, unknown message type: {msg}")
|
logger.error(f"{self} error, unknown message type: {msg}")
|
||||||
|
|
||||||
async def _receive_messages(self):
|
async def _receive_messages(self):
|
||||||
while True:
|
while True:
|
||||||
@@ -605,14 +608,16 @@ class CartesiaTTSService(AudioContextWordTTSService):
|
|||||||
await self._get_websocket().send(msg)
|
await self._get_websocket().send(msg)
|
||||||
await self.start_tts_usage_metrics(text)
|
await self.start_tts_usage_metrics(text)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
await self._disconnect()
|
await self._disconnect()
|
||||||
await self._connect()
|
await self._connect()
|
||||||
return
|
return
|
||||||
yield None
|
yield None
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
|
|
||||||
|
|
||||||
class CartesiaHttpTTSService(TTSService):
|
class CartesiaHttpTTSService(TTSService):
|
||||||
@@ -803,7 +808,8 @@ class CartesiaHttpTTSService(TTSService):
|
|||||||
async with session.post(url, json=payload, headers=headers) as response:
|
async with session.post(url, json=payload, headers=headers) as response:
|
||||||
if response.status != 200:
|
if response.status != 200:
|
||||||
error_text = await response.text()
|
error_text = await response.text()
|
||||||
yield ErrorFrame(error=f"Cartesia API error: {error_text}")
|
logger.error(f"Cartesia API error: {error_text}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"Cartesia API error: {error_text}"))
|
||||||
raise Exception(f"Cartesia API returned status {response.status}: {error_text}")
|
raise Exception(f"Cartesia API returned status {response.status}: {error_text}")
|
||||||
|
|
||||||
audio_data = await response.read()
|
audio_data = await response.read()
|
||||||
@@ -819,7 +825,8 @@ class CartesiaHttpTTSService(TTSService):
|
|||||||
yield frame
|
yield frame
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
finally:
|
finally:
|
||||||
await self.stop_ttfb_metrics()
|
await self.stop_ttfb_metrics()
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
|
|||||||
@@ -6,9 +6,7 @@
|
|||||||
|
|
||||||
"""Deepgram Flux speech-to-text service implementation."""
|
"""Deepgram Flux speech-to-text service implementation."""
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import json
|
import json
|
||||||
import time
|
|
||||||
from enum import Enum
|
from enum import Enum
|
||||||
from typing import Any, AsyncGenerator, Dict, Optional
|
from typing import Any, AsyncGenerator, Dict, Optional
|
||||||
from urllib.parse import urlencode
|
from urllib.parse import urlencode
|
||||||
@@ -96,7 +94,6 @@ class DeepgramFluxSTTService(WebsocketSTTService):
|
|||||||
mip_opt_out: Optional. Opts out requests from the Deepgram Model Improvement Program
|
mip_opt_out: Optional. Opts out requests from the Deepgram Model Improvement Program
|
||||||
(default False).
|
(default False).
|
||||||
tag: List of tags to label requests for identification during usage reporting.
|
tag: List of tags to label requests for identification during usage reporting.
|
||||||
min_confidence: Optional. Minimum confidence required confidence to create a TranscriptionFrame
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
eager_eot_threshold: Optional[float] = None
|
eager_eot_threshold: Optional[float] = None
|
||||||
@@ -105,7 +102,6 @@ class DeepgramFluxSTTService(WebsocketSTTService):
|
|||||||
keyterm: list = []
|
keyterm: list = []
|
||||||
mip_opt_out: Optional[bool] = None
|
mip_opt_out: Optional[bool] = None
|
||||||
tag: list = []
|
tag: list = []
|
||||||
min_confidence: Optional[float] = None # New parameter
|
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
self,
|
self,
|
||||||
@@ -150,17 +146,7 @@ class DeepgramFluxSTTService(WebsocketSTTService):
|
|||||||
params=params
|
params=params
|
||||||
)
|
)
|
||||||
"""
|
"""
|
||||||
# Note: For DeepgramFluxSTTService, differently from other processes, we need to create
|
super().__init__(sample_rate=sample_rate, **kwargs)
|
||||||
# the _receive_task inside _connect_websocket, because the websocket should only be
|
|
||||||
# considered connected and ready to send audio once we receive from Flux the message
|
|
||||||
# which confirms the connection has been established.
|
|
||||||
# If we try to keep the logic reconnect_on_error, when receiving a message, the
|
|
||||||
# _receive_task_handler would try to reconnect in case of error, invoking the
|
|
||||||
# _connect_websocket again and leading to a case where the first _receive_task_handler
|
|
||||||
# was never destroyed.
|
|
||||||
# So we can keep it here as false, because inside the method send_with_retry, it will
|
|
||||||
# already try to reconnect if needed.
|
|
||||||
super().__init__(sample_rate=sample_rate, reconnect_on_error=False, **kwargs)
|
|
||||||
|
|
||||||
self._api_key = api_key
|
self._api_key = api_key
|
||||||
self._url = url
|
self._url = url
|
||||||
@@ -177,13 +163,6 @@ class DeepgramFluxSTTService(WebsocketSTTService):
|
|||||||
self._register_event_handler("on_end_of_turn")
|
self._register_event_handler("on_end_of_turn")
|
||||||
self._register_event_handler("on_eager_end_of_turn")
|
self._register_event_handler("on_eager_end_of_turn")
|
||||||
self._register_event_handler("on_update")
|
self._register_event_handler("on_update")
|
||||||
self._connection_established_event = asyncio.Event()
|
|
||||||
# Watchdog task to prevent dangling tasks
|
|
||||||
# If we stop sending audio to Flux after we have received that the User has started speaking
|
|
||||||
# we never receive the user stopped speaking event unless we resume sending audio to it.
|
|
||||||
self._last_stt_time = None
|
|
||||||
self._watchdog_task = None
|
|
||||||
self._user_is_speaking = False
|
|
||||||
|
|
||||||
async def _connect(self):
|
async def _connect(self):
|
||||||
"""Connect to WebSocket and start background tasks.
|
"""Connect to WebSocket and start background tasks.
|
||||||
@@ -193,6 +172,9 @@ class DeepgramFluxSTTService(WebsocketSTTService):
|
|||||||
"""
|
"""
|
||||||
await self._connect_websocket()
|
await self._connect_websocket()
|
||||||
|
|
||||||
|
if self._websocket and not self._receive_task:
|
||||||
|
self._receive_task = self.create_task(self._receive_task_handler(self._report_error))
|
||||||
|
|
||||||
async def _disconnect(self):
|
async def _disconnect(self):
|
||||||
"""Disconnect from WebSocket and clean up tasks.
|
"""Disconnect from WebSocket and clean up tasks.
|
||||||
|
|
||||||
@@ -200,32 +182,21 @@ class DeepgramFluxSTTService(WebsocketSTTService):
|
|||||||
and cleans up resources to prevent memory leaks.
|
and cleans up resources to prevent memory leaks.
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
|
# Cancel background tasks BEFORE closing websocket
|
||||||
|
if self._receive_task:
|
||||||
|
await self.cancel_task(self._receive_task, timeout=2.0)
|
||||||
|
self._receive_task = None
|
||||||
|
|
||||||
|
# Now close the websocket
|
||||||
await self._disconnect_websocket()
|
await self._disconnect_websocket()
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
finally:
|
finally:
|
||||||
# Reset state only after everything is cleaned up
|
# Reset state only after everything is cleaned up
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
|
|
||||||
async def _send_silence(self, duration_secs: float = 0.5):
|
|
||||||
"""Send a block of silence of the specified duration (default 500 ms)."""
|
|
||||||
sample_width = 2 # bytes per sample for 16-bit PCM
|
|
||||||
num_channels = 1 # mono
|
|
||||||
num_samples = int(self.sample_rate * duration_secs)
|
|
||||||
silence = b"\x00" * (num_samples * sample_width * num_channels)
|
|
||||||
await self._websocket.send(silence)
|
|
||||||
|
|
||||||
async def _watchdog_task_handler(self):
|
|
||||||
while self._websocket and self._websocket.state is State.OPEN:
|
|
||||||
now = time.monotonic()
|
|
||||||
# More than 500 ms without sending new audio to Flux
|
|
||||||
if self._user_is_speaking and self._last_stt_time and now - self._last_stt_time > 0.5:
|
|
||||||
logger.warning("Sending silence to Flux to prevent dangling task")
|
|
||||||
await self._send_silence()
|
|
||||||
self._last_stt_time = time.monotonic()
|
|
||||||
# check every 100ms
|
|
||||||
await asyncio.sleep(0.1)
|
|
||||||
|
|
||||||
async def _connect_websocket(self):
|
async def _connect_websocket(self):
|
||||||
"""Establish WebSocket connection to API.
|
"""Establish WebSocket connection to API.
|
||||||
|
|
||||||
@@ -237,30 +208,15 @@ class DeepgramFluxSTTService(WebsocketSTTService):
|
|||||||
if self._websocket and self._websocket.state is State.OPEN:
|
if self._websocket and self._websocket.state is State.OPEN:
|
||||||
return
|
return
|
||||||
|
|
||||||
self._connection_established_event.clear()
|
|
||||||
self._user_is_speaking = False
|
|
||||||
self._websocket = await websocket_connect(
|
self._websocket = await websocket_connect(
|
||||||
self._websocket_url,
|
self._websocket_url,
|
||||||
additional_headers={"Authorization": f"Token {self._api_key}"},
|
additional_headers={"Authorization": f"Token {self._api_key}"},
|
||||||
)
|
)
|
||||||
|
|
||||||
# Creating the receiver task
|
|
||||||
if not self._receive_task:
|
|
||||||
self._receive_task = self.create_task(
|
|
||||||
self._receive_task_handler(self._report_error)
|
|
||||||
)
|
|
||||||
|
|
||||||
# Creating the watchdog task
|
|
||||||
if not self._watchdog_task:
|
|
||||||
self._watchdog_task = self.create_task(self._watchdog_task_handler())
|
|
||||||
|
|
||||||
# Now wait for the connection established event
|
|
||||||
logger.debug("WebSocket connected, waiting for server confirmation...")
|
|
||||||
await self._connection_established_event.wait()
|
|
||||||
logger.debug("Connected to Deepgram Flux Websocket")
|
logger.debug("Connected to Deepgram Flux Websocket")
|
||||||
await self._call_event_handler("on_connected")
|
await self._call_event_handler("on_connected")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
await self._call_event_handler("on_connection_error", f"{e}")
|
await self._call_event_handler("on_connection_error", f"{e}")
|
||||||
|
|
||||||
@@ -271,16 +227,6 @@ class DeepgramFluxSTTService(WebsocketSTTService):
|
|||||||
metrics collection. Handles disconnection errors gracefully.
|
metrics collection. Handles disconnection errors gracefully.
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
# Cancel background tasks BEFORE closing websocket
|
|
||||||
if self._receive_task:
|
|
||||||
await self.cancel_task(self._receive_task, timeout=2.0)
|
|
||||||
self._receive_task = None
|
|
||||||
if self._watchdog_task:
|
|
||||||
await self.cancel_task(self._watchdog_task, timeout=2.0)
|
|
||||||
self._watchdog_task = None
|
|
||||||
self._last_stt_time = None
|
|
||||||
|
|
||||||
self._connection_established_event.clear()
|
|
||||||
await self.stop_all_metrics()
|
await self.stop_all_metrics()
|
||||||
|
|
||||||
if self._websocket:
|
if self._websocket:
|
||||||
@@ -288,7 +234,8 @@ class DeepgramFluxSTTService(WebsocketSTTService):
|
|||||||
logger.debug("Disconnecting from Deepgram Flux Websocket")
|
logger.debug("Disconnecting from Deepgram Flux Websocket")
|
||||||
await self._websocket.close()
|
await self._websocket.close()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error closing websocket: {e}", exception=e)
|
logger.error(f"{self} error closing websocket: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
finally:
|
finally:
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
await self._call_event_handler("on_disconnected")
|
await self._call_event_handler("on_disconnected")
|
||||||
@@ -298,13 +245,10 @@ class DeepgramFluxSTTService(WebsocketSTTService):
|
|||||||
|
|
||||||
This signals to the server that no more audio data will be sent.
|
This signals to the server that no more audio data will be sent.
|
||||||
"""
|
"""
|
||||||
try:
|
if self._websocket:
|
||||||
if self._websocket:
|
logger.debug("Sending CloseStream message to Deepgram Flux")
|
||||||
logger.debug("Sending CloseStream message to Deepgram Flux")
|
message = {"type": "CloseStream"}
|
||||||
message = {"type": "CloseStream"}
|
await self._websocket.send(json.dumps(message))
|
||||||
await self._websocket.send(json.dumps(message))
|
|
||||||
except Exception as e:
|
|
||||||
await self.push_error(error_msg=f"Error sending closeStream: {e}", exception=e)
|
|
||||||
|
|
||||||
def can_generate_metrics(self) -> bool:
|
def can_generate_metrics(self) -> bool:
|
||||||
"""Check if this service can generate processing metrics.
|
"""Check if this service can generate processing metrics.
|
||||||
@@ -391,13 +335,15 @@ class DeepgramFluxSTTService(WebsocketSTTService):
|
|||||||
are issues sending the audio data.
|
are issues sending the audio data.
|
||||||
"""
|
"""
|
||||||
if not self._websocket:
|
if not self._websocket:
|
||||||
|
logger.error("Not connected to Deepgram Flux.")
|
||||||
|
yield ErrorFrame("Not connected to Deepgram Flux.")
|
||||||
return
|
return
|
||||||
|
|
||||||
try:
|
try:
|
||||||
self._last_stt_time = time.monotonic()
|
await self._websocket.send(audio)
|
||||||
await self.send_with_retry(audio, self._report_error)
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
return
|
return
|
||||||
|
|
||||||
yield None
|
yield None
|
||||||
@@ -474,7 +420,8 @@ class DeepgramFluxSTTService(WebsocketSTTService):
|
|||||||
# Skip malformed messages
|
# Skip malformed messages
|
||||||
continue
|
continue
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
# Error will be handled inside WebsocketService->_receive_task_handler
|
# Error will be handled inside WebsocketService->_receive_task_handler
|
||||||
raise
|
raise
|
||||||
else:
|
else:
|
||||||
@@ -516,8 +463,6 @@ class DeepgramFluxSTTService(WebsocketSTTService):
|
|||||||
transcription processing.
|
transcription processing.
|
||||||
"""
|
"""
|
||||||
logger.info("Connected to Flux - ready to stream audio")
|
logger.info("Connected to Flux - ready to stream audio")
|
||||||
# Notify connection is established
|
|
||||||
self._connection_established_event.set()
|
|
||||||
|
|
||||||
async def _handle_fatal_error(self, data: Dict[str, Any]):
|
async def _handle_fatal_error(self, data: Dict[str, Any]):
|
||||||
"""Handle fatal error messages from Deepgram Flux.
|
"""Handle fatal error messages from Deepgram Flux.
|
||||||
@@ -585,7 +530,6 @@ class DeepgramFluxSTTService(WebsocketSTTService):
|
|||||||
transcript: maybe the first few words of the turn.
|
transcript: maybe the first few words of the turn.
|
||||||
"""
|
"""
|
||||||
logger.debug("User started speaking")
|
logger.debug("User started speaking")
|
||||||
self._user_is_speaking = True
|
|
||||||
await self.push_interruption_task_frame_and_wait()
|
await self.push_interruption_task_frame_and_wait()
|
||||||
await self.broadcast_frame(UserStartedSpeakingFrame)
|
await self.broadcast_frame(UserStartedSpeakingFrame)
|
||||||
await self.start_metrics()
|
await self.start_metrics()
|
||||||
@@ -606,22 +550,6 @@ class DeepgramFluxSTTService(WebsocketSTTService):
|
|||||||
logger.trace(f"Received event TurnResumed: {event}")
|
logger.trace(f"Received event TurnResumed: {event}")
|
||||||
await self._call_event_handler("on_turn_resumed")
|
await self._call_event_handler("on_turn_resumed")
|
||||||
|
|
||||||
def _calculate_average_confidence(self, transcript_data) -> Optional[float]:
|
|
||||||
"""Calculate the average confidence from transcript data.
|
|
||||||
|
|
||||||
Return None if the data is missing or invalid.
|
|
||||||
"""
|
|
||||||
# Example: Assume transcript_data has a list of words with confidence
|
|
||||||
words = transcript_data.get("words")
|
|
||||||
if not words or not isinstance(words, list):
|
|
||||||
return None
|
|
||||||
confidences = [
|
|
||||||
w.get("confidence") for w in words if isinstance(w.get("confidence"), (float, int))
|
|
||||||
]
|
|
||||||
if not confidences:
|
|
||||||
return None
|
|
||||||
return sum(confidences) / len(confidences)
|
|
||||||
|
|
||||||
async def _handle_end_of_turn(self, transcript: str, data: Dict[str, Any]):
|
async def _handle_end_of_turn(self, transcript: str, data: Dict[str, Any]):
|
||||||
"""Handle EndOfTurn events from Deepgram Flux.
|
"""Handle EndOfTurn events from Deepgram Flux.
|
||||||
|
|
||||||
@@ -641,26 +569,16 @@ class DeepgramFluxSTTService(WebsocketSTTService):
|
|||||||
data: The TurnInfo message data containing event type, transcript and some extra metadata.
|
data: The TurnInfo message data containing event type, transcript and some extra metadata.
|
||||||
"""
|
"""
|
||||||
logger.debug("User stopped speaking")
|
logger.debug("User stopped speaking")
|
||||||
self._user_is_speaking = False
|
|
||||||
|
|
||||||
# Compute the average confidence
|
await self.push_frame(
|
||||||
average_confidence = self._calculate_average_confidence(data)
|
TranscriptionFrame(
|
||||||
|
transcript,
|
||||||
if not self._params.min_confidence or average_confidence > self._params.min_confidence:
|
self._user_id,
|
||||||
await self.push_frame(
|
time_now_iso8601(),
|
||||||
TranscriptionFrame(
|
self._language,
|
||||||
transcript,
|
result=data,
|
||||||
self._user_id,
|
|
||||||
time_now_iso8601(),
|
|
||||||
self._language,
|
|
||||||
result=data,
|
|
||||||
)
|
|
||||||
)
|
)
|
||||||
else:
|
)
|
||||||
logger.warning(
|
|
||||||
f"Transcription confidence below min_confidence threshold: {average_confidence}"
|
|
||||||
)
|
|
||||||
|
|
||||||
await self._handle_transcription(transcript, True, self._language)
|
await self._handle_transcription(transcript, True, self._language)
|
||||||
await self.stop_processing_metrics()
|
await self.stop_processing_metrics()
|
||||||
await self.push_frame(UserStoppedSpeakingFrame(), FrameDirection.DOWNSTREAM)
|
await self.push_frame(UserStoppedSpeakingFrame(), FrameDirection.DOWNSTREAM)
|
||||||
|
|||||||
@@ -233,7 +233,7 @@ class DeepgramSTTService(STTService):
|
|||||||
)
|
)
|
||||||
|
|
||||||
if not await self._connection.start(options=self._settings, addons=self._addons):
|
if not await self._connection.start(options=self._settings, addons=self._addons):
|
||||||
await self.push_error(error_msg=f"Unable to connect to Deepgram")
|
logger.error(f"{self}: unable to connect to Deepgram")
|
||||||
|
|
||||||
async def _disconnect(self):
|
async def _disconnect(self):
|
||||||
if await self._connection.is_connected():
|
if await self._connection.is_connected():
|
||||||
@@ -256,7 +256,7 @@ class DeepgramSTTService(STTService):
|
|||||||
async def _on_error(self, *args, **kwargs):
|
async def _on_error(self, *args, **kwargs):
|
||||||
error: ErrorResponse = kwargs["error"]
|
error: ErrorResponse = kwargs["error"]
|
||||||
logger.warning(f"{self} connection error, will retry: {error}")
|
logger.warning(f"{self} connection error, will retry: {error}")
|
||||||
await self.push_error(error_msg=f"{error}")
|
await self.push_error(ErrorFrame(error=f"{error}"))
|
||||||
await self.stop_all_metrics()
|
await self.stop_all_metrics()
|
||||||
# NOTE(aleix): we don't disconnect (i.e. call finish on the connection)
|
# NOTE(aleix): we don't disconnect (i.e. call finish on the connection)
|
||||||
# because this triggers more errors internally in the Deepgram SDK. So,
|
# because this triggers more errors internally in the Deepgram SDK. So,
|
||||||
|
|||||||
@@ -1,444 +0,0 @@
|
|||||||
#
|
|
||||||
# Copyright (c) 2024–2025, Daily
|
|
||||||
#
|
|
||||||
# SPDX-License-Identifier: BSD 2-Clause License
|
|
||||||
#
|
|
||||||
|
|
||||||
"""Deepgram speech-to-text service for AWS SageMaker.
|
|
||||||
|
|
||||||
This module provides a Pipecat STT service that connects to Deepgram models
|
|
||||||
deployed on AWS SageMaker endpoints. Uses HTTP/2 bidirectional streaming for
|
|
||||||
low-latency real-time transcription with support for interim results, multiple
|
|
||||||
languages, and various Deepgram features.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import json
|
|
||||||
from typing import AsyncGenerator, Optional
|
|
||||||
|
|
||||||
from loguru import logger
|
|
||||||
|
|
||||||
from pipecat.frames.frames import (
|
|
||||||
CancelFrame,
|
|
||||||
EndFrame,
|
|
||||||
ErrorFrame,
|
|
||||||
Frame,
|
|
||||||
InterimTranscriptionFrame,
|
|
||||||
StartFrame,
|
|
||||||
TranscriptionFrame,
|
|
||||||
UserStartedSpeakingFrame,
|
|
||||||
UserStoppedSpeakingFrame,
|
|
||||||
)
|
|
||||||
from pipecat.processors.frame_processor import FrameDirection
|
|
||||||
from pipecat.services.aws.sagemaker.bidi_client import SageMakerBidiClient
|
|
||||||
from pipecat.services.stt_service import STTService
|
|
||||||
from pipecat.transcriptions.language import Language
|
|
||||||
from pipecat.utils.time import time_now_iso8601
|
|
||||||
from pipecat.utils.tracing.service_decorators import traced_stt
|
|
||||||
|
|
||||||
try:
|
|
||||||
from deepgram import LiveOptions
|
|
||||||
except ModuleNotFoundError as e:
|
|
||||||
logger.error(f"Exception: {e}")
|
|
||||||
logger.error(
|
|
||||||
"In order to use DeepgramSageMakerSTTService, you need to `pip install pipecat-ai[deepgram,sagemaker]`."
|
|
||||||
)
|
|
||||||
raise Exception(f"Missing module: {e}")
|
|
||||||
|
|
||||||
|
|
||||||
class DeepgramSageMakerSTTService(STTService):
|
|
||||||
"""Deepgram speech-to-text service for AWS SageMaker.
|
|
||||||
|
|
||||||
Provides real-time speech recognition using Deepgram models deployed on
|
|
||||||
AWS SageMaker endpoints. Uses HTTP/2 bidirectional streaming for low-latency
|
|
||||||
transcription with support for interim results, speaker diarization, and
|
|
||||||
multiple languages.
|
|
||||||
|
|
||||||
Requirements:
|
|
||||||
|
|
||||||
- AWS credentials configured (via environment variables, AWS CLI, or instance metadata)
|
|
||||||
- A deployed SageMaker endpoint with Deepgram model: https://developers.deepgram.com/docs/deploy-amazon-sagemaker
|
|
||||||
- Deepgram SDK for LiveOptions configuration
|
|
||||||
|
|
||||||
Example::
|
|
||||||
|
|
||||||
stt = DeepgramSageMakerSTTService(
|
|
||||||
endpoint_name="my-deepgram-endpoint",
|
|
||||||
region="us-east-2",
|
|
||||||
live_options=LiveOptions(
|
|
||||||
model="nova-3",
|
|
||||||
language="en",
|
|
||||||
interim_results=True,
|
|
||||||
punctuate=True,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
*,
|
|
||||||
endpoint_name: str,
|
|
||||||
region: str,
|
|
||||||
sample_rate: Optional[int] = None,
|
|
||||||
live_options: Optional[LiveOptions] = None,
|
|
||||||
**kwargs,
|
|
||||||
):
|
|
||||||
"""Initialize the Deepgram SageMaker STT service.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
endpoint_name: Name of the SageMaker endpoint with Deepgram model
|
|
||||||
deployed (e.g., "my-deepgram-nova-3-endpoint").
|
|
||||||
region: AWS region where the endpoint is deployed (e.g., "us-east-2").
|
|
||||||
sample_rate: Audio sample rate in Hz. If None, uses value from
|
|
||||||
live_options or defaults to the value from StartFrame.
|
|
||||||
live_options: Deepgram LiveOptions for detailed configuration. If None,
|
|
||||||
uses sensible defaults (nova-3 model, English, interim results enabled).
|
|
||||||
**kwargs: Additional arguments passed to the parent STTService.
|
|
||||||
"""
|
|
||||||
sample_rate = sample_rate or (live_options.sample_rate if live_options else None)
|
|
||||||
super().__init__(sample_rate=sample_rate, **kwargs)
|
|
||||||
|
|
||||||
self._endpoint_name = endpoint_name
|
|
||||||
self._region = region
|
|
||||||
|
|
||||||
# Create default options similar to DeepgramSTTService
|
|
||||||
default_options = LiveOptions(
|
|
||||||
encoding="linear16",
|
|
||||||
language=Language.EN,
|
|
||||||
model="nova-3",
|
|
||||||
channels=1,
|
|
||||||
interim_results=True,
|
|
||||||
punctuate=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Merge with provided options
|
|
||||||
merged_options = default_options.to_dict()
|
|
||||||
if live_options:
|
|
||||||
default_model = default_options.model
|
|
||||||
merged_options.update(live_options.to_dict())
|
|
||||||
# Handle the "None" string bug from deepgram-sdk
|
|
||||||
if "model" in merged_options and merged_options["model"] == "None":
|
|
||||||
merged_options["model"] = default_model
|
|
||||||
|
|
||||||
# Convert Language enum to string if needed
|
|
||||||
if "language" in merged_options and isinstance(merged_options["language"], Language):
|
|
||||||
merged_options["language"] = merged_options["language"].value
|
|
||||||
|
|
||||||
self.set_model_name(merged_options["model"])
|
|
||||||
self._settings = merged_options
|
|
||||||
|
|
||||||
self._client: Optional[SageMakerBidiClient] = None
|
|
||||||
self._response_task: Optional[asyncio.Task] = None
|
|
||||||
self._keepalive_task: Optional[asyncio.Task] = None
|
|
||||||
|
|
||||||
def can_generate_metrics(self) -> bool:
|
|
||||||
"""Check if this service can generate processing metrics.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
True, as Deepgram SageMaker service supports metrics generation.
|
|
||||||
"""
|
|
||||||
return True
|
|
||||||
|
|
||||||
async def set_model(self, model: str):
|
|
||||||
"""Set the Deepgram model and reconnect.
|
|
||||||
|
|
||||||
Disconnects from the current session, updates the model setting, and
|
|
||||||
establishes a new connection with the updated model.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
model: The Deepgram model name to use (e.g., "nova-3").
|
|
||||||
"""
|
|
||||||
await super().set_model(model)
|
|
||||||
logger.info(f"Switching STT model to: [{model}]")
|
|
||||||
self._settings["model"] = model
|
|
||||||
await self._disconnect()
|
|
||||||
await self._connect()
|
|
||||||
|
|
||||||
async def set_language(self, language: Language):
|
|
||||||
"""Set the recognition language and reconnect.
|
|
||||||
|
|
||||||
Disconnects from the current session, updates the language setting, and
|
|
||||||
establishes a new connection with the updated language.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
language: The language to use for speech recognition (e.g., Language.EN,
|
|
||||||
Language.ES).
|
|
||||||
"""
|
|
||||||
logger.info(f"Switching STT language to: [{language}]")
|
|
||||||
self._settings["language"] = language
|
|
||||||
await self._disconnect()
|
|
||||||
await self._connect()
|
|
||||||
|
|
||||||
async def start(self, frame: StartFrame):
|
|
||||||
"""Start the Deepgram SageMaker STT service.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
frame: The start frame containing initialization parameters.
|
|
||||||
"""
|
|
||||||
await super().start(frame)
|
|
||||||
self._settings["sample_rate"] = self.sample_rate
|
|
||||||
await self._connect()
|
|
||||||
|
|
||||||
async def stop(self, frame: EndFrame):
|
|
||||||
"""Stop the Deepgram SageMaker STT service.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
frame: The end frame.
|
|
||||||
"""
|
|
||||||
await super().stop(frame)
|
|
||||||
await self._disconnect()
|
|
||||||
|
|
||||||
async def cancel(self, frame: CancelFrame):
|
|
||||||
"""Cancel the Deepgram SageMaker STT service.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
frame: The cancel frame.
|
|
||||||
"""
|
|
||||||
await super().cancel(frame)
|
|
||||||
await self._disconnect()
|
|
||||||
|
|
||||||
async def run_stt(self, audio: bytes) -> AsyncGenerator[Frame, None]:
|
|
||||||
"""Send audio data to Deepgram for transcription.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
audio: Raw audio bytes to transcribe.
|
|
||||||
|
|
||||||
Yields:
|
|
||||||
Frame: None (transcription results come via BiDi stream callbacks).
|
|
||||||
"""
|
|
||||||
if self._client and self._client.is_active:
|
|
||||||
try:
|
|
||||||
await self._client.send_audio_chunk(audio)
|
|
||||||
except Exception as e:
|
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
|
||||||
yield None
|
|
||||||
|
|
||||||
async def _connect(self):
|
|
||||||
"""Connect to the SageMaker endpoint and start the BiDi session.
|
|
||||||
|
|
||||||
Builds the Deepgram query string from settings, creates the BiDi client,
|
|
||||||
starts the streaming session, and launches background tasks for processing
|
|
||||||
responses and sending KeepAlive messages.
|
|
||||||
"""
|
|
||||||
logger.debug("Connecting to Deepgram on SageMaker...")
|
|
||||||
|
|
||||||
# Update sample rate in settings
|
|
||||||
self._settings["sample_rate"] = self.sample_rate
|
|
||||||
|
|
||||||
# Build query string from settings, converting booleans to strings
|
|
||||||
query_params = {}
|
|
||||||
for key, value in self._settings.items():
|
|
||||||
if value is not None:
|
|
||||||
# Convert boolean values to lowercase strings for Deepgram API
|
|
||||||
if isinstance(value, bool):
|
|
||||||
query_params[key] = str(value).lower()
|
|
||||||
else:
|
|
||||||
query_params[key] = str(value)
|
|
||||||
|
|
||||||
query_string = "&".join(f"{k}={v}" for k, v in query_params.items())
|
|
||||||
|
|
||||||
# Create BiDi client
|
|
||||||
self._client = SageMakerBidiClient(
|
|
||||||
endpoint_name=self._endpoint_name,
|
|
||||||
region=self._region,
|
|
||||||
model_invocation_path="v1/listen",
|
|
||||||
model_query_string=query_string,
|
|
||||||
)
|
|
||||||
|
|
||||||
try:
|
|
||||||
# Start the session
|
|
||||||
await self._client.start_session()
|
|
||||||
|
|
||||||
# Start processing responses in the background
|
|
||||||
self._response_task = self.create_task(self._process_responses())
|
|
||||||
|
|
||||||
# Start keepalive task to maintain connection
|
|
||||||
self._keepalive_task = self.create_task(self._send_keepalive())
|
|
||||||
|
|
||||||
logger.debug("Connected to Deepgram on SageMaker")
|
|
||||||
await self._call_event_handler("on_connected")
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
|
||||||
await self._call_event_handler("on_connection_error", str(e))
|
|
||||||
|
|
||||||
async def _disconnect(self):
|
|
||||||
"""Disconnect from the SageMaker endpoint.
|
|
||||||
|
|
||||||
Sends a CloseStream message to Deepgram, cancels background tasks
|
|
||||||
(KeepAlive and response processing), and closes the BiDi session.
|
|
||||||
Safe to call multiple times.
|
|
||||||
"""
|
|
||||||
if self._client and self._client.is_active:
|
|
||||||
logger.debug("Disconnecting from Deepgram on SageMaker...")
|
|
||||||
|
|
||||||
# Send CloseStream message to Deepgram
|
|
||||||
try:
|
|
||||||
await self._client.send_json({"type": "CloseStream"})
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning(f"Failed to send CloseStream message: {e}")
|
|
||||||
|
|
||||||
# Cancel keepalive task
|
|
||||||
if self._keepalive_task and not self._keepalive_task.done():
|
|
||||||
await self.cancel_task(self._keepalive_task)
|
|
||||||
|
|
||||||
# Cancel response processing task
|
|
||||||
if self._response_task and not self._response_task.done():
|
|
||||||
await self.cancel_task(self._response_task)
|
|
||||||
|
|
||||||
# Close the BiDi session
|
|
||||||
await self._client.close_session()
|
|
||||||
|
|
||||||
logger.debug("Disconnected from Deepgram on SageMaker")
|
|
||||||
await self._call_event_handler("on_disconnected")
|
|
||||||
|
|
||||||
async def _send_keepalive(self):
|
|
||||||
"""Send periodic KeepAlive messages to maintain the connection.
|
|
||||||
|
|
||||||
Sends a KeepAlive JSON message to Deepgram every 5 seconds while the
|
|
||||||
connection is active. This prevents the connection from timing out during
|
|
||||||
periods of silence.
|
|
||||||
"""
|
|
||||||
while self._client and self._client.is_active:
|
|
||||||
await asyncio.sleep(5)
|
|
||||||
if self._client and self._client.is_active:
|
|
||||||
try:
|
|
||||||
await self._client.send_json({"type": "KeepAlive"})
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning(f"Failed to send KeepAlive: {e}")
|
|
||||||
|
|
||||||
async def _process_responses(self):
|
|
||||||
"""Process streaming responses from Deepgram on SageMaker.
|
|
||||||
|
|
||||||
Continuously receives responses from the BiDi stream, decodes the payload,
|
|
||||||
parses JSON responses from Deepgram, and processes transcription results.
|
|
||||||
Runs as a background task until the connection is closed or cancelled.
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
while self._client and self._client.is_active:
|
|
||||||
result = await self._client.receive_response()
|
|
||||||
|
|
||||||
if result is None:
|
|
||||||
break
|
|
||||||
|
|
||||||
# Check if this is a PayloadPart with bytes
|
|
||||||
if hasattr(result, "value") and hasattr(result.value, "bytes_"):
|
|
||||||
if result.value.bytes_:
|
|
||||||
response_data = result.value.bytes_.decode("utf-8")
|
|
||||||
|
|
||||||
try:
|
|
||||||
# Parse JSON response from Deepgram
|
|
||||||
parsed = json.loads(response_data)
|
|
||||||
|
|
||||||
# Extract and process transcript if available
|
|
||||||
if "channel" in parsed:
|
|
||||||
await self._handle_transcript_response(parsed)
|
|
||||||
|
|
||||||
except json.JSONDecodeError:
|
|
||||||
logger.warning(f"Non-JSON response: {response_data}")
|
|
||||||
|
|
||||||
except asyncio.CancelledError:
|
|
||||||
logger.debug("Response processor cancelled")
|
|
||||||
except Exception as e:
|
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
|
||||||
finally:
|
|
||||||
logger.debug("Response processor stopped")
|
|
||||||
|
|
||||||
async def _handle_transcript_response(self, parsed: dict):
|
|
||||||
"""Handle a transcript response from Deepgram.
|
|
||||||
|
|
||||||
Extracts the transcript text, determines if it's final or interim, extracts
|
|
||||||
language information, and pushes the appropriate frame (TranscriptionFrame
|
|
||||||
or InterimTranscriptionFrame) downstream.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
parsed: The parsed JSON response from Deepgram containing channel,
|
|
||||||
alternatives, transcript, and metadata.
|
|
||||||
"""
|
|
||||||
alternatives = parsed.get("channel", {}).get("alternatives", [])
|
|
||||||
if not alternatives or not alternatives[0].get("transcript"):
|
|
||||||
return
|
|
||||||
|
|
||||||
transcript = alternatives[0]["transcript"]
|
|
||||||
if not transcript.strip():
|
|
||||||
return
|
|
||||||
|
|
||||||
# Stop TTFB metrics on first transcript
|
|
||||||
await self.stop_ttfb_metrics()
|
|
||||||
|
|
||||||
is_final = parsed.get("is_final", False)
|
|
||||||
speech_final = parsed.get("speech_final", False)
|
|
||||||
|
|
||||||
# Extract language if available
|
|
||||||
language = None
|
|
||||||
if alternatives[0].get("languages"):
|
|
||||||
language = alternatives[0]["languages"][0]
|
|
||||||
language = Language(language)
|
|
||||||
|
|
||||||
if is_final and speech_final:
|
|
||||||
# Final transcription
|
|
||||||
await self.push_frame(
|
|
||||||
TranscriptionFrame(
|
|
||||||
transcript,
|
|
||||||
self._user_id,
|
|
||||||
time_now_iso8601(),
|
|
||||||
language,
|
|
||||||
result=parsed,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
await self._handle_transcription(transcript, is_final, language)
|
|
||||||
await self.stop_processing_metrics()
|
|
||||||
else:
|
|
||||||
# Interim transcription
|
|
||||||
await self.push_frame(
|
|
||||||
InterimTranscriptionFrame(
|
|
||||||
transcript,
|
|
||||||
self._user_id,
|
|
||||||
time_now_iso8601(),
|
|
||||||
language,
|
|
||||||
result=parsed,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
@traced_stt
|
|
||||||
async def _handle_transcription(
|
|
||||||
self, transcript: str, is_final: bool, language: Optional[Language] = None
|
|
||||||
):
|
|
||||||
"""Handle a transcription result with tracing.
|
|
||||||
|
|
||||||
This method is decorated with @traced_stt for observability and tracing
|
|
||||||
integration. The actual transcription processing is handled by the parent
|
|
||||||
class and observers.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
transcript: The transcribed text.
|
|
||||||
is_final: Whether this is a final transcription result.
|
|
||||||
language: The detected language of the transcription, if available.
|
|
||||||
"""
|
|
||||||
pass
|
|
||||||
|
|
||||||
async def start_metrics(self):
|
|
||||||
"""Start TTFB and processing metrics collection."""
|
|
||||||
await self.start_ttfb_metrics()
|
|
||||||
await self.start_processing_metrics()
|
|
||||||
|
|
||||||
async def process_frame(self, frame: Frame, direction: FrameDirection):
|
|
||||||
"""Process frames with Deepgram SageMaker-specific handling.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
frame: The frame to process.
|
|
||||||
direction: The direction of frame processing.
|
|
||||||
"""
|
|
||||||
await super().process_frame(frame, direction)
|
|
||||||
|
|
||||||
# Start metrics when user starts speaking (if VAD is not provided by Deepgram)
|
|
||||||
if isinstance(frame, UserStartedSpeakingFrame):
|
|
||||||
await self.start_metrics()
|
|
||||||
elif isinstance(frame, UserStoppedSpeakingFrame):
|
|
||||||
# Send finalize message to Deepgram when user stops speaking
|
|
||||||
# This tells Deepgram to flush any remaining audio and return final results
|
|
||||||
if self._client and self._client.is_active:
|
|
||||||
try:
|
|
||||||
await self._client.send_json({"type": "Finalize"})
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning(f"Error sending Finalize message: {e}")
|
|
||||||
@@ -10,45 +10,35 @@ This module provides integration with Deepgram's text-to-speech API
|
|||||||
for generating speech from text using various voice models.
|
for generating speech from text using various voice models.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import json
|
|
||||||
from typing import AsyncGenerator, Optional
|
from typing import AsyncGenerator, Optional
|
||||||
|
|
||||||
import aiohttp
|
import aiohttp
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
from pipecat.frames.frames import (
|
from pipecat.frames.frames import (
|
||||||
CancelFrame,
|
|
||||||
EndFrame,
|
|
||||||
ErrorFrame,
|
ErrorFrame,
|
||||||
Frame,
|
Frame,
|
||||||
InterruptionFrame,
|
|
||||||
LLMFullResponseEndFrame,
|
|
||||||
StartFrame,
|
|
||||||
TTSAudioRawFrame,
|
TTSAudioRawFrame,
|
||||||
TTSStartedFrame,
|
TTSStartedFrame,
|
||||||
TTSStoppedFrame,
|
TTSStoppedFrame,
|
||||||
)
|
)
|
||||||
from pipecat.processors.frame_processor import FrameDirection
|
from pipecat.services.tts_service import TTSService
|
||||||
from pipecat.services.tts_service import TTSService, WebsocketTTSService
|
|
||||||
from pipecat.utils.tracing.service_decorators import traced_tts
|
from pipecat.utils.tracing.service_decorators import traced_tts
|
||||||
|
|
||||||
try:
|
try:
|
||||||
from websockets.asyncio.client import connect as websocket_connect
|
from deepgram import DeepgramClient, DeepgramClientOptions, SpeakOptions
|
||||||
from websockets.protocol import State
|
|
||||||
except ModuleNotFoundError as e:
|
except ModuleNotFoundError as e:
|
||||||
logger.error(f"Exception: {e}")
|
logger.error(f"Exception: {e}")
|
||||||
logger.error(
|
logger.error("In order to use Deepgram, you need to `pip install pipecat-ai[deepgram]`.")
|
||||||
"In order to use DeepgramWebsocketTTSService, you need to `pip install pipecat-ai[deepgram]`."
|
|
||||||
)
|
|
||||||
raise Exception(f"Missing module: {e}")
|
raise Exception(f"Missing module: {e}")
|
||||||
|
|
||||||
|
|
||||||
class DeepgramTTSService(WebsocketTTSService):
|
class DeepgramTTSService(TTSService):
|
||||||
"""Deepgram WebSocket-based text-to-speech service.
|
"""Deepgram text-to-speech service.
|
||||||
|
|
||||||
Provides real-time text-to-speech synthesis using Deepgram's WebSocket API.
|
Provides text-to-speech synthesis using Deepgram's streaming API.
|
||||||
Supports streaming audio generation with interruption handling via the Clear
|
Supports various voice models and audio encoding formats with
|
||||||
message for conversational AI use cases.
|
configurable sample rates and quality settings.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
@@ -56,211 +46,51 @@ class DeepgramTTSService(WebsocketTTSService):
|
|||||||
*,
|
*,
|
||||||
api_key: str,
|
api_key: str,
|
||||||
voice: str = "aura-2-helena-en",
|
voice: str = "aura-2-helena-en",
|
||||||
base_url: str = "wss://api.deepgram.com",
|
base_url: str = "",
|
||||||
sample_rate: Optional[int] = None,
|
sample_rate: Optional[int] = None,
|
||||||
encoding: str = "linear16",
|
encoding: str = "linear16",
|
||||||
**kwargs,
|
**kwargs,
|
||||||
):
|
):
|
||||||
"""Initialize the Deepgram WebSocket TTS service.
|
"""Initialize the Deepgram TTS service.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
api_key: Deepgram API key for authentication.
|
api_key: Deepgram API key for authentication.
|
||||||
voice: Voice model to use for synthesis. Defaults to "aura-2-helena-en".
|
voice: Voice model to use for synthesis. Defaults to "aura-2-helena-en".
|
||||||
base_url: WebSocket base URL for Deepgram API. Defaults to "wss://api.deepgram.com".
|
base_url: Custom base URL for Deepgram API. Uses default if empty.
|
||||||
sample_rate: Audio sample rate in Hz. If None, uses service default.
|
sample_rate: Audio sample rate in Hz. If None, uses service default.
|
||||||
encoding: Audio encoding format. Defaults to "linear16".
|
encoding: Audio encoding format. Defaults to "linear16".
|
||||||
**kwargs: Additional arguments passed to parent InterruptibleTTSService class.
|
**kwargs: Additional arguments passed to parent TTSService class.
|
||||||
"""
|
"""
|
||||||
super().__init__(sample_rate=sample_rate, **kwargs)
|
super().__init__(sample_rate=sample_rate, **kwargs)
|
||||||
|
|
||||||
self._api_key = api_key
|
|
||||||
self._base_url = base_url
|
|
||||||
self._settings = {
|
self._settings = {
|
||||||
"encoding": encoding,
|
"encoding": encoding,
|
||||||
}
|
}
|
||||||
self.set_voice(voice)
|
self.set_voice(voice)
|
||||||
|
|
||||||
self._receive_task = None
|
client_options = DeepgramClientOptions(url=base_url)
|
||||||
|
self._deepgram_client = DeepgramClient(api_key, config=client_options)
|
||||||
|
|
||||||
def can_generate_metrics(self) -> bool:
|
def can_generate_metrics(self) -> bool:
|
||||||
"""Check if the service can generate metrics.
|
"""Check if the service can generate metrics.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
True, as Deepgram WebSocket TTS service supports metrics generation.
|
True, as Deepgram TTS service supports metrics generation.
|
||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
async def start(self, frame: StartFrame):
|
@property
|
||||||
"""Start the Deepgram WebSocket TTS service.
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that Deepgram TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
Args:
|
Returns:
|
||||||
frame: The start frame containing initialization parameters.
|
True, indicating that Deepgram's text frames include necessary inter-frame spaces.
|
||||||
"""
|
"""
|
||||||
await super().start(frame)
|
return True
|
||||||
await self._connect()
|
|
||||||
|
|
||||||
async def stop(self, frame: EndFrame):
|
|
||||||
"""Stop the Deepgram WebSocket TTS service.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
frame: The end frame.
|
|
||||||
"""
|
|
||||||
await super().stop(frame)
|
|
||||||
await self._disconnect()
|
|
||||||
|
|
||||||
async def cancel(self, frame: CancelFrame):
|
|
||||||
"""Cancel the Deepgram WebSocket TTS service.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
frame: The cancel frame.
|
|
||||||
"""
|
|
||||||
await super().cancel(frame)
|
|
||||||
await self._disconnect()
|
|
||||||
|
|
||||||
async def process_frame(self, frame: Frame, direction: FrameDirection):
|
|
||||||
"""Process frames with special handling for LLM response end.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
frame: The frame to process.
|
|
||||||
direction: The direction of frame processing.
|
|
||||||
"""
|
|
||||||
await super().process_frame(frame, direction)
|
|
||||||
|
|
||||||
# When the LLM finishes responding, flush any remaining text in Deepgram's buffer
|
|
||||||
if isinstance(frame, (LLMFullResponseEndFrame, EndFrame)):
|
|
||||||
await self.flush_audio()
|
|
||||||
|
|
||||||
async def _connect(self):
|
|
||||||
"""Connect to Deepgram WebSocket and start receive task."""
|
|
||||||
await self._connect_websocket()
|
|
||||||
|
|
||||||
if self._websocket and not self._receive_task:
|
|
||||||
self._receive_task = self.create_task(self._receive_task_handler(self._report_error))
|
|
||||||
|
|
||||||
async def _disconnect(self):
|
|
||||||
"""Disconnect from Deepgram WebSocket and clean up tasks."""
|
|
||||||
if self._receive_task:
|
|
||||||
await self.cancel_task(self._receive_task)
|
|
||||||
self._receive_task = None
|
|
||||||
|
|
||||||
await self._disconnect_websocket()
|
|
||||||
|
|
||||||
async def _connect_websocket(self):
|
|
||||||
"""Connect to Deepgram WebSocket API with configured settings."""
|
|
||||||
try:
|
|
||||||
if self._websocket and self._websocket.state is State.OPEN:
|
|
||||||
return
|
|
||||||
|
|
||||||
logger.debug("Connecting to Deepgram WebSocket")
|
|
||||||
|
|
||||||
# Build WebSocket URL with query parameters
|
|
||||||
params = []
|
|
||||||
params.append(f"model={self._voice_id}")
|
|
||||||
params.append(f"encoding={self._settings['encoding']}")
|
|
||||||
params.append(f"sample_rate={self.sample_rate}")
|
|
||||||
|
|
||||||
url = f"{self._base_url}/v1/speak?{'&'.join(params)}"
|
|
||||||
|
|
||||||
headers = {"Authorization": f"Token {self._api_key}"}
|
|
||||||
|
|
||||||
self._websocket = await websocket_connect(url, additional_headers=headers)
|
|
||||||
|
|
||||||
await self._call_event_handler("on_connected")
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"{self} exception: {e}")
|
|
||||||
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
|
||||||
self._websocket = None
|
|
||||||
await self._call_event_handler("on_connection_error", f"{e}")
|
|
||||||
|
|
||||||
async def _disconnect_websocket(self):
|
|
||||||
"""Close WebSocket connection and reset state."""
|
|
||||||
try:
|
|
||||||
await self.stop_all_metrics()
|
|
||||||
|
|
||||||
if self._websocket:
|
|
||||||
logger.debug("Disconnecting from Deepgram WebSocket")
|
|
||||||
# Send Close message to gracefully close the connection
|
|
||||||
await self._websocket.send(json.dumps({"type": "Close"}))
|
|
||||||
await self._websocket.close()
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"{self} exception: {e}")
|
|
||||||
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
|
||||||
finally:
|
|
||||||
self._websocket = None
|
|
||||||
await self._call_event_handler("on_disconnected")
|
|
||||||
|
|
||||||
def _get_websocket(self):
|
|
||||||
"""Get active websocket connection or raise exception."""
|
|
||||||
if self._websocket:
|
|
||||||
return self._websocket
|
|
||||||
raise Exception("Websocket not connected")
|
|
||||||
|
|
||||||
async def _handle_interruption(self, frame: InterruptionFrame, direction: FrameDirection):
|
|
||||||
"""Handle interruption by sending Clear message to Deepgram.
|
|
||||||
|
|
||||||
The Clear message will clear Deepgram's internal text buffer and stop
|
|
||||||
sending audio, allowing for a new response to be generated.
|
|
||||||
"""
|
|
||||||
await super()._handle_interruption(frame, direction)
|
|
||||||
|
|
||||||
# Send Clear message to stop current audio generation
|
|
||||||
if self._websocket:
|
|
||||||
try:
|
|
||||||
clear_msg = {"type": "Clear"}
|
|
||||||
await self._websocket.send(json.dumps(clear_msg))
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"{self} error sending Clear message: {e}")
|
|
||||||
|
|
||||||
async def _receive_messages(self):
|
|
||||||
"""Receive and process messages from Deepgram WebSocket."""
|
|
||||||
async for message in self._get_websocket():
|
|
||||||
if isinstance(message, bytes):
|
|
||||||
# Binary message contains audio data
|
|
||||||
await self.stop_ttfb_metrics()
|
|
||||||
frame = TTSAudioRawFrame(message, self.sample_rate, 1)
|
|
||||||
await self.push_frame(frame)
|
|
||||||
elif isinstance(message, str):
|
|
||||||
# Text message contains metadata or control messages
|
|
||||||
try:
|
|
||||||
msg = json.loads(message)
|
|
||||||
msg_type = msg.get("type")
|
|
||||||
|
|
||||||
if msg_type == "Metadata":
|
|
||||||
logger.trace(f"Received metadata: {msg}")
|
|
||||||
elif msg_type == "Flushed":
|
|
||||||
logger.trace(f"Received Flushed: {msg}")
|
|
||||||
# Flushed indicates the end of audio generation for the current buffer
|
|
||||||
# This happens after flush_audio() is called
|
|
||||||
await self.push_frame(TTSStoppedFrame())
|
|
||||||
elif msg_type == "Cleared":
|
|
||||||
logger.trace(f"Received Cleared: {msg}")
|
|
||||||
# Buffer has been cleared after interruption
|
|
||||||
# TTSStoppedFrame will be sent by the interruption handler
|
|
||||||
elif msg_type == "Warning":
|
|
||||||
logger.warning(
|
|
||||||
f"{self} warning: {msg.get('description', 'Unknown warning')}"
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
logger.debug(f"Received unknown message type: {msg}")
|
|
||||||
except json.JSONDecodeError:
|
|
||||||
logger.error(f"Invalid JSON message: {message}")
|
|
||||||
|
|
||||||
async def flush_audio(self):
|
|
||||||
"""Flush any pending audio synthesis by sending Flush command.
|
|
||||||
|
|
||||||
This should be called when the LLM finishes a complete response to force
|
|
||||||
generation of audio from Deepgram's internal text buffer.
|
|
||||||
"""
|
|
||||||
if self._websocket:
|
|
||||||
try:
|
|
||||||
flush_msg = {"type": "Flush"}
|
|
||||||
await self._websocket.send(json.dumps(flush_msg))
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"{self} error sending Flush message: {e}")
|
|
||||||
|
|
||||||
@traced_tts
|
@traced_tts
|
||||||
async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]:
|
async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]:
|
||||||
"""Generate speech from text using Deepgram's WebSocket TTS API.
|
"""Generate speech from text using Deepgram's TTS API.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
text: The text to synthesize into speech.
|
text: The text to synthesize into speech.
|
||||||
@@ -270,27 +100,33 @@ class DeepgramTTSService(WebsocketTTSService):
|
|||||||
"""
|
"""
|
||||||
logger.debug(f"{self}: Generating TTS [{text}]")
|
logger.debug(f"{self}: Generating TTS [{text}]")
|
||||||
|
|
||||||
|
options = SpeakOptions(
|
||||||
|
model=self._voice_id,
|
||||||
|
encoding=self._settings["encoding"],
|
||||||
|
sample_rate=self.sample_rate,
|
||||||
|
container="none",
|
||||||
|
)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
# Reconnect if the websocket is closed
|
|
||||||
if not self._websocket or self._websocket.state is State.CLOSED:
|
|
||||||
await self._connect()
|
|
||||||
|
|
||||||
await self.start_ttfb_metrics()
|
await self.start_ttfb_metrics()
|
||||||
await self.start_tts_usage_metrics(text)
|
|
||||||
|
|
||||||
|
response = await self._deepgram_client.speak.asyncrest.v("1").stream_raw(
|
||||||
|
{"text": text}, options
|
||||||
|
)
|
||||||
|
|
||||||
|
await self.start_tts_usage_metrics(text)
|
||||||
yield TTSStartedFrame()
|
yield TTSStartedFrame()
|
||||||
|
|
||||||
# Send text message to Deepgram
|
async for data in response.aiter_bytes():
|
||||||
# Note: We don't send Flush here - that should only be sent when the
|
await self.stop_ttfb_metrics()
|
||||||
# LLM finishes a complete response via flush_audio()
|
if data:
|
||||||
speak_msg = {"type": "Speak", "text": text}
|
yield TTSAudioRawFrame(audio=data, sample_rate=self.sample_rate, num_channels=1)
|
||||||
await self._get_websocket().send(json.dumps(speak_msg))
|
|
||||||
|
|
||||||
# The actual audio frames will be handled in _receive_messages
|
yield TTSStoppedFrame()
|
||||||
yield None
|
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
|
|
||||||
|
|
||||||
class DeepgramHttpTTSService(TTSService):
|
class DeepgramHttpTTSService(TTSService):
|
||||||
@@ -341,6 +177,15 @@ class DeepgramHttpTTSService(TTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that Deepgram TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that Deepgram's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
@traced_tts
|
@traced_tts
|
||||||
async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]:
|
async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]:
|
||||||
"""Generate speech from text using Deepgram's TTS API.
|
"""Generate speech from text using Deepgram's TTS API.
|
||||||
@@ -400,4 +245,5 @@ class DeepgramHttpTTSService(TTSService):
|
|||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
|
logger.exception(f"{self} exception: {e}")
|
||||||
yield ErrorFrame(f"Error getting audio: {str(e)}")
|
yield ErrorFrame(f"Error getting audio: {str(e)}")
|
||||||
|
|||||||
@@ -351,7 +351,8 @@ class ElevenLabsSTTService(SegmentedSTTService):
|
|||||||
)
|
)
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
|
|
||||||
|
|
||||||
def audio_format_from_sample_rate(sample_rate: int) -> str:
|
def audio_format_from_sample_rate(sample_rate: int) -> str:
|
||||||
@@ -415,8 +416,6 @@ class ElevenLabsRealtimeSTTService(WebsocketSTTService):
|
|||||||
Only used when commit_strategy is VAD. None uses ElevenLabs default.
|
Only used when commit_strategy is VAD. None uses ElevenLabs default.
|
||||||
min_silence_duration_ms: Minimum silence duration for VAD (50-2000ms).
|
min_silence_duration_ms: Minimum silence duration for VAD (50-2000ms).
|
||||||
Only used when commit_strategy is VAD. None uses ElevenLabs default.
|
Only used when commit_strategy is VAD. None uses ElevenLabs default.
|
||||||
include_timestamps: Whether to include word-level timestamps in transcripts.
|
|
||||||
enable_logging: Whether to enable logging on ElevenLabs' side.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
language_code: Optional[str] = None
|
language_code: Optional[str] = None
|
||||||
@@ -425,8 +424,6 @@ class ElevenLabsRealtimeSTTService(WebsocketSTTService):
|
|||||||
vad_threshold: Optional[float] = None
|
vad_threshold: Optional[float] = None
|
||||||
min_speech_duration_ms: Optional[int] = None
|
min_speech_duration_ms: Optional[int] = None
|
||||||
min_silence_duration_ms: Optional[int] = None
|
min_silence_duration_ms: Optional[int] = None
|
||||||
include_timestamps: bool = False
|
|
||||||
enable_logging: bool = False
|
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
self,
|
self,
|
||||||
@@ -462,8 +459,6 @@ class ElevenLabsRealtimeSTTService(WebsocketSTTService):
|
|||||||
self._audio_format = "" # initialized in start()
|
self._audio_format = "" # initialized in start()
|
||||||
self._receive_task = None
|
self._receive_task = None
|
||||||
|
|
||||||
self._settings = {"language": params.language_code}
|
|
||||||
|
|
||||||
def can_generate_metrics(self) -> bool:
|
def can_generate_metrics(self) -> bool:
|
||||||
"""Check if the service can generate processing metrics.
|
"""Check if the service can generate processing metrics.
|
||||||
|
|
||||||
@@ -482,13 +477,7 @@ class ElevenLabsRealtimeSTTService(WebsocketSTTService):
|
|||||||
Changing language requires reconnecting to the WebSocket.
|
Changing language requires reconnecting to the WebSocket.
|
||||||
"""
|
"""
|
||||||
logger.info(f"Switching STT language to: [{language}]")
|
logger.info(f"Switching STT language to: [{language}]")
|
||||||
new_language = (
|
self._params.language_code = language.value if isinstance(language, Language) else language
|
||||||
language_to_elevenlabs_language(language)
|
|
||||||
if isinstance(language, Language)
|
|
||||||
else language
|
|
||||||
)
|
|
||||||
self._params.language_code = new_language
|
|
||||||
self._settings["language"] = new_language
|
|
||||||
# Reconnect with new settings
|
# Reconnect with new settings
|
||||||
await self._disconnect()
|
await self._disconnect()
|
||||||
await self._connect()
|
await self._connect()
|
||||||
@@ -597,6 +586,7 @@ class ElevenLabsRealtimeSTTService(WebsocketSTTService):
|
|||||||
}
|
}
|
||||||
await self._websocket.send(json.dumps(message))
|
await self._websocket.send(json.dumps(message))
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
|
logger.error(f"Error sending audio: {e}")
|
||||||
yield ErrorFrame(f"ElevenLabs Realtime STT error: {str(e)}")
|
yield ErrorFrame(f"ElevenLabs Realtime STT error: {str(e)}")
|
||||||
|
|
||||||
yield None
|
yield None
|
||||||
@@ -630,16 +620,10 @@ class ElevenLabsRealtimeSTTService(WebsocketSTTService):
|
|||||||
if self._params.language_code:
|
if self._params.language_code:
|
||||||
params.append(f"language_code={self._params.language_code}")
|
params.append(f"language_code={self._params.language_code}")
|
||||||
|
|
||||||
params.append(f"audio_format={self._audio_format}")
|
params.append(f"encoding={self._audio_format}")
|
||||||
|
params.append(f"sample_rate={self.sample_rate}")
|
||||||
params.append(f"commit_strategy={self._params.commit_strategy.value}")
|
params.append(f"commit_strategy={self._params.commit_strategy.value}")
|
||||||
|
|
||||||
# Add optional parameters
|
|
||||||
if self._params.include_timestamps:
|
|
||||||
params.append(f"include_timestamps={str(self._params.include_timestamps).lower()}")
|
|
||||||
|
|
||||||
if self._params.enable_logging:
|
|
||||||
params.append(f"enable_logging={str(self._params.enable_logging).lower()}")
|
|
||||||
|
|
||||||
# Add VAD parameters if using VAD commit strategy and values are specified
|
# Add VAD parameters if using VAD commit strategy and values are specified
|
||||||
if self._params.commit_strategy == CommitStrategy.VAD:
|
if self._params.commit_strategy == CommitStrategy.VAD:
|
||||||
if self._params.vad_silence_threshold_secs is not None:
|
if self._params.vad_silence_threshold_secs is not None:
|
||||||
@@ -661,9 +645,8 @@ class ElevenLabsRealtimeSTTService(WebsocketSTTService):
|
|||||||
await self._call_event_handler("on_connected")
|
await self._call_event_handler("on_connected")
|
||||||
logger.debug("Connected to ElevenLabs Realtime STT")
|
logger.debug("Connected to ElevenLabs Realtime STT")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(
|
logger.error(f"{self}: unable to connect to ElevenLabs Realtime STT: {e}")
|
||||||
error_msg=f"Unable to connect to ElevenLabs Realtime STT: {e}", exception=e
|
await self.push_error(ErrorFrame(f"Connection error: {str(e)}"))
|
||||||
)
|
|
||||||
|
|
||||||
async def _disconnect_websocket(self):
|
async def _disconnect_websocket(self):
|
||||||
"""Disconnect from ElevenLabs Realtime STT WebSocket."""
|
"""Disconnect from ElevenLabs Realtime STT WebSocket."""
|
||||||
@@ -672,7 +655,7 @@ class ElevenLabsRealtimeSTTService(WebsocketSTTService):
|
|||||||
logger.debug("Disconnecting from ElevenLabs Realtime STT")
|
logger.debug("Disconnecting from ElevenLabs Realtime STT")
|
||||||
await self._websocket.close()
|
await self._websocket.close()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error closing websocket: {e}", exception=e)
|
logger.error(f"{self} error closing websocket: {e}")
|
||||||
finally:
|
finally:
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
await self._call_event_handler("on_disconnected")
|
await self._call_event_handler("on_disconnected")
|
||||||
@@ -729,20 +712,15 @@ class ElevenLabsRealtimeSTTService(WebsocketSTTService):
|
|||||||
elif message_type == "committed_transcript_with_timestamps":
|
elif message_type == "committed_transcript_with_timestamps":
|
||||||
await self._on_committed_transcript_with_timestamps(data)
|
await self._on_committed_transcript_with_timestamps(data)
|
||||||
|
|
||||||
elif message_type == "error":
|
elif message_type == "input_error":
|
||||||
error_msg = data.get("error", "Unknown error")
|
error_msg = data.get("error", "Unknown input error")
|
||||||
logger.error(f"ElevenLabs error: {error_msg}")
|
logger.error(f"ElevenLabs input error: {error_msg}")
|
||||||
await self.push_error(error_msg=f"Error: {error_msg}")
|
await self.push_error(ErrorFrame(f"Input error: {error_msg}"))
|
||||||
|
|
||||||
elif message_type == "auth_error":
|
elif message_type in ["auth_error", "quota_exceeded", "transcriber_error", "error"]:
|
||||||
error_msg = data.get("error", "Authentication error")
|
error_msg = data.get("error", data.get("message", "Unknown error"))
|
||||||
logger.error(f"ElevenLabs auth error: {error_msg}")
|
logger.error(f"ElevenLabs error ({message_type}): {error_msg}")
|
||||||
await self.push_error(error_msg=f"Auth error: {error_msg}")
|
await self.push_error(ErrorFrame(f"{message_type}: {error_msg}"))
|
||||||
|
|
||||||
elif message_type == "quota_exceeded_error":
|
|
||||||
error_msg = data.get("error", "Quota exceeded")
|
|
||||||
logger.error(f"ElevenLabs quota exceeded: {error_msg}")
|
|
||||||
await self.push_error(error_msg=f"Quota exceeded: {error_msg}")
|
|
||||||
|
|
||||||
else:
|
else:
|
||||||
logger.debug(f"Unknown message type: {message_type}")
|
logger.debug(f"Unknown message type: {message_type}")
|
||||||
@@ -787,11 +765,6 @@ class ElevenLabsRealtimeSTTService(WebsocketSTTService):
|
|||||||
Args:
|
Args:
|
||||||
data: Committed transcript data.
|
data: Committed transcript data.
|
||||||
"""
|
"""
|
||||||
# If timestamps are enabled, skip this message and wait for the
|
|
||||||
# committed_transcript_with_timestamps message which contains all the data
|
|
||||||
if self._params.include_timestamps:
|
|
||||||
return
|
|
||||||
|
|
||||||
text = data.get("text", "").strip()
|
text = data.get("text", "").strip()
|
||||||
if not text:
|
if not text:
|
||||||
return
|
return
|
||||||
@@ -819,18 +792,6 @@ class ElevenLabsRealtimeSTTService(WebsocketSTTService):
|
|||||||
async def _on_committed_transcript_with_timestamps(self, data: dict):
|
async def _on_committed_transcript_with_timestamps(self, data: dict):
|
||||||
"""Handle committed transcript with word-level timestamps.
|
"""Handle committed transcript with word-level timestamps.
|
||||||
|
|
||||||
This message is sent when include_timestamps=true. The result data includes:
|
|
||||||
- text: The transcribed text
|
|
||||||
- language_code: Detected language (if available)
|
|
||||||
- words: Array of word objects with timing information:
|
|
||||||
- text: The word text
|
|
||||||
- start: Start time in seconds
|
|
||||||
- end: End time in seconds
|
|
||||||
- type: "word" or "spacing"
|
|
||||||
- speaker_id: Speaker identifier (if available)
|
|
||||||
- logprob: Log probability score (if available)
|
|
||||||
- characters: Array of character strings (if available)
|
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
data: Committed transcript data with timestamps.
|
data: Committed transcript data with timestamps.
|
||||||
"""
|
"""
|
||||||
@@ -838,24 +799,9 @@ class ElevenLabsRealtimeSTTService(WebsocketSTTService):
|
|||||||
if not text:
|
if not text:
|
||||||
return
|
return
|
||||||
|
|
||||||
await self.stop_ttfb_metrics()
|
|
||||||
await self.stop_processing_metrics()
|
|
||||||
|
|
||||||
# Get language if provided
|
|
||||||
language = data.get("language_code")
|
|
||||||
|
|
||||||
logger.debug(f"Committed transcript with timestamps: [{text}]")
|
logger.debug(f"Committed transcript with timestamps: [{text}]")
|
||||||
|
logger.trace(f"Timestamps: {data.get('words', [])}")
|
||||||
|
|
||||||
await self._handle_transcription(text, True, language)
|
# This is sent after the committed_transcript, so we don't need to
|
||||||
|
# push another TranscriptionFrame, but we could use the timestamps
|
||||||
# This message is sent after committed_transcript when include_timestamps=true.
|
# for additional processing if needed in the future
|
||||||
# It contains the full transcript data including text and word-level timestamps.
|
|
||||||
await self.push_frame(
|
|
||||||
TranscriptionFrame(
|
|
||||||
text,
|
|
||||||
self._user_id,
|
|
||||||
time_now_iso8601(),
|
|
||||||
language,
|
|
||||||
result=data,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|||||||
@@ -424,7 +424,8 @@ class ElevenLabsTTSService(AudioContextWordTTSService):
|
|||||||
json.dumps({"context_id": self._context_id, "close_context": True})
|
json.dumps({"context_id": self._context_id, "close_context": True})
|
||||||
)
|
)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
self._context_id = None
|
self._context_id = None
|
||||||
self._started = False
|
self._started = False
|
||||||
|
|
||||||
@@ -535,8 +536,9 @@ class ElevenLabsTTSService(AudioContextWordTTSService):
|
|||||||
|
|
||||||
await self._call_event_handler("on_connected")
|
await self._call_event_handler("on_connected")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
|
logger.error(f"{self} exception: {e}")
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
await self._call_event_handler("on_connection_error", f"{e}")
|
await self._call_event_handler("on_connection_error", f"{e}")
|
||||||
|
|
||||||
async def _disconnect_websocket(self):
|
async def _disconnect_websocket(self):
|
||||||
@@ -551,7 +553,8 @@ class ElevenLabsTTSService(AudioContextWordTTSService):
|
|||||||
await self._websocket.close()
|
await self._websocket.close()
|
||||||
logger.debug("Disconnected from ElevenLabs")
|
logger.debug("Disconnected from ElevenLabs")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
finally:
|
finally:
|
||||||
self._started = False
|
self._started = False
|
||||||
self._context_id = None
|
self._context_id = None
|
||||||
@@ -581,7 +584,8 @@ class ElevenLabsTTSService(AudioContextWordTTSService):
|
|||||||
json.dumps({"context_id": self._context_id, "close_context": True})
|
json.dumps({"context_id": self._context_id, "close_context": True})
|
||||||
)
|
)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
self._context_id = None
|
self._context_id = None
|
||||||
self._started = False
|
self._started = False
|
||||||
self._partial_word = ""
|
self._partial_word = ""
|
||||||
@@ -736,13 +740,15 @@ class ElevenLabsTTSService(AudioContextWordTTSService):
|
|||||||
else:
|
else:
|
||||||
await self._send_text(text)
|
await self._send_text(text)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
|
logger.error(f"{self} exception: {e}")
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
self._started = False
|
self._started = False
|
||||||
return
|
return
|
||||||
yield None
|
yield None
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
|
|
||||||
|
|
||||||
class ElevenLabsHttpTTSService(WordTTSService):
|
class ElevenLabsHttpTTSService(WordTTSService):
|
||||||
@@ -1037,6 +1043,7 @@ class ElevenLabsHttpTTSService(WordTTSService):
|
|||||||
) as response:
|
) as response:
|
||||||
if response.status != 200:
|
if response.status != 200:
|
||||||
error_text = await response.text()
|
error_text = await response.text()
|
||||||
|
logger.error(f"{self} error: {error_text}")
|
||||||
yield ErrorFrame(error=f"ElevenLabs API error: {error_text}")
|
yield ErrorFrame(error=f"ElevenLabs API error: {error_text}")
|
||||||
return
|
return
|
||||||
|
|
||||||
@@ -1084,7 +1091,8 @@ class ElevenLabsHttpTTSService(WordTTSService):
|
|||||||
logger.warning(f"Failed to parse JSON from stream: {e}")
|
logger.warning(f"Failed to parse JSON from stream: {e}")
|
||||||
continue
|
continue
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
continue
|
continue
|
||||||
|
|
||||||
# After processing all chunks, emit any remaining partial word
|
# After processing all chunks, emit any remaining partial word
|
||||||
@@ -1108,7 +1116,8 @@ class ElevenLabsHttpTTSService(WordTTSService):
|
|||||||
self._previous_text = text
|
self._previous_text = text
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
finally:
|
finally:
|
||||||
await self.stop_ttfb_metrics()
|
await self.stop_ttfb_metrics()
|
||||||
# Let the parent class handle TTSStoppedFrame
|
# Let the parent class handle TTSStoppedFrame
|
||||||
|
|||||||
@@ -110,6 +110,7 @@ class FalImageGenService(ImageGenService):
|
|||||||
image_url = response["images"][0]["url"] if response else None
|
image_url = response["images"][0]["url"] if response else None
|
||||||
|
|
||||||
if not image_url:
|
if not image_url:
|
||||||
|
logger.error(f"{self} error: image generation failed")
|
||||||
yield ErrorFrame("Image generation failed")
|
yield ErrorFrame("Image generation failed")
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|||||||
@@ -290,4 +290,5 @@ class FalSTTService(SegmentedSTTService):
|
|||||||
)
|
)
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
|
|||||||
@@ -76,7 +76,7 @@ class FishAudioTTSService(InterruptibleTTSService):
|
|||||||
api_key: str,
|
api_key: str,
|
||||||
reference_id: Optional[str] = None, # This is the voice ID
|
reference_id: Optional[str] = None, # This is the voice ID
|
||||||
model: Optional[str] = None, # Deprecated
|
model: Optional[str] = None, # Deprecated
|
||||||
model_id: str = "s1",
|
model_id: str = "speech-1.5",
|
||||||
output_format: FishAudioOutputFormat = "pcm",
|
output_format: FishAudioOutputFormat = "pcm",
|
||||||
sample_rate: Optional[int] = None,
|
sample_rate: Optional[int] = None,
|
||||||
params: Optional[InputParams] = None,
|
params: Optional[InputParams] = None,
|
||||||
@@ -93,7 +93,7 @@ class FishAudioTTSService(InterruptibleTTSService):
|
|||||||
The `model` parameter is deprecated and will be removed in version 0.1.0.
|
The `model` parameter is deprecated and will be removed in version 0.1.0.
|
||||||
Use `reference_id` instead to specify the voice model.
|
Use `reference_id` instead to specify the voice model.
|
||||||
|
|
||||||
model_id: Specify which Fish Audio TTS model to use (e.g. "s1")
|
model_id: Specify which Fish Audio TTS model to use (e.g. "speech-1.5")
|
||||||
output_format: Audio output format. Defaults to "pcm".
|
output_format: Audio output format. Defaults to "pcm".
|
||||||
sample_rate: Audio sample rate. If None, uses default.
|
sample_rate: Audio sample rate. If None, uses default.
|
||||||
params: Additional input parameters for voice customization.
|
params: Additional input parameters for voice customization.
|
||||||
@@ -159,6 +159,15 @@ class FishAudioTTSService(InterruptibleTTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that Fish Audio TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that Fish Audio's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
async def set_model(self, model: str):
|
async def set_model(self, model: str):
|
||||||
"""Set the TTS model and reconnect.
|
"""Set the TTS model and reconnect.
|
||||||
|
|
||||||
@@ -228,7 +237,8 @@ class FishAudioTTSService(InterruptibleTTSService):
|
|||||||
|
|
||||||
await self._call_event_handler("on_connected")
|
await self._call_event_handler("on_connected")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
await self._call_event_handler("on_connection_error", f"{e}")
|
await self._call_event_handler("on_connection_error", f"{e}")
|
||||||
|
|
||||||
@@ -242,7 +252,8 @@ class FishAudioTTSService(InterruptibleTTSService):
|
|||||||
await self._websocket.send(ormsgpack.packb(stop_message))
|
await self._websocket.send(ormsgpack.packb(stop_message))
|
||||||
await self._websocket.close()
|
await self._websocket.close()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
finally:
|
finally:
|
||||||
self._request_id = None
|
self._request_id = None
|
||||||
self._started = False
|
self._started = False
|
||||||
@@ -284,7 +295,8 @@ class FishAudioTTSService(InterruptibleTTSService):
|
|||||||
continue
|
continue
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
|
|
||||||
@traced_tts
|
@traced_tts
|
||||||
async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]:
|
async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]:
|
||||||
@@ -320,7 +332,8 @@ class FishAudioTTSService(InterruptibleTTSService):
|
|||||||
flush_message = {"event": "flush"}
|
flush_message = {"event": "flush"}
|
||||||
await self._get_websocket().send(ormsgpack.packb(flush_message))
|
await self._get_websocket().send(ormsgpack.packb(flush_message))
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
await self._disconnect()
|
await self._disconnect()
|
||||||
await self._connect()
|
await self._connect()
|
||||||
@@ -328,4 +341,5 @@ class FishAudioTTSService(InterruptibleTTSService):
|
|||||||
yield None
|
yield None
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
|
|||||||
@@ -468,7 +468,8 @@ class GladiaSTTService(STTService):
|
|||||||
break
|
break
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
self._connection_active = False
|
self._connection_active = False
|
||||||
|
|
||||||
if not self._should_reconnect:
|
if not self._should_reconnect:
|
||||||
@@ -558,7 +559,8 @@ class GladiaSTTService(STTService):
|
|||||||
except websockets.exceptions.ConnectionClosed:
|
except websockets.exceptions.ConnectionClosed:
|
||||||
logger.debug("Connection closed during keepalive")
|
logger.debug("Connection closed during keepalive")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
|
|
||||||
async def _receive_task_handler(self):
|
async def _receive_task_handler(self):
|
||||||
try:
|
try:
|
||||||
@@ -621,7 +623,8 @@ class GladiaSTTService(STTService):
|
|||||||
# Expected when closing the connection
|
# Expected when closing the connection
|
||||||
pass
|
pass
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
|
|
||||||
async def _maybe_reconnect(self) -> bool:
|
async def _maybe_reconnect(self) -> bool:
|
||||||
"""Handle exponential backoff reconnection logic."""
|
"""Handle exponential backoff reconnection logic."""
|
||||||
@@ -629,9 +632,7 @@ class GladiaSTTService(STTService):
|
|||||||
return False
|
return False
|
||||||
self._reconnection_attempts += 1
|
self._reconnection_attempts += 1
|
||||||
if self._reconnection_attempts > self._max_reconnection_attempts:
|
if self._reconnection_attempts > self._max_reconnection_attempts:
|
||||||
await self.push_error(
|
logger.error(f"Max reconnection attempts ({self._max_reconnection_attempts}) reached")
|
||||||
error_msg=f"Max reconnection attempts ({self._max_reconnection_attempts}) reached",
|
|
||||||
)
|
|
||||||
self._should_reconnect = False
|
self._should_reconnect = False
|
||||||
return False
|
return False
|
||||||
delay = self._reconnection_delay * (2 ** (self._reconnection_attempts - 1))
|
delay = self._reconnection_delay * (2 ** (self._reconnection_attempts - 1))
|
||||||
|
|||||||
@@ -1175,7 +1175,7 @@ class GeminiLiveLLMService(LLMService):
|
|||||||
self._connection_task = self.create_task(self._connection_task_handler(config=config))
|
self._connection_task = self.create_task(self._connection_task_handler(config=config))
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Initialization error: {e}", exception=e)
|
await self.push_error(ErrorFrame(error=f"{self} Initialization error: {e}"))
|
||||||
|
|
||||||
async def _connection_task_handler(self, config: LiveConnectConfig):
|
async def _connection_task_handler(self, config: LiveConnectConfig):
|
||||||
async with self._client.aio.live.connect(model=self._model_name, config=config) as session:
|
async with self._client.aio.live.connect(model=self._model_name, config=config) as session:
|
||||||
@@ -1252,11 +1252,11 @@ class GeminiLiveLLMService(LLMService):
|
|||||||
)
|
)
|
||||||
|
|
||||||
if self._consecutive_failures >= MAX_CONSECUTIVE_FAILURES:
|
if self._consecutive_failures >= MAX_CONSECUTIVE_FAILURES:
|
||||||
error_msg = (
|
logger.error(
|
||||||
f"Max consecutive failures ({MAX_CONSECUTIVE_FAILURES}) reached, "
|
f"Max consecutive failures ({MAX_CONSECUTIVE_FAILURES}) reached, "
|
||||||
"treating as fatal error"
|
"treating as fatal error"
|
||||||
)
|
)
|
||||||
await self.push_error(error_msg=error_msg, exception=error)
|
await self.push_error(ErrorFrame(error=f"{self} Error in receive loop: {error}"))
|
||||||
return False
|
return False
|
||||||
else:
|
else:
|
||||||
logger.info(
|
logger.info(
|
||||||
@@ -1284,7 +1284,7 @@ class GeminiLiveLLMService(LLMService):
|
|||||||
self._completed_tool_calls = set()
|
self._completed_tool_calls = set()
|
||||||
self._disconnecting = False
|
self._disconnecting = False
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error disconnecting: {e}", exception=e)
|
logger.error(f"{self} error disconnecting: {e}")
|
||||||
|
|
||||||
async def _send_user_audio(self, frame):
|
async def _send_user_audio(self, frame):
|
||||||
"""Send user audio frame to Gemini Live API."""
|
"""Send user audio frame to Gemini Live API."""
|
||||||
@@ -1448,11 +1448,13 @@ class GeminiLiveLLMService(LLMService):
|
|||||||
# Update bot responding state and send service start frame
|
# Update bot responding state and send service start frame
|
||||||
# (AUDIO modality case)
|
# (AUDIO modality case)
|
||||||
await self._set_bot_is_responding(True)
|
await self._set_bot_is_responding(True)
|
||||||
await self.push_frame(LLMFullResponseStartFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseStartFrame())
|
||||||
|
|
||||||
self._bot_text_buffer += text
|
self._bot_text_buffer += text
|
||||||
self._search_result_buffer += text # Also accumulate for grounding
|
self._search_result_buffer += text # Also accumulate for grounding
|
||||||
frame = LLMTextFrame(text=text, skip_tts=self._get_skip_tts())
|
frame = LLMTextFrame(text=text)
|
||||||
|
# Gemini Live text already includes any necessary inter-chunk spaces
|
||||||
|
frame.includes_inter_frame_spaces = True
|
||||||
await self.push_frame(frame)
|
await self.push_frame(frame)
|
||||||
|
|
||||||
# Check for grounding metadata in server content
|
# Check for grounding metadata in server content
|
||||||
@@ -1491,7 +1493,7 @@ class GeminiLiveLLMService(LLMService):
|
|||||||
if not self._bot_is_responding:
|
if not self._bot_is_responding:
|
||||||
await self._set_bot_is_responding(True)
|
await self._set_bot_is_responding(True)
|
||||||
await self.push_frame(TTSStartedFrame())
|
await self.push_frame(TTSStartedFrame())
|
||||||
await self.push_frame(LLMFullResponseStartFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseStartFrame())
|
||||||
|
|
||||||
self._bot_audio_buffer.extend(audio)
|
self._bot_audio_buffer.extend(audio)
|
||||||
frame = TTSAudioRawFrame(
|
frame = TTSAudioRawFrame(
|
||||||
@@ -1552,10 +1554,10 @@ class GeminiLiveLLMService(LLMService):
|
|||||||
if not text:
|
if not text:
|
||||||
# AUDIO modality case
|
# AUDIO modality case
|
||||||
await self.push_frame(TTSStoppedFrame())
|
await self.push_frame(TTSStoppedFrame())
|
||||||
await self.push_frame(LLMFullResponseEndFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseEndFrame())
|
||||||
else:
|
else:
|
||||||
# TEXT modality case
|
# TEXT modality case
|
||||||
await self.push_frame(LLMFullResponseEndFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseEndFrame())
|
||||||
|
|
||||||
@traced_stt
|
@traced_stt
|
||||||
async def _handle_user_transcription(
|
async def _handle_user_transcription(
|
||||||
@@ -1643,7 +1645,7 @@ class GeminiLiveLLMService(LLMService):
|
|||||||
if not self._bot_is_responding:
|
if not self._bot_is_responding:
|
||||||
await self._set_bot_is_responding(True)
|
await self._set_bot_is_responding(True)
|
||||||
await self.push_frame(TTSStartedFrame())
|
await self.push_frame(TTSStartedFrame())
|
||||||
await self.push_frame(LLMFullResponseStartFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseStartFrame())
|
||||||
|
|
||||||
frame = TTSTextFrame(text=text, aggregated_by=AggregationType.SENTENCE)
|
frame = TTSTextFrame(text=text, aggregated_by=AggregationType.SENTENCE)
|
||||||
# Gemini Live text already includes any necessary inter-chunk spaces
|
# Gemini Live text already includes any necessary inter-chunk spaces
|
||||||
@@ -1723,8 +1725,6 @@ class GeminiLiveLLMService(LLMService):
|
|||||||
prompt_tokens=prompt_tokens,
|
prompt_tokens=prompt_tokens,
|
||||||
completion_tokens=completion_tokens,
|
completion_tokens=completion_tokens,
|
||||||
total_tokens=total_tokens,
|
total_tokens=total_tokens,
|
||||||
cache_read_input_tokens=usage.cached_content_token_count,
|
|
||||||
reasoning_tokens=usage.thoughts_token_count,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
await self.start_llm_usage_metrics(tokens)
|
await self.start_llm_usage_metrics(tokens)
|
||||||
@@ -1745,7 +1745,7 @@ class GeminiLiveLLMService(LLMService):
|
|||||||
# state management, and that exponential backoff for retries can have
|
# state management, and that exponential backoff for retries can have
|
||||||
# cost/stability implications for a service cluster, let's just treat a
|
# cost/stability implications for a service cluster, let's just treat a
|
||||||
# send-side error as fatal.
|
# send-side error as fatal.
|
||||||
await self.push_error(error_msg=f"Send error: {error}")
|
await self.push_error(ErrorFrame(error=f"{self} Send error: {error}", fatal=True))
|
||||||
|
|
||||||
def create_context_aggregator(
|
def create_context_aggregator(
|
||||||
self,
|
self,
|
||||||
|
|||||||
@@ -110,6 +110,7 @@ class GoogleImageGenService(ImageGenService):
|
|||||||
await self.stop_ttfb_metrics()
|
await self.stop_ttfb_metrics()
|
||||||
|
|
||||||
if not response or not response.generated_images:
|
if not response or not response.generated_images:
|
||||||
|
logger.error(f"{self} error: image generation failed")
|
||||||
yield ErrorFrame("Image generation failed")
|
yield ErrorFrame("Image generation failed")
|
||||||
return
|
return
|
||||||
|
|
||||||
@@ -127,4 +128,5 @@ class GoogleImageGenService(ImageGenService):
|
|||||||
yield frame
|
yield frame
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
|
logger.error(f"{self} error generating image: {e}")
|
||||||
yield ErrorFrame(f"Image generation error: {str(e)}")
|
yield ErrorFrame(f"Image generation error: {str(e)}")
|
||||||
|
|||||||
@@ -793,7 +793,7 @@ class GoogleLLMService(LLMService):
|
|||||||
return
|
return
|
||||||
generation_params.setdefault("thinking_config", {})["thinking_budget"] = 0
|
generation_params.setdefault("thinking_config", {})["thinking_budget"] = 0
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Failed to unset thinking budget: {e}")
|
logger.exception(f"Failed to unset thinking budget: {e}")
|
||||||
|
|
||||||
async def _stream_content(
|
async def _stream_content(
|
||||||
self, params_from_context: GeminiLLMInvocationParams
|
self, params_from_context: GeminiLLMInvocationParams
|
||||||
@@ -876,7 +876,7 @@ class GoogleLLMService(LLMService):
|
|||||||
|
|
||||||
@traced_llm
|
@traced_llm
|
||||||
async def _process_context(self, context: OpenAILLMContext | LLMContext):
|
async def _process_context(self, context: OpenAILLMContext | LLMContext):
|
||||||
await self.push_frame(LLMFullResponseStartFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseStartFrame())
|
||||||
|
|
||||||
prompt_tokens = 0
|
prompt_tokens = 0
|
||||||
completion_tokens = 0
|
completion_tokens = 0
|
||||||
@@ -920,9 +920,9 @@ class GoogleLLMService(LLMService):
|
|||||||
for part in candidate.content.parts:
|
for part in candidate.content.parts:
|
||||||
if not part.thought and part.text:
|
if not part.thought and part.text:
|
||||||
search_result += part.text
|
search_result += part.text
|
||||||
await self.push_frame(
|
frame = LLMTextFrame(part.text)
|
||||||
LLMTextFrame(part.text, skip_tts=self._get_skip_tts())
|
frame.includes_inter_frame_spaces = True
|
||||||
)
|
await self.push_frame(frame)
|
||||||
elif part.function_call:
|
elif part.function_call:
|
||||||
function_call = part.function_call
|
function_call = part.function_call
|
||||||
id = function_call.id or str(uuid.uuid4())
|
id = function_call.id or str(uuid.uuid4())
|
||||||
@@ -985,7 +985,7 @@ class GoogleLLMService(LLMService):
|
|||||||
except DeadlineExceeded:
|
except DeadlineExceeded:
|
||||||
await self._call_event_handler("on_completion_timeout")
|
await self._call_event_handler("on_completion_timeout")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.exception(f"{self} exception: {e}")
|
||||||
finally:
|
finally:
|
||||||
if grounding_metadata and isinstance(grounding_metadata, dict):
|
if grounding_metadata and isinstance(grounding_metadata, dict):
|
||||||
llm_search_frame = LLMSearchResponseFrame(
|
llm_search_frame = LLMSearchResponseFrame(
|
||||||
@@ -1004,7 +1004,7 @@ class GoogleLLMService(LLMService):
|
|||||||
reasoning_tokens=reasoning_tokens,
|
reasoning_tokens=reasoning_tokens,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
await self.push_frame(LLMFullResponseEndFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseEndFrame())
|
||||||
|
|
||||||
async def process_frame(self, frame: Frame, direction: FrameDirection):
|
async def process_frame(self, frame: Frame, direction: FrameDirection):
|
||||||
"""Process incoming frames and handle different frame types.
|
"""Process incoming frames and handle different frame types.
|
||||||
|
|||||||
@@ -136,9 +136,7 @@ class GoogleLLMOpenAIBetaService(OpenAILLMService):
|
|||||||
# Keep iterating through the response to collect all the argument fragments
|
# Keep iterating through the response to collect all the argument fragments
|
||||||
arguments += tool_call.function.arguments
|
arguments += tool_call.function.arguments
|
||||||
elif chunk.choices[0].delta.content:
|
elif chunk.choices[0].delta.content:
|
||||||
await self.push_frame(
|
await self.push_frame(LLMTextFrame(chunk.choices[0].delta.content))
|
||||||
LLMTextFrame(chunk.choices[0].delta.content, skip_tts=self._get_skip_tts())
|
|
||||||
)
|
|
||||||
|
|
||||||
# if we got a function name and arguments, check to see if it's a function with
|
# if we got a function name and arguments, check to see if it's a function with
|
||||||
# a registered handler. If so, run the registered callback, save the result to
|
# a registered handler. If so, run the registered callback, save the result to
|
||||||
|
|||||||
@@ -774,7 +774,8 @@ class GoogleSTTService(STTService):
|
|||||||
yield cloud_speech.StreamingRecognizeRequest(audio=audio_data)
|
yield cloud_speech.StreamingRecognizeRequest(audio=audio_data)
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
raise
|
raise
|
||||||
|
|
||||||
async def _stream_audio(self):
|
async def _stream_audio(self):
|
||||||
@@ -805,13 +806,15 @@ class GoogleSTTService(STTService):
|
|||||||
break
|
break
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
|
|
||||||
await asyncio.sleep(1) # Brief delay before reconnecting
|
await asyncio.sleep(1) # Brief delay before reconnecting
|
||||||
self._stream_start_time = int(time.time() * 1000)
|
self._stream_start_time = int(time.time() * 1000)
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
|
|
||||||
async def run_stt(self, audio: bytes) -> AsyncGenerator[Frame, None]:
|
async def run_stt(self, audio: bytes) -> AsyncGenerator[Frame, None]:
|
||||||
"""Process an audio chunk for STT transcription.
|
"""Process an audio chunk for STT transcription.
|
||||||
@@ -899,7 +902,8 @@ class GoogleSTTService(STTService):
|
|||||||
)
|
)
|
||||||
raise
|
raise
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
# Re-raise the exception to let it propagate (e.g. in the case of a
|
# Re-raise the exception to let it propagate (e.g. in the case of a
|
||||||
# timeout, propagate to _stream_audio to reconnect)
|
# timeout, propagate to _stream_audio to reconnect)
|
||||||
raise
|
raise
|
||||||
|
|||||||
@@ -596,6 +596,15 @@ class GoogleHttpTTSService(TTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that Google TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that Google's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
def language_to_service_language(self, language: Language) -> Optional[str]:
|
def language_to_service_language(self, language: Language) -> Optional[str]:
|
||||||
"""Convert a Language enum to Google TTS language format.
|
"""Convert a Language enum to Google TTS language format.
|
||||||
|
|
||||||
@@ -737,6 +746,7 @@ class GoogleHttpTTSService(TTSService):
|
|||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
|
logger.error(f"{self} exception: {e}")
|
||||||
error_message = f"TTS generation error: {str(e)}"
|
error_message = f"TTS generation error: {str(e)}"
|
||||||
yield ErrorFrame(error=error_message)
|
yield ErrorFrame(error=error_message)
|
||||||
|
|
||||||
@@ -793,6 +803,15 @@ class GoogleBaseTTSService(TTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that Google and Gemini TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that Google's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
def language_to_service_language(self, language: Language) -> Optional[str]:
|
def language_to_service_language(self, language: Language) -> Optional[str]:
|
||||||
"""Convert a Language enum to Google TTS language format.
|
"""Convert a Language enum to Google TTS language format.
|
||||||
|
|
||||||
@@ -995,7 +1014,9 @@ class GoogleTTSService(GoogleBaseTTSService):
|
|||||||
yield frame
|
yield frame
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"TTS generation error: {str(e)}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
error_message = f"TTS generation error: {str(e)}"
|
||||||
|
yield ErrorFrame(error=error_message)
|
||||||
|
|
||||||
|
|
||||||
class GeminiTTSService(GoogleBaseTTSService):
|
class GeminiTTSService(GoogleBaseTTSService):
|
||||||
@@ -1245,5 +1266,6 @@ class GeminiTTSService(GoogleBaseTTSService):
|
|||||||
yield frame
|
yield frame
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
|
logger.error(f"{self} exception: {e}")
|
||||||
error_message = f"Gemini TTS generation error: {str(e)}"
|
error_message = f"Gemini TTS generation error: {str(e)}"
|
||||||
yield ErrorFrame(error=error_message)
|
yield ErrorFrame(error=error_message)
|
||||||
|
|||||||
@@ -123,8 +123,6 @@ class GrokLLMService(OpenAILLMService):
|
|||||||
self._prompt_tokens = 0
|
self._prompt_tokens = 0
|
||||||
self._completion_tokens = 0
|
self._completion_tokens = 0
|
||||||
self._total_tokens = 0
|
self._total_tokens = 0
|
||||||
self._cache_read_input_tokens = None
|
|
||||||
self._reasoning_tokens = None
|
|
||||||
self._has_reported_prompt_tokens = False
|
self._has_reported_prompt_tokens = False
|
||||||
self._is_processing = True
|
self._is_processing = True
|
||||||
|
|
||||||
@@ -139,8 +137,6 @@ class GrokLLMService(OpenAILLMService):
|
|||||||
prompt_tokens=self._prompt_tokens,
|
prompt_tokens=self._prompt_tokens,
|
||||||
completion_tokens=self._completion_tokens,
|
completion_tokens=self._completion_tokens,
|
||||||
total_tokens=self._total_tokens,
|
total_tokens=self._total_tokens,
|
||||||
cache_read_input_tokens=self._cache_read_input_tokens,
|
|
||||||
reasoning_tokens=self._reasoning_tokens,
|
|
||||||
)
|
)
|
||||||
await super().start_llm_usage_metrics(tokens)
|
await super().start_llm_usage_metrics(tokens)
|
||||||
|
|
||||||
@@ -153,7 +149,7 @@ class GrokLLMService(OpenAILLMService):
|
|||||||
|
|
||||||
Args:
|
Args:
|
||||||
tokens: The token usage metrics for the current chunk of processing,
|
tokens: The token usage metrics for the current chunk of processing,
|
||||||
containing prompt_tokens, completion_tokens, and optional cached/reasoning tokens.
|
containing prompt_tokens and completion_tokens counts.
|
||||||
"""
|
"""
|
||||||
# Only accumulate metrics during active processing
|
# Only accumulate metrics during active processing
|
||||||
if not self._is_processing:
|
if not self._is_processing:
|
||||||
@@ -168,13 +164,6 @@ class GrokLLMService(OpenAILLMService):
|
|||||||
if tokens.completion_tokens > self._completion_tokens:
|
if tokens.completion_tokens > self._completion_tokens:
|
||||||
self._completion_tokens = tokens.completion_tokens
|
self._completion_tokens = tokens.completion_tokens
|
||||||
|
|
||||||
# Capture cached & reasoning tokens (these typically only appear once per request)
|
|
||||||
if tokens.cache_read_input_tokens is not None:
|
|
||||||
self._cache_read_input_tokens = tokens.cache_read_input_tokens
|
|
||||||
|
|
||||||
if tokens.reasoning_tokens is not None:
|
|
||||||
self._reasoning_tokens = tokens.reasoning_tokens
|
|
||||||
|
|
||||||
def create_context_aggregator(
|
def create_context_aggregator(
|
||||||
self,
|
self,
|
||||||
context: OpenAILLMContext,
|
context: OpenAILLMContext,
|
||||||
|
|||||||
@@ -111,6 +111,15 @@ class GroqTTSService(TTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that Groq TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that Groq's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
@traced_tts
|
@traced_tts
|
||||||
async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]:
|
async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]:
|
||||||
"""Generate speech from text using Groq's TTS API.
|
"""Generate speech from text using Groq's TTS API.
|
||||||
@@ -146,6 +155,7 @@ class GroqTTSService(TTSService):
|
|||||||
bytes = w.readframes(num_frames)
|
bytes = w.readframes(num_frames)
|
||||||
yield TTSAudioRawFrame(bytes, frame_rate, channels)
|
yield TTSAudioRawFrame(bytes, frame_rate, channels)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
|
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
|
|||||||
@@ -179,7 +179,7 @@ class HeyGenClient:
|
|||||||
await self._task_manager.cancel_task(self._event_task)
|
await self._task_manager.cancel_task(self._event_task)
|
||||||
self._event_task = None
|
self._event_task = None
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Exception during cleanup: {e}")
|
logger.exception(f"Exception during cleanup: {e}")
|
||||||
|
|
||||||
async def start(self, frame: StartFrame, audio_chunk_size: int) -> None:
|
async def start(self, frame: StartFrame, audio_chunk_size: int) -> None:
|
||||||
"""Start the client and establish all necessary connections.
|
"""Start the client and establish all necessary connections.
|
||||||
|
|||||||
@@ -14,14 +14,12 @@ from pydantic import BaseModel
|
|||||||
from pipecat.frames.frames import (
|
from pipecat.frames.frames import (
|
||||||
ErrorFrame,
|
ErrorFrame,
|
||||||
Frame,
|
Frame,
|
||||||
InterruptionFrame,
|
|
||||||
StartFrame,
|
StartFrame,
|
||||||
TTSAudioRawFrame,
|
TTSAudioRawFrame,
|
||||||
TTSStartedFrame,
|
TTSStartedFrame,
|
||||||
TTSStoppedFrame,
|
TTSStoppedFrame,
|
||||||
)
|
)
|
||||||
from pipecat.processors.frame_processor import FrameDirection
|
from pipecat.services.tts_service import TTSService
|
||||||
from pipecat.services.tts_service import WordTTSService
|
|
||||||
from pipecat.utils.tracing.service_decorators import traced_tts
|
from pipecat.utils.tracing.service_decorators import traced_tts
|
||||||
|
|
||||||
try:
|
try:
|
||||||
@@ -31,7 +29,6 @@ try:
|
|||||||
PostedUtterance,
|
PostedUtterance,
|
||||||
PostedUtteranceVoiceWithId,
|
PostedUtteranceVoiceWithId,
|
||||||
)
|
)
|
||||||
from hume.tts.types import TimestampMessage
|
|
||||||
except ModuleNotFoundError as e: # pragma: no cover - import-time guidance
|
except ModuleNotFoundError as e: # pragma: no cover - import-time guidance
|
||||||
logger.error(f"Exception: {e}")
|
logger.error(f"Exception: {e}")
|
||||||
logger.error("In order to use Hume, you need to `pip install pipecat-ai[hume]`.")
|
logger.error("In order to use Hume, you need to `pip install pipecat-ai[hume]`.")
|
||||||
@@ -41,7 +38,7 @@ except ModuleNotFoundError as e: # pragma: no cover - import-time guidance
|
|||||||
HUME_SAMPLE_RATE = 48_000 # Hume TTS streams at 48 kHz
|
HUME_SAMPLE_RATE = 48_000 # Hume TTS streams at 48 kHz
|
||||||
|
|
||||||
|
|
||||||
class HumeTTSService(WordTTSService):
|
class HumeTTSService(TTSService):
|
||||||
"""Hume Octave Text-to-Speech service.
|
"""Hume Octave Text-to-Speech service.
|
||||||
|
|
||||||
Streams PCM audio via Hume's HTTP output streaming (JSON chunks) endpoint
|
Streams PCM audio via Hume's HTTP output streaming (JSON chunks) endpoint
|
||||||
@@ -51,7 +48,6 @@ class HumeTTSService(WordTTSService):
|
|||||||
|
|
||||||
- Generates speech from text using Hume TTS.
|
- Generates speech from text using Hume TTS.
|
||||||
- Streams PCM audio.
|
- Streams PCM audio.
|
||||||
- Supports word-level timestamps for precise audio-text synchronization.
|
|
||||||
- Supports dynamic updates of voice and synthesis parameters at runtime.
|
- Supports dynamic updates of voice and synthesis parameters at runtime.
|
||||||
- Provides metrics for Time To First Byte (TTFB) and TTS usage.
|
- Provides metrics for Time To First Byte (TTFB) and TTS usage.
|
||||||
"""
|
"""
|
||||||
@@ -96,13 +92,7 @@ class HumeTTSService(WordTTSService):
|
|||||||
f"Hume TTS streams at {HUME_SAMPLE_RATE} Hz; configured sample_rate={sample_rate}"
|
f"Hume TTS streams at {HUME_SAMPLE_RATE} Hz; configured sample_rate={sample_rate}"
|
||||||
)
|
)
|
||||||
|
|
||||||
# WordTTSService sets push_text_frames=False by default, which we want
|
super().__init__(sample_rate=sample_rate, **kwargs)
|
||||||
super().__init__(
|
|
||||||
sample_rate=sample_rate,
|
|
||||||
push_text_frames=False,
|
|
||||||
push_stop_frames=True,
|
|
||||||
**kwargs,
|
|
||||||
)
|
|
||||||
|
|
||||||
self._client = AsyncHumeClient(api_key=api_key)
|
self._client = AsyncHumeClient(api_key=api_key)
|
||||||
self._params = params or HumeTTSService.InputParams()
|
self._params = params or HumeTTSService.InputParams()
|
||||||
@@ -112,10 +102,6 @@ class HumeTTSService(WordTTSService):
|
|||||||
|
|
||||||
self._audio_bytes = b""
|
self._audio_bytes = b""
|
||||||
|
|
||||||
# Track cumulative time for word timestamps across utterances
|
|
||||||
self._cumulative_time = 0.0
|
|
||||||
self._started = False
|
|
||||||
|
|
||||||
def can_generate_metrics(self) -> bool:
|
def can_generate_metrics(self) -> bool:
|
||||||
"""Can generate metrics.
|
"""Can generate metrics.
|
||||||
|
|
||||||
@@ -124,6 +110,15 @@ class HumeTTSService(WordTTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that Hume TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that Hume's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
async def start(self, frame: StartFrame) -> None:
|
async def start(self, frame: StartFrame) -> None:
|
||||||
"""Start the service.
|
"""Start the service.
|
||||||
|
|
||||||
@@ -131,27 +126,6 @@ class HumeTTSService(WordTTSService):
|
|||||||
frame: The start frame.
|
frame: The start frame.
|
||||||
"""
|
"""
|
||||||
await super().start(frame)
|
await super().start(frame)
|
||||||
self._reset_state()
|
|
||||||
|
|
||||||
def _reset_state(self):
|
|
||||||
"""Reset internal state variables."""
|
|
||||||
self._cumulative_time = 0.0
|
|
||||||
self._started = False
|
|
||||||
|
|
||||||
async def push_frame(self, frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM):
|
|
||||||
"""Push a frame and handle state changes.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
frame: The frame to push.
|
|
||||||
direction: The direction to push the frame.
|
|
||||||
"""
|
|
||||||
await super().push_frame(frame, direction)
|
|
||||||
if isinstance(frame, (InterruptionFrame, TTSStoppedFrame)):
|
|
||||||
# Reset timing on interruption or stop
|
|
||||||
self._reset_state()
|
|
||||||
|
|
||||||
if isinstance(frame, TTSStoppedFrame):
|
|
||||||
await self.add_word_timestamps([("Reset", 0)])
|
|
||||||
|
|
||||||
async def update_setting(self, key: str, value: Any) -> None:
|
async def update_setting(self, key: str, value: Any) -> None:
|
||||||
"""Runtime updates via `TTSUpdateSettingsFrame`.
|
"""Runtime updates via `TTSUpdateSettingsFrame`.
|
||||||
@@ -168,7 +142,7 @@ class HumeTTSService(WordTTSService):
|
|||||||
|
|
||||||
if key_l == "voice_id":
|
if key_l == "voice_id":
|
||||||
self.set_voice(str(value))
|
self.set_voice(str(value))
|
||||||
logger.debug(f"HumeTTSService voice_id set to: {self.voice}")
|
logger.info(f"HumeTTSService voice_id set to: {self.voice}")
|
||||||
elif key_l == "description":
|
elif key_l == "description":
|
||||||
self._params.description = None if value is None else str(value)
|
self._params.description = None if value is None else str(value)
|
||||||
elif key_l == "speed":
|
elif key_l == "speed":
|
||||||
@@ -181,7 +155,7 @@ class HumeTTSService(WordTTSService):
|
|||||||
|
|
||||||
@traced_tts
|
@traced_tts
|
||||||
async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]:
|
async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]:
|
||||||
"""Generate speech from text using Hume TTS with word timestamps.
|
"""Generate speech from text using Hume TTS.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
text: The text to be synthesized.
|
text: The text to be synthesized.
|
||||||
@@ -212,12 +186,7 @@ class HumeTTSService(WordTTSService):
|
|||||||
|
|
||||||
await self.start_ttfb_metrics()
|
await self.start_ttfb_metrics()
|
||||||
await self.start_tts_usage_metrics(text)
|
await self.start_tts_usage_metrics(text)
|
||||||
|
yield TTSStartedFrame()
|
||||||
# Start TTS sequence if not already started
|
|
||||||
if not self._started:
|
|
||||||
self.start_word_timestamps()
|
|
||||||
yield TTSStartedFrame()
|
|
||||||
self._started = True
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
# Instant mode is always enabled here (not user-configurable)
|
# Instant mode is always enabled here (not user-configurable)
|
||||||
@@ -228,50 +197,23 @@ class HumeTTSService(WordTTSService):
|
|||||||
# Use version "2" by default if no description is provided
|
# Use version "2" by default if no description is provided
|
||||||
# Version "1" is needed when description is used
|
# Version "1" is needed when description is used
|
||||||
version = "1" if self._params.description is not None else "2"
|
version = "1" if self._params.description is not None else "2"
|
||||||
|
|
||||||
# Track the duration of this utterance based on the last timestamp
|
|
||||||
utterance_duration = 0.0
|
|
||||||
|
|
||||||
async for chunk in self._client.tts.synthesize_json_streaming(
|
async for chunk in self._client.tts.synthesize_json_streaming(
|
||||||
utterances=[utterance],
|
utterances=[utterance],
|
||||||
format=pcm_fmt,
|
format=pcm_fmt,
|
||||||
instant_mode=True,
|
instant_mode=True,
|
||||||
version=version,
|
version=version,
|
||||||
include_timestamp_types=["word"], # Request word-level timestamps
|
|
||||||
):
|
):
|
||||||
# Process audio chunks
|
|
||||||
audio_b64 = getattr(chunk, "audio", None)
|
audio_b64 = getattr(chunk, "audio", None)
|
||||||
if audio_b64:
|
if not audio_b64:
|
||||||
await self.stop_ttfb_metrics()
|
continue
|
||||||
pcm_bytes = base64.b64decode(audio_b64)
|
|
||||||
self._audio_bytes += pcm_bytes
|
|
||||||
|
|
||||||
# Buffer audio until we have enough to avoid glitches
|
pcm_bytes = base64.b64decode(audio_b64)
|
||||||
if len(self._audio_bytes) >= self.chunk_size:
|
self._audio_bytes += pcm_bytes
|
||||||
frame = TTSAudioRawFrame(
|
|
||||||
audio=self._audio_bytes,
|
|
||||||
sample_rate=self.sample_rate,
|
|
||||||
num_channels=1,
|
|
||||||
)
|
|
||||||
yield frame
|
|
||||||
self._audio_bytes = b""
|
|
||||||
|
|
||||||
# Process timestamp messages
|
# Buffer audio until we have enough to avoid glitches
|
||||||
if isinstance(chunk, TimestampMessage):
|
if len(self._audio_bytes) < self.chunk_size:
|
||||||
timestamp = chunk.timestamp
|
continue
|
||||||
if timestamp.type == "word":
|
|
||||||
# Convert milliseconds to seconds and add cumulative offset
|
|
||||||
word_start_time = self._cumulative_time + (timestamp.time.begin / 1000.0)
|
|
||||||
word_end_time = self._cumulative_time + (timestamp.time.end / 1000.0)
|
|
||||||
|
|
||||||
# Track the maximum end time for this utterance
|
|
||||||
utterance_duration = max(utterance_duration, word_end_time)
|
|
||||||
|
|
||||||
# Add word timestamp
|
|
||||||
await self.add_word_timestamps([(timestamp.text, word_start_time)])
|
|
||||||
|
|
||||||
# Flush any remaining audio bytes
|
|
||||||
if self._audio_bytes:
|
|
||||||
frame = TTSAudioRawFrame(
|
frame = TTSAudioRawFrame(
|
||||||
audio=self._audio_bytes,
|
audio=self._audio_bytes,
|
||||||
sample_rate=self.sample_rate,
|
sample_rate=self.sample_rate,
|
||||||
@@ -282,13 +224,10 @@ class HumeTTSService(WordTTSService):
|
|||||||
|
|
||||||
self._audio_bytes = b""
|
self._audio_bytes = b""
|
||||||
|
|
||||||
# Update cumulative time for next utterance
|
|
||||||
if utterance_duration > 0:
|
|
||||||
self._cumulative_time = utterance_duration
|
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
finally:
|
finally:
|
||||||
# Ensure TTFB timer is stopped even on early failures
|
# Ensure TTFB timer is stopped even on early failures
|
||||||
await self.stop_ttfb_metrics()
|
await self.stop_ttfb_metrics()
|
||||||
# Let the parent class handle TTSStoppedFrame via push_stop_frames
|
yield TTSStoppedFrame()
|
||||||
|
|||||||
@@ -146,8 +146,6 @@ class InworldTTSService(TTSService):
|
|||||||
Parameters:
|
Parameters:
|
||||||
temperature: Voice temperature control for synthesis variability (e.g., 1.1).
|
temperature: Voice temperature control for synthesis variability (e.g., 1.1).
|
||||||
Valid range: [0, 2]. Higher values increase variability.
|
Valid range: [0, 2]. Higher values increase variability.
|
||||||
speaking_rate: Speaking speed control (range: [0.5, 1.5]). Defaults to 1.0 when
|
|
||||||
unset.
|
|
||||||
|
|
||||||
Note:
|
Note:
|
||||||
Language is automatically inferred from the input text by Inworld's TTS models,
|
Language is automatically inferred from the input text by Inworld's TTS models,
|
||||||
@@ -155,7 +153,6 @@ class InworldTTSService(TTSService):
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
temperature: Optional[float] = None # optional temperature control (range: [0, 2])
|
temperature: Optional[float] = None # optional temperature control (range: [0, 2])
|
||||||
speaking_rate: Optional[float] = None # optional speaking rate control (range: [0.5, 1.5])
|
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
self,
|
self,
|
||||||
@@ -201,7 +198,6 @@ class InworldTTSService(TTSService):
|
|||||||
- Other formats as supported by Inworld API
|
- Other formats as supported by Inworld API
|
||||||
params: Optional input parameters for additional configuration. Use this to specify:
|
params: Optional input parameters for additional configuration. Use this to specify:
|
||||||
- temperature: Voice temperature control for variability (range: [0, 2], e.g., 1.1, optional)
|
- temperature: Voice temperature control for variability (range: [0, 2], e.g., 1.1, optional)
|
||||||
- speaking_rate: Set desired speaking speed (range: [0.5, 1.5], optional)
|
|
||||||
Language is automatically inferred from input text.
|
Language is automatically inferred from input text.
|
||||||
**kwargs: Additional arguments passed to the parent TTSService class.
|
**kwargs: Additional arguments passed to the parent TTSService class.
|
||||||
|
|
||||||
@@ -232,18 +228,15 @@ class InworldTTSService(TTSService):
|
|||||||
self._settings = {
|
self._settings = {
|
||||||
"voiceId": voice_id, # Voice selection from direct parameter
|
"voiceId": voice_id, # Voice selection from direct parameter
|
||||||
"modelId": model, # TTS model selection from direct parameter
|
"modelId": model, # TTS model selection from direct parameter
|
||||||
"audioConfig": { # Audio format configuration
|
"audio_config": { # Audio format configuration
|
||||||
"audioEncoding": encoding, # Format: LINEAR16, MP3, etc.
|
"audio_encoding": encoding, # Format: LINEAR16, MP3, etc.
|
||||||
"sampleRateHertz": 0, # Will be set in start() from parent service
|
"sample_rate_hertz": 0, # Will be set in start() from parent service
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
# Add optional temperature parameter if provided (valid range: [0, 2])
|
# Add optional temperature parameter if provided (valid range: [0, 2])
|
||||||
if params and params.temperature is not None:
|
if params and params.temperature is not None:
|
||||||
self._settings["temperature"] = params.temperature
|
self._settings["temperature"] = params.temperature
|
||||||
# Add optional speaking rate if provided (valid range: [0.5, 1.5])
|
|
||||||
if params and params.speaking_rate is not None:
|
|
||||||
self._settings["audioConfig"]["speakingRate"] = params.speaking_rate
|
|
||||||
|
|
||||||
# Register voice and model with parent service for metrics and tracking
|
# Register voice and model with parent service for metrics and tracking
|
||||||
self.set_voice(voice_id) # Used for logging and metrics
|
self.set_voice(voice_id) # Used for logging and metrics
|
||||||
@@ -257,6 +250,15 @@ class InworldTTSService(TTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that Inworld TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that Inworld's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
async def start(self, frame: StartFrame):
|
async def start(self, frame: StartFrame):
|
||||||
"""Start the Inworld TTS service.
|
"""Start the Inworld TTS service.
|
||||||
|
|
||||||
@@ -264,7 +266,7 @@ class InworldTTSService(TTSService):
|
|||||||
frame: The start frame containing initialization parameters.
|
frame: The start frame containing initialization parameters.
|
||||||
"""
|
"""
|
||||||
await super().start(frame)
|
await super().start(frame)
|
||||||
self._settings["audioConfig"]["sampleRateHertz"] = self.sample_rate
|
self._settings["audio_config"]["sample_rate_hertz"] = self.sample_rate
|
||||||
|
|
||||||
async def stop(self, frame: EndFrame):
|
async def stop(self, frame: EndFrame):
|
||||||
"""Stop the Inworld TTS service.
|
"""Stop the Inworld TTS service.
|
||||||
@@ -330,7 +332,9 @@ class InworldTTSService(TTSService):
|
|||||||
"text": text, # Text to synthesize
|
"text": text, # Text to synthesize
|
||||||
"voiceId": self._settings["voiceId"], # Voice selection (Ashley, Hades, etc.)
|
"voiceId": self._settings["voiceId"], # Voice selection (Ashley, Hades, etc.)
|
||||||
"modelId": self._settings["modelId"], # TTS model (inworld-tts-1)
|
"modelId": self._settings["modelId"], # TTS model (inworld-tts-1)
|
||||||
"audioConfig": self._settings["audioConfig"], # Audio format settings (LINEAR16, 48kHz)
|
"audio_config": self._settings[
|
||||||
|
"audio_config"
|
||||||
|
], # Audio format settings (LINEAR16, 48kHz)
|
||||||
}
|
}
|
||||||
|
|
||||||
# Add optional temperature parameter if configured (valid range: [0, 2])
|
# Add optional temperature parameter if configured (valid range: [0, 2])
|
||||||
@@ -397,7 +401,8 @@ class InworldTTSService(TTSService):
|
|||||||
# STEP 7: ERROR HANDLING
|
# STEP 7: ERROR HANDLING
|
||||||
# ================================================================================
|
# ================================================================================
|
||||||
# Log any unexpected errors and notify the pipeline
|
# Log any unexpected errors and notify the pipeline
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
finally:
|
finally:
|
||||||
# ================================================================================
|
# ================================================================================
|
||||||
# STEP 8: CLEANUP AND COMPLETION
|
# STEP 8: CLEANUP AND COMPLETION
|
||||||
@@ -512,7 +517,7 @@ class InworldTTSService(TTSService):
|
|||||||
# Extract the base64-encoded audio content from response
|
# Extract the base64-encoded audio content from response
|
||||||
if "audioContent" not in response_data:
|
if "audioContent" not in response_data:
|
||||||
logger.error("No audioContent in Inworld API response")
|
logger.error("No audioContent in Inworld API response")
|
||||||
yield ErrorFrame(error="No audioContent in response")
|
await self.push_error(ErrorFrame(error="No audioContent in response"))
|
||||||
return
|
return
|
||||||
|
|
||||||
# ================================================================================
|
# ================================================================================
|
||||||
|
|||||||
@@ -9,7 +9,17 @@
|
|||||||
import asyncio
|
import asyncio
|
||||||
import inspect
|
import inspect
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from typing import Any, Awaitable, Callable, Dict, Mapping, Optional, Protocol, Sequence, Type
|
from typing import (
|
||||||
|
Any,
|
||||||
|
Awaitable,
|
||||||
|
Callable,
|
||||||
|
Dict,
|
||||||
|
Mapping,
|
||||||
|
Optional,
|
||||||
|
Protocol,
|
||||||
|
Sequence,
|
||||||
|
Type,
|
||||||
|
)
|
||||||
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
@@ -275,13 +285,17 @@ class LLMService(AIService):
|
|||||||
elif isinstance(frame, LLMConfigureOutputFrame):
|
elif isinstance(frame, LLMConfigureOutputFrame):
|
||||||
self._skip_tts = frame.skip_tts
|
self._skip_tts = frame.skip_tts
|
||||||
|
|
||||||
def _get_skip_tts(self) -> bool:
|
async def push_frame(self, frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM):
|
||||||
"""Get the current skip_tts configuration.
|
"""Pushes a frame.
|
||||||
|
|
||||||
Returns:
|
Args:
|
||||||
The current skip_tts setting for frames generated by this LLM.
|
frame: The frame to push.
|
||||||
|
direction: The direction of frame pushing.
|
||||||
"""
|
"""
|
||||||
return self._skip_tts
|
if isinstance(frame, (LLMTextFrame, LLMFullResponseStartFrame, LLMFullResponseEndFrame)):
|
||||||
|
frame.skip_tts = self._skip_tts
|
||||||
|
|
||||||
|
await super().push_frame(frame, direction)
|
||||||
|
|
||||||
async def _handle_interruptions(self, _: InterruptionFrame):
|
async def _handle_interruptions(self, _: InterruptionFrame):
|
||||||
for function_name, entry in self._functions.items():
|
for function_name, entry in self._functions.items():
|
||||||
|
|||||||
@@ -124,6 +124,15 @@ class LmntTTSService(InterruptibleTTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that LMNT TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that LMNT's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
def language_to_service_language(self, language: Language) -> Optional[str]:
|
def language_to_service_language(self, language: Language) -> Optional[str]:
|
||||||
"""Convert a Language enum to LMNT service language format.
|
"""Convert a Language enum to LMNT service language format.
|
||||||
|
|
||||||
@@ -214,7 +223,8 @@ class LmntTTSService(InterruptibleTTSService):
|
|||||||
|
|
||||||
await self._call_event_handler("on_connected")
|
await self._call_event_handler("on_connected")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
await self._call_event_handler("on_connection_error", f"{e}")
|
await self._call_event_handler("on_connection_error", f"{e}")
|
||||||
|
|
||||||
@@ -230,7 +240,8 @@ class LmntTTSService(InterruptibleTTSService):
|
|||||||
# await self._websocket.send(json.dumps({"eof": True}))
|
# await self._websocket.send(json.dumps({"eof": True}))
|
||||||
await self._websocket.close()
|
await self._websocket.close()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error disconnecting from LMNT: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
finally:
|
finally:
|
||||||
self._started = False
|
self._started = False
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
@@ -264,9 +275,10 @@ class LmntTTSService(InterruptibleTTSService):
|
|||||||
try:
|
try:
|
||||||
msg = json.loads(message)
|
msg = json.loads(message)
|
||||||
if "error" in msg:
|
if "error" in msg:
|
||||||
|
logger.error(f"{self} error: {msg['error']}")
|
||||||
await self.push_frame(TTSStoppedFrame())
|
await self.push_frame(TTSStoppedFrame())
|
||||||
await self.stop_all_metrics()
|
await self.stop_all_metrics()
|
||||||
await self.push_error(error_msg=f"Error: {msg['error']}")
|
await self.push_error(ErrorFrame(error=f"{self} error: {msg['error']}"))
|
||||||
return
|
return
|
||||||
except json.JSONDecodeError:
|
except json.JSONDecodeError:
|
||||||
logger.error(f"Invalid JSON message: {message}")
|
logger.error(f"Invalid JSON message: {message}")
|
||||||
@@ -299,11 +311,13 @@ class LmntTTSService(InterruptibleTTSService):
|
|||||||
await self._get_websocket().send(json.dumps({"flush": True}))
|
await self._get_websocket().send(json.dumps({"flush": True}))
|
||||||
await self.start_tts_usage_metrics(text)
|
await self.start_tts_usage_metrics(text)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
await self._disconnect()
|
await self._disconnect()
|
||||||
await self._connect()
|
await self._connect()
|
||||||
return
|
return
|
||||||
yield None
|
yield None
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
|
|||||||
@@ -176,6 +176,7 @@ class MCPClient(BaseObject):
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
error_msg = f"Error calling mcp tool {params.function_name}: {str(e)}"
|
error_msg = f"Error calling mcp tool {params.function_name}: {str(e)}"
|
||||||
logger.error(error_msg)
|
logger.error(error_msg)
|
||||||
|
logger.exception("Full exception details:")
|
||||||
await params.result_callback(error_msg)
|
await params.result_callback(error_msg)
|
||||||
|
|
||||||
async def _stdio_list_tools(self) -> ToolsSchema:
|
async def _stdio_list_tools(self) -> ToolsSchema:
|
||||||
@@ -206,6 +207,7 @@ class MCPClient(BaseObject):
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
error_msg = f"Error calling mcp tool {params.function_name}: {str(e)}"
|
error_msg = f"Error calling mcp tool {params.function_name}: {str(e)}"
|
||||||
logger.error(error_msg)
|
logger.error(error_msg)
|
||||||
|
logger.exception("Full exception details:")
|
||||||
await params.result_callback(error_msg)
|
await params.result_callback(error_msg)
|
||||||
|
|
||||||
async def _streamable_http_list_tools(self) -> ToolsSchema:
|
async def _streamable_http_list_tools(self) -> ToolsSchema:
|
||||||
@@ -244,6 +246,7 @@ class MCPClient(BaseObject):
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
error_msg = f"Error calling mcp tool {params.function_name}: {str(e)}"
|
error_msg = f"Error calling mcp tool {params.function_name}: {str(e)}"
|
||||||
logger.error(error_msg)
|
logger.error(error_msg)
|
||||||
|
logger.exception("Full exception details:")
|
||||||
await params.result_callback(error_msg)
|
await params.result_callback(error_msg)
|
||||||
|
|
||||||
async def _call_tool(self, session, function_name, arguments, result_callback):
|
async def _call_tool(self, session, function_name, arguments, result_callback):
|
||||||
@@ -299,6 +302,7 @@ class MCPClient(BaseObject):
|
|||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Failed to read tool '{tool_name}': {str(e)}")
|
logger.error(f"Failed to read tool '{tool_name}': {str(e)}")
|
||||||
|
logger.exception("Full exception details:")
|
||||||
continue
|
continue
|
||||||
|
|
||||||
logger.debug(f"Completed reading {len(tool_schemas)} tools")
|
logger.debug(f"Completed reading {len(tool_schemas)} tools")
|
||||||
|
|||||||
@@ -253,9 +253,8 @@ class Mem0MemoryService(FrameProcessor):
|
|||||||
# Otherwise, pass the enhanced context frame downstream
|
# Otherwise, pass the enhanced context frame downstream
|
||||||
await self.push_frame(frame)
|
await self.push_frame(frame)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(
|
logger.error(f"Error processing with Mem0: {str(e)}")
|
||||||
error_msg=f"Error processing with Mem0: {str(e)}", exception=e
|
await self.push_frame(ErrorFrame(f"Error processing with Mem0: {str(e)}"))
|
||||||
)
|
|
||||||
await self.push_frame(frame) # Still pass the original frame through
|
await self.push_frame(frame) # Still pass the original frame through
|
||||||
else:
|
else:
|
||||||
# For non-context frames, just pass them through
|
# For non-context frames, just pass them through
|
||||||
|
|||||||
@@ -40,40 +40,24 @@ def language_to_minimax_language(language: Language) -> Optional[str]:
|
|||||||
The corresponding MiniMax language name, or None if not supported.
|
The corresponding MiniMax language name, or None if not supported.
|
||||||
"""
|
"""
|
||||||
LANGUAGE_MAP = {
|
LANGUAGE_MAP = {
|
||||||
Language.AF: "Afrikaans",
|
|
||||||
Language.AR: "Arabic",
|
Language.AR: "Arabic",
|
||||||
Language.BG: "Bulgarian",
|
|
||||||
Language.CA: "Catalan",
|
|
||||||
Language.CS: "Czech",
|
Language.CS: "Czech",
|
||||||
Language.DA: "Danish",
|
|
||||||
Language.DE: "German",
|
Language.DE: "German",
|
||||||
Language.EL: "Greek",
|
Language.EL: "Greek",
|
||||||
Language.EN: "English",
|
Language.EN: "English",
|
||||||
Language.ES: "Spanish",
|
Language.ES: "Spanish",
|
||||||
Language.FA: "Persian", # ⚠️ Only supported by speech-2.6-* models
|
|
||||||
Language.FI: "Finnish",
|
Language.FI: "Finnish",
|
||||||
Language.FIL: "Filipino", # ⚠️ Only supported by speech-2.6-* models
|
|
||||||
Language.FR: "French",
|
Language.FR: "French",
|
||||||
Language.HE: "Hebrew",
|
|
||||||
Language.HI: "Hindi",
|
Language.HI: "Hindi",
|
||||||
Language.HR: "Croatian",
|
|
||||||
Language.HU: "Hungarian",
|
|
||||||
Language.ID: "Indonesian",
|
Language.ID: "Indonesian",
|
||||||
Language.IT: "Italian",
|
Language.IT: "Italian",
|
||||||
Language.JA: "Japanese",
|
Language.JA: "Japanese",
|
||||||
Language.KO: "Korean",
|
Language.KO: "Korean",
|
||||||
Language.MS: "Malay",
|
|
||||||
Language.NB: "Norwegian",
|
|
||||||
Language.NN: "Nynorsk",
|
|
||||||
Language.NL: "Dutch",
|
Language.NL: "Dutch",
|
||||||
Language.PL: "Polish",
|
Language.PL: "Polish",
|
||||||
Language.PT: "Portuguese",
|
Language.PT: "Portuguese",
|
||||||
Language.RO: "Romanian",
|
Language.RO: "Romanian",
|
||||||
Language.RU: "Russian",
|
Language.RU: "Russian",
|
||||||
Language.SK: "Slovak",
|
|
||||||
Language.SL: "Slovenian",
|
|
||||||
Language.SV: "Swedish",
|
|
||||||
Language.TA: "Tamil", # ⚠️ Only supported by speech-2.6-* models
|
|
||||||
Language.TH: "Thai",
|
Language.TH: "Thai",
|
||||||
Language.TR: "Turkish",
|
Language.TR: "Turkish",
|
||||||
Language.UK: "Ukrainian",
|
Language.UK: "Ukrainian",
|
||||||
@@ -100,22 +84,13 @@ class MiniMaxHttpTTSService(TTSService):
|
|||||||
"""Configuration parameters for MiniMax TTS.
|
"""Configuration parameters for MiniMax TTS.
|
||||||
|
|
||||||
Parameters:
|
Parameters:
|
||||||
language: Language for TTS generation. Supports 40 languages.
|
language: Language for TTS generation.
|
||||||
Note: Filipino, Tamil, and Persian require speech-2.6-* models.
|
|
||||||
speed: Speech speed (range: 0.5 to 2.0).
|
speed: Speech speed (range: 0.5 to 2.0).
|
||||||
volume: Speech volume (range: 0 to 10).
|
volume: Speech volume (range: 0 to 10).
|
||||||
pitch: Pitch adjustment (range: -12 to 12).
|
pitch: Pitch adjustment (range: -12 to 12).
|
||||||
emotion: Emotional tone (options: "happy", "sad", "angry", "fearful",
|
emotion: Emotional tone (options: "happy", "sad", "angry", "fearful",
|
||||||
"disgusted", "surprised", "calm", "fluent").
|
"disgusted", "surprised", "neutral").
|
||||||
english_normalization: Deprecated; use `text_normalization` instead
|
english_normalization: Whether to apply English text normalization.
|
||||||
|
|
||||||
.. deprecated:: 0.0.96
|
|
||||||
The `english_normalization` parameter is deprecated and will be removed in a future version.
|
|
||||||
Use the `text_normalization` parameter instead.
|
|
||||||
|
|
||||||
text_normalization: Enable text normalization (Chinese/English).
|
|
||||||
latex_read: Enable LaTeX formula reading.
|
|
||||||
exclude_aggregated_audio: Whether to exclude aggregated audio in final chunk.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
language: Optional[Language] = Language.EN
|
language: Optional[Language] = Language.EN
|
||||||
@@ -123,10 +98,7 @@ class MiniMaxHttpTTSService(TTSService):
|
|||||||
volume: Optional[float] = 1.0
|
volume: Optional[float] = 1.0
|
||||||
pitch: Optional[int] = 0
|
pitch: Optional[int] = 0
|
||||||
emotion: Optional[str] = None
|
emotion: Optional[str] = None
|
||||||
english_normalization: Optional[bool] = None # Deprecated
|
english_normalization: Optional[bool] = None
|
||||||
text_normalization: Optional[bool] = None
|
|
||||||
latex_read: Optional[bool] = None
|
|
||||||
exclude_aggregated_audio: Optional[bool] = None
|
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
self,
|
self,
|
||||||
@@ -148,12 +120,9 @@ class MiniMaxHttpTTSService(TTSService):
|
|||||||
base_url: API base URL, defaults to MiniMax's T2A endpoint.
|
base_url: API base URL, defaults to MiniMax's T2A endpoint.
|
||||||
Global: https://api.minimax.io/v1/t2a_v2
|
Global: https://api.minimax.io/v1/t2a_v2
|
||||||
Mainland China: https://api.minimaxi.chat/v1/t2a_v2
|
Mainland China: https://api.minimaxi.chat/v1/t2a_v2
|
||||||
Western United States: https://api-uw.minimax.io/v1/t2a_v2
|
|
||||||
group_id: MiniMax Group ID to identify project.
|
group_id: MiniMax Group ID to identify project.
|
||||||
model: TTS model name. Defaults to "speech-02-turbo". Options include:
|
model: TTS model name. Defaults to "speech-02-turbo". Options include
|
||||||
"speech-2.6-hd", "speech-2.6-turbo" (latest, supports Filipino/Tamil/Persian),
|
"speech-02-hd", "speech-02-turbo", "speech-01-hd", "speech-01-turbo".
|
||||||
"speech-02-hd", "speech-02-turbo",
|
|
||||||
"speech-01-hd", "speech-01-turbo".
|
|
||||||
voice_id: Voice identifier. Defaults to "Calm_Woman".
|
voice_id: Voice identifier. Defaults to "Calm_Woman".
|
||||||
aiohttp_session: aiohttp.ClientSession for API communication.
|
aiohttp_session: aiohttp.ClientSession for API communication.
|
||||||
sample_rate: Output audio sample rate in Hz. If None, uses pipeline default.
|
sample_rate: Output audio sample rate in Hz. If None, uses pipeline default.
|
||||||
@@ -207,34 +176,15 @@ class MiniMaxHttpTTSService(TTSService):
|
|||||||
"disgusted",
|
"disgusted",
|
||||||
"surprised",
|
"surprised",
|
||||||
"neutral",
|
"neutral",
|
||||||
"fluent",
|
|
||||||
]
|
]
|
||||||
if params.emotion in supported_emotions:
|
if params.emotion in supported_emotions:
|
||||||
self._settings["voice_setting"]["emotion"] = params.emotion
|
self._settings["voice_setting"]["emotion"] = params.emotion
|
||||||
else:
|
else:
|
||||||
logger.warning(
|
logger.warning(f"Unsupported emotion: {params.emotion}. Using default.")
|
||||||
f"Unsupported emotion: {params.emotion}. Supported emotions: {supported_emotions}"
|
|
||||||
)
|
|
||||||
|
|
||||||
# If `english_normalization`, add `text_normalization` and print warning
|
# Add english_normalization if provided
|
||||||
if params.english_normalization is not None:
|
if params.english_normalization is not None:
|
||||||
import warnings
|
self._settings["english_normalization"] = params.english_normalization
|
||||||
|
|
||||||
with warnings.catch_warnings():
|
|
||||||
warnings.simplefilter("always")
|
|
||||||
warnings.warn(
|
|
||||||
"Parameter `english_normalization` is deprecated and will be removed in a future version. Use `text_normalization` instead.",
|
|
||||||
DeprecationWarning,
|
|
||||||
)
|
|
||||||
self._settings["voice_setting"]["text_normalization"] = params.english_normalization
|
|
||||||
|
|
||||||
# Add text_normalization if provided (corrected parameter name)
|
|
||||||
if params.text_normalization is not None:
|
|
||||||
self._settings["voice_setting"]["text_normalization"] = params.text_normalization
|
|
||||||
|
|
||||||
# Add latex_read if provided
|
|
||||||
if params.latex_read is not None:
|
|
||||||
self._settings["voice_setting"]["latex_read"] = params.latex_read
|
|
||||||
|
|
||||||
def can_generate_metrics(self) -> bool:
|
def can_generate_metrics(self) -> bool:
|
||||||
"""Check if this service can generate processing metrics.
|
"""Check if this service can generate processing metrics.
|
||||||
@@ -244,6 +194,15 @@ class MiniMaxHttpTTSService(TTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that MiniMax TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that MiniMax's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
def language_to_service_language(self, language: Language) -> Optional[str]:
|
def language_to_service_language(self, language: Language) -> Optional[str]:
|
||||||
"""Convert a Language enum to MiniMax service language format.
|
"""Convert a Language enum to MiniMax service language format.
|
||||||
|
|
||||||
@@ -281,7 +240,7 @@ class MiniMaxHttpTTSService(TTSService):
|
|||||||
"""
|
"""
|
||||||
await super().start(frame)
|
await super().start(frame)
|
||||||
self._settings["audio_setting"]["sample_rate"] = self.sample_rate
|
self._settings["audio_setting"]["sample_rate"] = self.sample_rate
|
||||||
logger.debug(f"MiniMax TTS initialized with sample_rate: {self.sample_rate}")
|
logger.debug(f"MiniMax TTS initialized with sample rate: {self.sample_rate}")
|
||||||
|
|
||||||
@traced_tts
|
@traced_tts
|
||||||
async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]:
|
async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]:
|
||||||
@@ -314,6 +273,7 @@ class MiniMaxHttpTTSService(TTSService):
|
|||||||
) as response:
|
) as response:
|
||||||
if response.status != 200:
|
if response.status != 200:
|
||||||
error_message = f"MiniMax TTS error: HTTP {response.status}"
|
error_message = f"MiniMax TTS error: HTTP {response.status}"
|
||||||
|
logger.error(error_message)
|
||||||
yield ErrorFrame(error=error_message)
|
yield ErrorFrame(error=error_message)
|
||||||
return
|
return
|
||||||
|
|
||||||
@@ -379,19 +339,16 @@ class MiniMaxHttpTTSService(TTSService):
|
|||||||
num_channels=1,
|
num_channels=1,
|
||||||
)
|
)
|
||||||
except ValueError as e:
|
except ValueError as e:
|
||||||
logger.error(
|
logger.error(f"Error converting hex to binary: {e}")
|
||||||
f"Error converting hex to binary: {e}",
|
|
||||||
)
|
|
||||||
continue
|
continue
|
||||||
|
|
||||||
except json.JSONDecodeError as e:
|
except json.JSONDecodeError as e:
|
||||||
logger.error(
|
logger.error(f"Error decoding JSON: {e}, data: {data_block[:100]}")
|
||||||
f"Error decoding JSON: {e}, data: {data_block[:100]}",
|
|
||||||
)
|
|
||||||
continue
|
continue
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
finally:
|
finally:
|
||||||
await self.stop_ttfb_metrics()
|
await self.stop_ttfb_metrics()
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
|
|||||||
@@ -110,6 +110,7 @@ class MoondreamService(VisionService):
|
|||||||
if analysis fails.
|
if analysis fails.
|
||||||
"""
|
"""
|
||||||
if not self._model:
|
if not self._model:
|
||||||
|
logger.error(f"{self} error: Moondream model not available ({self.model_name})")
|
||||||
yield ErrorFrame("Moondream model not available")
|
yield ErrorFrame("Moondream model not available")
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|||||||
@@ -151,6 +151,15 @@ class NeuphonicTTSService(InterruptibleTTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that Neuphonic TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that Neuphonic's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
def language_to_service_language(self, language: Language) -> Optional[str]:
|
def language_to_service_language(self, language: Language) -> Optional[str]:
|
||||||
"""Convert a Language enum to Neuphonic service language format.
|
"""Convert a Language enum to Neuphonic service language format.
|
||||||
|
|
||||||
@@ -285,7 +294,8 @@ class NeuphonicTTSService(InterruptibleTTSService):
|
|||||||
|
|
||||||
await self._call_event_handler("on_connected")
|
await self._call_event_handler("on_connected")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
await self._call_event_handler("on_connection_error", f"{e}")
|
await self._call_event_handler("on_connection_error", f"{e}")
|
||||||
|
|
||||||
@@ -298,7 +308,8 @@ class NeuphonicTTSService(InterruptibleTTSService):
|
|||||||
logger.debug("Disconnecting from Neuphonic")
|
logger.debug("Disconnecting from Neuphonic")
|
||||||
await self._websocket.close()
|
await self._websocket.close()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
finally:
|
finally:
|
||||||
self._started = False
|
self._started = False
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
@@ -363,14 +374,16 @@ class NeuphonicTTSService(InterruptibleTTSService):
|
|||||||
await self._send_text(text)
|
await self._send_text(text)
|
||||||
await self.start_tts_usage_metrics(text)
|
await self.start_tts_usage_metrics(text)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
await self._disconnect()
|
await self._disconnect()
|
||||||
await self._connect()
|
await self._connect()
|
||||||
return
|
return
|
||||||
yield None
|
yield None
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
|
|
||||||
|
|
||||||
class NeuphonicHttpTTSService(TTSService):
|
class NeuphonicHttpTTSService(TTSService):
|
||||||
@@ -436,6 +449,15 @@ class NeuphonicHttpTTSService(TTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that Neuphonic TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that Neuphonic's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
def language_to_service_language(self, language: Language) -> Optional[str]:
|
def language_to_service_language(self, language: Language) -> Optional[str]:
|
||||||
"""Convert a Language enum to Neuphonic service language format.
|
"""Convert a Language enum to Neuphonic service language format.
|
||||||
|
|
||||||
@@ -534,6 +556,7 @@ class NeuphonicHttpTTSService(TTSService):
|
|||||||
if response.status != 200:
|
if response.status != 200:
|
||||||
error_text = await response.text()
|
error_text = await response.text()
|
||||||
error_message = f"Neuphonic API error: HTTP {response.status} - {error_text}"
|
error_message = f"Neuphonic API error: HTTP {response.status} - {error_text}"
|
||||||
|
logger.error(error_message)
|
||||||
yield ErrorFrame(error=error_message)
|
yield ErrorFrame(error=error_message)
|
||||||
return
|
return
|
||||||
|
|
||||||
@@ -563,7 +586,8 @@ class NeuphonicHttpTTSService(TTSService):
|
|||||||
yield TTSAudioRawFrame(audio_bytes, self.sample_rate, 1)
|
yield TTSAudioRawFrame(audio_bytes, self.sample_rate, 1)
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
# Don't yield error frame for individual message failures
|
# Don't yield error frame for individual message failures
|
||||||
continue
|
continue
|
||||||
|
|
||||||
@@ -571,7 +595,8 @@ class NeuphonicHttpTTSService(TTSService):
|
|||||||
logger.debug("TTS generation cancelled")
|
logger.debug("TTS generation cancelled")
|
||||||
raise
|
raise
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
finally:
|
finally:
|
||||||
await self.stop_ttfb_metrics()
|
await self.stop_ttfb_metrics()
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
|
|||||||
@@ -13,7 +13,13 @@ from typing import Any, Dict, List, Mapping, Optional
|
|||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
from openai import NOT_GIVEN, APITimeoutError, AsyncOpenAI, AsyncStream, DefaultAsyncHttpxClient
|
from openai import (
|
||||||
|
NOT_GIVEN,
|
||||||
|
APITimeoutError,
|
||||||
|
AsyncOpenAI,
|
||||||
|
AsyncStream,
|
||||||
|
DefaultAsyncHttpxClient,
|
||||||
|
)
|
||||||
from openai.types.chat import ChatCompletionChunk, ChatCompletionMessageParam
|
from openai.types.chat import ChatCompletionChunk, ChatCompletionMessageParam
|
||||||
from pydantic import BaseModel, Field
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
@@ -340,17 +346,11 @@ class BaseOpenAILLMService(LLMService):
|
|||||||
if chunk.usage.prompt_tokens_details
|
if chunk.usage.prompt_tokens_details
|
||||||
else None
|
else None
|
||||||
)
|
)
|
||||||
reasoning_tokens = (
|
|
||||||
chunk.usage.completion_tokens_details.reasoning_tokens
|
|
||||||
if chunk.usage.completion_tokens_details
|
|
||||||
else None
|
|
||||||
)
|
|
||||||
tokens = LLMTokenUsage(
|
tokens = LLMTokenUsage(
|
||||||
prompt_tokens=chunk.usage.prompt_tokens,
|
prompt_tokens=chunk.usage.prompt_tokens,
|
||||||
completion_tokens=chunk.usage.completion_tokens,
|
completion_tokens=chunk.usage.completion_tokens,
|
||||||
total_tokens=chunk.usage.total_tokens,
|
total_tokens=chunk.usage.total_tokens,
|
||||||
cache_read_input_tokens=cached_tokens,
|
cache_read_input_tokens=cached_tokens,
|
||||||
reasoning_tokens=reasoning_tokens,
|
|
||||||
)
|
)
|
||||||
await self.start_llm_usage_metrics(tokens)
|
await self.start_llm_usage_metrics(tokens)
|
||||||
|
|
||||||
@@ -390,20 +390,16 @@ class BaseOpenAILLMService(LLMService):
|
|||||||
# Keep iterating through the response to collect all the argument fragments
|
# Keep iterating through the response to collect all the argument fragments
|
||||||
arguments += tool_call.function.arguments
|
arguments += tool_call.function.arguments
|
||||||
elif chunk.choices[0].delta.content:
|
elif chunk.choices[0].delta.content:
|
||||||
await self.push_frame(
|
frame = LLMTextFrame(chunk.choices[0].delta.content)
|
||||||
LLMTextFrame(chunk.choices[0].delta.content, skip_tts=self._get_skip_tts())
|
frame.includes_inter_frame_spaces = True
|
||||||
)
|
await self.push_frame(frame)
|
||||||
|
|
||||||
# When gpt-4o-audio / gpt-4o-mini-audio is used for llm or stt+llm
|
# When gpt-4o-audio / gpt-4o-mini-audio is used for llm or stt+llm
|
||||||
# we need to get LLMTextFrame for the transcript
|
# we need to get LLMTextFrame for the transcript
|
||||||
elif hasattr(chunk.choices[0].delta, "audio") and chunk.choices[0].delta.audio.get(
|
elif hasattr(chunk.choices[0].delta, "audio") and chunk.choices[0].delta.audio.get(
|
||||||
"transcript"
|
"transcript"
|
||||||
):
|
):
|
||||||
await self.push_frame(
|
await self.push_frame(LLMTextFrame(chunk.choices[0].delta.audio["transcript"]))
|
||||||
LLMTextFrame(
|
|
||||||
chunk.choices[0].delta.audio["transcript"], skip_tts=self._get_skip_tts()
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
# if we got a function name and arguments, check to see if it's a function with
|
# if we got a function name and arguments, check to see if it's a function with
|
||||||
# a registered handler. If so, run the registered callback, save the result to
|
# a registered handler. If so, run the registered callback, save the result to
|
||||||
@@ -463,11 +459,11 @@ class BaseOpenAILLMService(LLMService):
|
|||||||
|
|
||||||
if context:
|
if context:
|
||||||
try:
|
try:
|
||||||
await self.push_frame(LLMFullResponseStartFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseStartFrame())
|
||||||
await self.start_processing_metrics()
|
await self.start_processing_metrics()
|
||||||
await self._process_context(context)
|
await self._process_context(context)
|
||||||
except httpx.TimeoutException:
|
except httpx.TimeoutException:
|
||||||
await self._call_event_handler("on_completion_timeout")
|
await self._call_event_handler("on_completion_timeout")
|
||||||
finally:
|
finally:
|
||||||
await self.stop_processing_metrics()
|
await self.stop_processing_metrics()
|
||||||
await self.push_frame(LLMFullResponseEndFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseEndFrame())
|
||||||
|
|||||||
@@ -76,6 +76,7 @@ class OpenAIImageGenService(ImageGenService):
|
|||||||
image_url = image.data[0].url
|
image_url = image.data[0].url
|
||||||
|
|
||||||
if not image_url:
|
if not image_url:
|
||||||
|
logger.error(f"{self} No image provided in response: {image}")
|
||||||
yield ErrorFrame("Image generation failed")
|
yield ErrorFrame("Image generation failed")
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|||||||
@@ -15,7 +15,9 @@ from typing import Optional
|
|||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
from pipecat.adapters.schemas.tools_schema import ToolsSchema
|
from pipecat.adapters.schemas.tools_schema import ToolsSchema
|
||||||
from pipecat.adapters.services.open_ai_realtime_adapter import OpenAIRealtimeLLMAdapter
|
from pipecat.adapters.services.open_ai_realtime_adapter import (
|
||||||
|
OpenAIRealtimeLLMAdapter,
|
||||||
|
)
|
||||||
from pipecat.frames.frames import (
|
from pipecat.frames.frames import (
|
||||||
AggregationType,
|
AggregationType,
|
||||||
BotStoppedSpeakingFrame,
|
BotStoppedSpeakingFrame,
|
||||||
@@ -55,6 +57,7 @@ from pipecat.processors.aggregators.openai_llm_context import (
|
|||||||
)
|
)
|
||||||
from pipecat.processors.frame_processor import FrameDirection
|
from pipecat.processors.frame_processor import FrameDirection
|
||||||
from pipecat.services.llm_service import FunctionCallFromLLM, LLMService
|
from pipecat.services.llm_service import FunctionCallFromLLM, LLMService
|
||||||
|
from pipecat.services.openai.llm import OpenAIContextAggregatorPair
|
||||||
from pipecat.transcriptions.language import Language
|
from pipecat.transcriptions.language import Language
|
||||||
from pipecat.utils.time import time_now_iso8601
|
from pipecat.utils.time import time_now_iso8601
|
||||||
from pipecat.utils.tracing.service_decorators import traced_openai_realtime, traced_stt
|
from pipecat.utils.tracing.service_decorators import traced_openai_realtime, traced_stt
|
||||||
@@ -282,7 +285,7 @@ class OpenAIRealtimeLLMService(LLMService):
|
|||||||
await self._truncate_current_audio_response()
|
await self._truncate_current_audio_response()
|
||||||
await self.stop_all_metrics()
|
await self.stop_all_metrics()
|
||||||
if self._current_assistant_response:
|
if self._current_assistant_response:
|
||||||
await self.push_frame(LLMFullResponseEndFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseEndFrame())
|
||||||
# Only push TTSStoppedFrame if audio modality is enabled
|
# Only push TTSStoppedFrame if audio modality is enabled
|
||||||
if self._is_modality_enabled("audio"):
|
if self._is_modality_enabled("audio"):
|
||||||
await self.push_frame(TTSStoppedFrame())
|
await self.push_frame(TTSStoppedFrame())
|
||||||
@@ -441,7 +444,7 @@ class OpenAIRealtimeLLMService(LLMService):
|
|||||||
)
|
)
|
||||||
self._receive_task = self.create_task(self._receive_task_handler())
|
self._receive_task = self.create_task(self._receive_task_handler())
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error connecting: {e}", exception=e)
|
logger.error(f"{self} initialization error: {e}")
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
|
|
||||||
async def _disconnect(self):
|
async def _disconnect(self):
|
||||||
@@ -458,7 +461,7 @@ class OpenAIRealtimeLLMService(LLMService):
|
|||||||
self._completed_tool_calls = set()
|
self._completed_tool_calls = set()
|
||||||
self._disconnecting = False
|
self._disconnecting = False
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error disconnecting: {e}", exception=e)
|
logger.error(f"{self} error disconnecting: {e}")
|
||||||
|
|
||||||
async def _ws_send(self, realtime_message):
|
async def _ws_send(self, realtime_message):
|
||||||
try:
|
try:
|
||||||
@@ -471,11 +474,12 @@ class OpenAIRealtimeLLMService(LLMService):
|
|||||||
# somehow *started* the websocket send attempt while we still
|
# somehow *started* the websocket send attempt while we still
|
||||||
# had a connection)
|
# had a connection)
|
||||||
return
|
return
|
||||||
|
logger.error(f"Error sending message to websocket: {e}")
|
||||||
# In server-to-server contexts, a WebSocket error should be quite rare. Given how hard
|
# In server-to-server contexts, a WebSocket error should be quite rare. Given how hard
|
||||||
# it is to recover from a send-side error with proper state management, and that exponential
|
# it is to recover from a send-side error with proper state management, and that exponential
|
||||||
# backoff for retries can have cost/stability implications for a service cluster, let's just
|
# backoff for retries can have cost/stability implications for a service cluster, let's just
|
||||||
# treat a send-side error as fatal.
|
# treat a send-side error as fatal.
|
||||||
await self.push_error(error_msg=f"Error sending client event: {e}", exception=e)
|
await self.push_error(ErrorFrame(error=f"Error sending client event: {e}"))
|
||||||
|
|
||||||
async def _update_settings(self):
|
async def _update_settings(self):
|
||||||
settings = self._session_properties
|
settings = self._session_properties
|
||||||
@@ -606,7 +610,7 @@ class OpenAIRealtimeLLMService(LLMService):
|
|||||||
|
|
||||||
if evt.item.role == "assistant":
|
if evt.item.role == "assistant":
|
||||||
self._current_assistant_response = evt.item
|
self._current_assistant_response = evt.item
|
||||||
await self.push_frame(LLMFullResponseStartFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseStartFrame())
|
||||||
|
|
||||||
async def _handle_evt_conversation_item_done(self, evt):
|
async def _handle_evt_conversation_item_done(self, evt):
|
||||||
"""Handle conversation.item.done event - item is fully completed."""
|
"""Handle conversation.item.done event - item is fully completed."""
|
||||||
@@ -653,25 +657,18 @@ class OpenAIRealtimeLLMService(LLMService):
|
|||||||
async def _handle_evt_response_done(self, evt):
|
async def _handle_evt_response_done(self, evt):
|
||||||
# todo: figure out whether there's anything we need to do for "cancelled" events
|
# todo: figure out whether there's anything we need to do for "cancelled" events
|
||||||
# usage metrics
|
# usage metrics
|
||||||
cached_tokens = (
|
|
||||||
evt.response.usage.input_token_details.cached_tokens
|
|
||||||
if hasattr(evt.response.usage, "input_token_details")
|
|
||||||
and evt.response.usage.input_token_details
|
|
||||||
else None
|
|
||||||
)
|
|
||||||
tokens = LLMTokenUsage(
|
tokens = LLMTokenUsage(
|
||||||
prompt_tokens=evt.response.usage.input_tokens,
|
prompt_tokens=evt.response.usage.input_tokens,
|
||||||
completion_tokens=evt.response.usage.output_tokens,
|
completion_tokens=evt.response.usage.output_tokens,
|
||||||
total_tokens=evt.response.usage.total_tokens,
|
total_tokens=evt.response.usage.total_tokens,
|
||||||
cache_read_input_tokens=cached_tokens,
|
|
||||||
)
|
)
|
||||||
await self.start_llm_usage_metrics(tokens)
|
await self.start_llm_usage_metrics(tokens)
|
||||||
await self.stop_processing_metrics()
|
await self.stop_processing_metrics()
|
||||||
await self.push_frame(LLMFullResponseEndFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseEndFrame())
|
||||||
self._current_assistant_response = None
|
self._current_assistant_response = None
|
||||||
# error handling
|
# error handling
|
||||||
if evt.response.status == "failed":
|
if evt.response.status == "failed":
|
||||||
await self.push_error(error_msg=evt.response.status_details["error"]["message"])
|
await self.push_error(ErrorFrame(error=evt.response.status_details["error"]["message"]))
|
||||||
return
|
return
|
||||||
# response content
|
# response content
|
||||||
for item in evt.response.output:
|
for item in evt.response.output:
|
||||||
@@ -681,7 +678,9 @@ class OpenAIRealtimeLLMService(LLMService):
|
|||||||
# We receive text deltas (as opposed to audio transcript deltas) when
|
# We receive text deltas (as opposed to audio transcript deltas) when
|
||||||
# the output modality is "text"
|
# the output modality is "text"
|
||||||
if evt.delta:
|
if evt.delta:
|
||||||
frame = LLMTextFrame(evt.delta, skip_tts=self._get_skip_tts())
|
frame = LLMTextFrame(evt.delta)
|
||||||
|
# OpenAI Realtime text already includes any necessary inter-chunk spaces
|
||||||
|
frame.includes_inter_frame_spaces = True
|
||||||
await self.push_frame(frame)
|
await self.push_frame(frame)
|
||||||
|
|
||||||
async def _handle_evt_audio_transcript_delta(self, evt):
|
async def _handle_evt_audio_transcript_delta(self, evt):
|
||||||
@@ -763,7 +762,7 @@ class OpenAIRealtimeLLMService(LLMService):
|
|||||||
|
|
||||||
async def _handle_evt_error(self, evt):
|
async def _handle_evt_error(self, evt):
|
||||||
# Errors are fatal to this connection. Send an ErrorFrame.
|
# Errors are fatal to this connection. Send an ErrorFrame.
|
||||||
await self.push_error(error_msg=f"Error: {evt}")
|
await self.push_error(ErrorFrame(error=f"Error: {evt}"))
|
||||||
|
|
||||||
#
|
#
|
||||||
# state and client events for the current conversation
|
# state and client events for the current conversation
|
||||||
@@ -813,9 +812,9 @@ class OpenAIRealtimeLLMService(LLMService):
|
|||||||
# We're done configuring the LLM for this session
|
# We're done configuring the LLM for this session
|
||||||
self._llm_needs_conversation_setup = False
|
self._llm_needs_conversation_setup = False
|
||||||
|
|
||||||
logger.debug("Creating response")
|
logger.debug(f"Creating response")
|
||||||
|
|
||||||
await self.push_frame(LLMFullResponseStartFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseStartFrame())
|
||||||
await self.start_processing_metrics()
|
await self.start_processing_metrics()
|
||||||
await self.start_ttfb_metrics()
|
await self.start_ttfb_metrics()
|
||||||
await self.send_client_event(
|
await self.send_client_event(
|
||||||
|
|||||||
@@ -131,6 +131,15 @@ class OpenAITTSService(TTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that OpenAI TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that OpenAI's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
async def set_model(self, model: str):
|
async def set_model(self, model: str):
|
||||||
"""Set the TTS model to use.
|
"""Set the TTS model to use.
|
||||||
|
|
||||||
@@ -206,4 +215,5 @@ class OpenAITTSService(TTSService):
|
|||||||
yield frame
|
yield frame
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
except BadRequestError as e:
|
except BadRequestError as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.exception(f"{self} error generating TTS: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
|
|||||||
@@ -79,5 +79,5 @@ class AzureRealtimeBetaLLMService(OpenAIRealtimeBetaLLMService):
|
|||||||
)
|
)
|
||||||
self._receive_task = self.create_task(self._receive_task_handler())
|
self._receive_task = self.create_task(self._receive_task_handler())
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error connecting: {e}", exception=e)
|
logger.error(f"{self} initialization error: {e}")
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
|
|||||||
@@ -265,7 +265,7 @@ class OpenAIRealtimeBetaLLMService(LLMService):
|
|||||||
await self._truncate_current_audio_response()
|
await self._truncate_current_audio_response()
|
||||||
await self.stop_all_metrics()
|
await self.stop_all_metrics()
|
||||||
if self._current_assistant_response:
|
if self._current_assistant_response:
|
||||||
await self.push_frame(LLMFullResponseEndFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseEndFrame())
|
||||||
# Only push TTSStoppedFrame if audio modality is enabled
|
# Only push TTSStoppedFrame if audio modality is enabled
|
||||||
if self._is_modality_enabled("audio"):
|
if self._is_modality_enabled("audio"):
|
||||||
await self.push_frame(TTSStoppedFrame())
|
await self.push_frame(TTSStoppedFrame())
|
||||||
@@ -425,7 +425,7 @@ class OpenAIRealtimeBetaLLMService(LLMService):
|
|||||||
)
|
)
|
||||||
self._receive_task = self.create_task(self._receive_task_handler())
|
self._receive_task = self.create_task(self._receive_task_handler())
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error connecting: {e}", exception=e)
|
logger.error(f"{self} initialization error: {e}")
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
|
|
||||||
async def _disconnect(self):
|
async def _disconnect(self):
|
||||||
@@ -441,7 +441,7 @@ class OpenAIRealtimeBetaLLMService(LLMService):
|
|||||||
self._receive_task = None
|
self._receive_task = None
|
||||||
self._disconnecting = False
|
self._disconnecting = False
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error disconnecting: {e}", exception=e)
|
logger.error(f"{self} error disconnecting: {e}")
|
||||||
|
|
||||||
async def _ws_send(self, realtime_message):
|
async def _ws_send(self, realtime_message):
|
||||||
try:
|
try:
|
||||||
@@ -450,11 +450,12 @@ class OpenAIRealtimeBetaLLMService(LLMService):
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
if self._disconnecting:
|
if self._disconnecting:
|
||||||
return
|
return
|
||||||
|
logger.error(f"Error sending message to websocket: {e}")
|
||||||
# In server-to-server contexts, a WebSocket error should be quite rare. Given how hard
|
# In server-to-server contexts, a WebSocket error should be quite rare. Given how hard
|
||||||
# it is to recover from a send-side error with proper state management, and that exponential
|
# it is to recover from a send-side error with proper state management, and that exponential
|
||||||
# backoff for retries can have cost/stability implications for a service cluster, let's just
|
# backoff for retries can have cost/stability implications for a service cluster, let's just
|
||||||
# treat a send-side error as fatal.
|
# treat a send-side error as fatal.
|
||||||
await self.push_error(error_msg=f"Error sending client event: {e}", exception=e)
|
await self.push_error(ErrorFrame(error=f"Error sending client event: {e}"))
|
||||||
|
|
||||||
async def _update_settings(self):
|
async def _update_settings(self):
|
||||||
settings = self._session_properties
|
settings = self._session_properties
|
||||||
@@ -564,7 +565,7 @@ class OpenAIRealtimeBetaLLMService(LLMService):
|
|||||||
self._user_and_response_message_tuple = (evt.item, {"done": False, "output": []})
|
self._user_and_response_message_tuple = (evt.item, {"done": False, "output": []})
|
||||||
elif evt.item.role == "assistant":
|
elif evt.item.role == "assistant":
|
||||||
self._current_assistant_response = evt.item
|
self._current_assistant_response = evt.item
|
||||||
await self.push_frame(LLMFullResponseStartFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseStartFrame())
|
||||||
|
|
||||||
async def _handle_evt_input_audio_transcription_delta(self, evt):
|
async def _handle_evt_input_audio_transcription_delta(self, evt):
|
||||||
if self._send_transcription_frames:
|
if self._send_transcription_frames:
|
||||||
@@ -623,7 +624,7 @@ class OpenAIRealtimeBetaLLMService(LLMService):
|
|||||||
)
|
)
|
||||||
await self.start_llm_usage_metrics(tokens)
|
await self.start_llm_usage_metrics(tokens)
|
||||||
await self.stop_processing_metrics()
|
await self.stop_processing_metrics()
|
||||||
await self.push_frame(LLMFullResponseEndFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseEndFrame())
|
||||||
self._current_assistant_response = None
|
self._current_assistant_response = None
|
||||||
# error handling
|
# error handling
|
||||||
if evt.response.status == "failed":
|
if evt.response.status == "failed":
|
||||||
@@ -647,11 +648,11 @@ class OpenAIRealtimeBetaLLMService(LLMService):
|
|||||||
|
|
||||||
async def _handle_evt_text_delta(self, evt):
|
async def _handle_evt_text_delta(self, evt):
|
||||||
if evt.delta:
|
if evt.delta:
|
||||||
await self.push_frame(LLMTextFrame(evt.delta, skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMTextFrame(evt.delta))
|
||||||
|
|
||||||
async def _handle_evt_audio_transcript_delta(self, evt):
|
async def _handle_evt_audio_transcript_delta(self, evt):
|
||||||
if evt.delta:
|
if evt.delta:
|
||||||
await self.push_frame(LLMTextFrame(evt.delta, skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMTextFrame(evt.delta))
|
||||||
await self.push_frame(TTSTextFrame(evt.delta, aggregated_by=AggregationType.SENTENCE))
|
await self.push_frame(TTSTextFrame(evt.delta, aggregated_by=AggregationType.SENTENCE))
|
||||||
|
|
||||||
async def _handle_evt_speech_started(self, evt):
|
async def _handle_evt_speech_started(self, evt):
|
||||||
@@ -685,7 +686,7 @@ class OpenAIRealtimeBetaLLMService(LLMService):
|
|||||||
|
|
||||||
async def _handle_evt_error(self, evt):
|
async def _handle_evt_error(self, evt):
|
||||||
# Errors are fatal to this connection. Send an ErrorFrame.
|
# Errors are fatal to this connection. Send an ErrorFrame.
|
||||||
await self.push_error(error_msg=f"Error: {evt}")
|
await self.push_error(ErrorFrame(error=f"Error: {evt}"))
|
||||||
|
|
||||||
async def _handle_assistant_output(self, output):
|
async def _handle_assistant_output(self, output):
|
||||||
# We haven't seen intermixed audio and function_call items in the same response. But let's
|
# We haven't seen intermixed audio and function_call items in the same response. But let's
|
||||||
@@ -747,7 +748,7 @@ class OpenAIRealtimeBetaLLMService(LLMService):
|
|||||||
|
|
||||||
logger.debug(f"Creating response: {self._context.get_messages_for_logging()}")
|
logger.debug(f"Creating response: {self._context.get_messages_for_logging()}")
|
||||||
|
|
||||||
await self.push_frame(LLMFullResponseStartFrame(skip_tts=self._get_skip_tts()))
|
await self.push_frame(LLMFullResponseStartFrame())
|
||||||
await self.start_processing_metrics()
|
await self.start_processing_metrics()
|
||||||
await self.start_ttfb_metrics()
|
await self.start_ttfb_metrics()
|
||||||
await self.send_client_event(
|
await self.send_client_event(
|
||||||
|
|||||||
@@ -66,6 +66,15 @@ class PiperTTSService(TTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that Piper TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that Piper's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
@traced_tts
|
@traced_tts
|
||||||
async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]:
|
async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]:
|
||||||
"""Generate speech from text using Piper's HTTP API.
|
"""Generate speech from text using Piper's HTTP API.
|
||||||
@@ -88,6 +97,9 @@ class PiperTTSService(TTSService):
|
|||||||
) as response:
|
) as response:
|
||||||
if response.status != 200:
|
if response.status != 200:
|
||||||
error = await response.text()
|
error = await response.text()
|
||||||
|
logger.error(
|
||||||
|
f"{self} error getting audio (status: {response.status}, error: {error})"
|
||||||
|
)
|
||||||
yield ErrorFrame(
|
yield ErrorFrame(
|
||||||
error=f"Error getting audio (status: {response.status}, error: {error})"
|
error=f"Error getting audio (status: {response.status}, error: {error})"
|
||||||
)
|
)
|
||||||
@@ -106,7 +118,7 @@ class PiperTTSService(TTSService):
|
|||||||
yield frame
|
yield frame
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"{self} exception: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
finally:
|
finally:
|
||||||
logger.debug(f"{self}: Finished TTS [{text}]")
|
logger.debug(f"{self}: Finished TTS [{text}]")
|
||||||
await self.stop_ttfb_metrics()
|
await self.stop_ttfb_metrics()
|
||||||
|
|||||||
@@ -266,7 +266,8 @@ class PlayHTTTSService(InterruptibleTTSService):
|
|||||||
self._websocket = None
|
self._websocket = None
|
||||||
await self._call_event_handler("on_connection_error", f"{e}")
|
await self._call_event_handler("on_connection_error", f"{e}")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error connecting: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
await self._call_event_handler("on_connection_error", f"{e}")
|
await self._call_event_handler("on_connection_error", f"{e}")
|
||||||
|
|
||||||
@@ -279,7 +280,8 @@ class PlayHTTTSService(InterruptibleTTSService):
|
|||||||
logger.debug("Disconnecting from PlayHT")
|
logger.debug("Disconnecting from PlayHT")
|
||||||
await self._websocket.close()
|
await self._websocket.close()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error disconnecting: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
finally:
|
finally:
|
||||||
self._request_id = None
|
self._request_id = None
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
@@ -349,7 +351,8 @@ class PlayHTTTSService(InterruptibleTTSService):
|
|||||||
await self.push_frame(TTSStoppedFrame())
|
await self.push_frame(TTSStoppedFrame())
|
||||||
self._request_id = None
|
self._request_id = None
|
||||||
elif "error" in msg:
|
elif "error" in msg:
|
||||||
await self.push_error(error_msg=f"Error: {msg['error']}")
|
logger.error(f"{self} error: {msg}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {msg['error']}"))
|
||||||
except json.JSONDecodeError:
|
except json.JSONDecodeError:
|
||||||
logger.error(f"Invalid JSON message: {message}")
|
logger.error(f"Invalid JSON message: {message}")
|
||||||
|
|
||||||
@@ -391,7 +394,8 @@ class PlayHTTTSService(InterruptibleTTSService):
|
|||||||
await self._get_websocket().send(json.dumps(tts_command))
|
await self._get_websocket().send(json.dumps(tts_command))
|
||||||
await self.start_tts_usage_metrics(text)
|
await self.start_tts_usage_metrics(text)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
await self._disconnect()
|
await self._disconnect()
|
||||||
await self._connect()
|
await self._connect()
|
||||||
@@ -401,7 +405,8 @@ class PlayHTTTSService(InterruptibleTTSService):
|
|||||||
yield None
|
yield None
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
|
|
||||||
|
|
||||||
class PlayHTHttpTTSService(TTSService):
|
class PlayHTHttpTTSService(TTSService):
|
||||||
@@ -621,7 +626,8 @@ class PlayHTHttpTTSService(TTSService):
|
|||||||
yield frame
|
yield frame
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
finally:
|
finally:
|
||||||
await self.stop_ttfb_metrics()
|
await self.stop_ttfb_metrics()
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
|
|||||||
@@ -300,7 +300,8 @@ class RimeTTSService(AudioContextWordTTSService):
|
|||||||
|
|
||||||
await self._call_event_handler("on_connected")
|
await self._call_event_handler("on_connected")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error connecting: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
await self._call_event_handler("on_connection_error", f"{e}")
|
await self._call_event_handler("on_connection_error", f"{e}")
|
||||||
|
|
||||||
@@ -312,7 +313,8 @@ class RimeTTSService(AudioContextWordTTSService):
|
|||||||
await self._websocket.send(json.dumps(self._build_eos_msg()))
|
await self._websocket.send(json.dumps(self._build_eos_msg()))
|
||||||
await self._websocket.close()
|
await self._websocket.close()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error disconnecting: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
finally:
|
finally:
|
||||||
self._context_id = None
|
self._context_id = None
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
@@ -405,9 +407,10 @@ class RimeTTSService(AudioContextWordTTSService):
|
|||||||
logger.debug(f"Updated cumulative time to: {self._cumulative_time}")
|
logger.debug(f"Updated cumulative time to: {self._cumulative_time}")
|
||||||
|
|
||||||
elif msg["type"] == "error":
|
elif msg["type"] == "error":
|
||||||
|
logger.error(f"{self} error: {msg}")
|
||||||
await self.push_frame(TTSStoppedFrame())
|
await self.push_frame(TTSStoppedFrame())
|
||||||
await self.stop_all_metrics()
|
await self.stop_all_metrics()
|
||||||
await self.push_error(error_msg=f"Error: {msg['message']}")
|
await self.push_error(ErrorFrame(error=f"{self} error: {msg['message']}"))
|
||||||
self._context_id = None
|
self._context_id = None
|
||||||
|
|
||||||
async def push_frame(self, frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM):
|
async def push_frame(self, frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM):
|
||||||
@@ -449,14 +452,16 @@ class RimeTTSService(AudioContextWordTTSService):
|
|||||||
await self._get_websocket().send(json.dumps(msg))
|
await self._get_websocket().send(json.dumps(msg))
|
||||||
await self.start_tts_usage_metrics(text)
|
await self.start_tts_usage_metrics(text)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
await self._disconnect()
|
await self._disconnect()
|
||||||
await self._connect()
|
await self._connect()
|
||||||
return
|
return
|
||||||
yield None
|
yield None
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
|
|
||||||
|
|
||||||
class RimeHttpTTSService(TTSService):
|
class RimeHttpTTSService(TTSService):
|
||||||
@@ -537,6 +542,15 @@ class RimeHttpTTSService(TTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that Rime TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that Rime's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
def language_to_service_language(self, language: Language) -> str | None:
|
def language_to_service_language(self, language: Language) -> str | None:
|
||||||
"""Convert pipecat language to Rime language code.
|
"""Convert pipecat language to Rime language code.
|
||||||
|
|
||||||
@@ -587,6 +601,7 @@ class RimeHttpTTSService(TTSService):
|
|||||||
) as response:
|
) as response:
|
||||||
if response.status != 200:
|
if response.status != 200:
|
||||||
error_message = f"Rime TTS error: HTTP {response.status}"
|
error_message = f"Rime TTS error: HTTP {response.status}"
|
||||||
|
logger.error(error_message)
|
||||||
yield ErrorFrame(error=error_message)
|
yield ErrorFrame(error=error_message)
|
||||||
return
|
return
|
||||||
|
|
||||||
@@ -604,7 +619,8 @@ class RimeHttpTTSService(TTSService):
|
|||||||
yield frame
|
yield frame
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
finally:
|
finally:
|
||||||
await self.stop_ttfb_metrics()
|
await self.stop_ttfb_metrics()
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
|
|||||||
@@ -655,10 +655,12 @@ class RivaSegmentedSTTService(SegmentedSTTService):
|
|||||||
logger.debug("No transcription results found in Riva response")
|
logger.debug("No transcription results found in Riva response")
|
||||||
|
|
||||||
except AttributeError as ae:
|
except AttributeError as ae:
|
||||||
|
logger.error(f"Unexpected response structure from Riva: {ae}")
|
||||||
yield ErrorFrame(f"Unexpected Riva response format: {str(ae)}")
|
yield ErrorFrame(f"Unexpected Riva response format: {str(ae)}")
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
|
|
||||||
|
|
||||||
class ParakeetSTTService(RivaSTTService):
|
class ParakeetSTTService(RivaSTTService):
|
||||||
|
|||||||
@@ -113,6 +113,15 @@ class RivaTTSService(TTSService):
|
|||||||
riva.client.proto.riva_tts_pb2.RivaSynthesisConfigRequest()
|
riva.client.proto.riva_tts_pb2.RivaSynthesisConfigRequest()
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that Riva TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that Riva's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
async def set_model(self, model: str):
|
async def set_model(self, model: str):
|
||||||
"""Attempt to set the TTS model.
|
"""Attempt to set the TTS model.
|
||||||
|
|
||||||
@@ -157,6 +166,7 @@ class RivaTTSService(TTSService):
|
|||||||
add_response(None)
|
add_response(None)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"{self} exception: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
add_response(None)
|
add_response(None)
|
||||||
|
|
||||||
await self.start_ttfb_metrics()
|
await self.start_ttfb_metrics()
|
||||||
@@ -180,7 +190,7 @@ class RivaTTSService(TTSService):
|
|||||||
yield frame
|
yield frame
|
||||||
resp = await asyncio.wait_for(queue.get(), timeout=RIVA_TTS_TIMEOUT_SECS)
|
resp = await asyncio.wait_for(queue.get(), timeout=RIVA_TTS_TIMEOUT_SECS)
|
||||||
except asyncio.TimeoutError:
|
except asyncio.TimeoutError:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} timeout waiting for audio response")
|
||||||
|
|
||||||
await self.start_tts_usage_metrics(text)
|
await self.start_tts_usage_metrics(text)
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
|
|||||||
@@ -14,7 +14,9 @@ from openai import AsyncStream
|
|||||||
from openai.types.chat import ChatCompletionChunk
|
from openai.types.chat import ChatCompletionChunk
|
||||||
|
|
||||||
from pipecat.adapters.services.open_ai_adapter import OpenAILLMInvocationParams
|
from pipecat.adapters.services.open_ai_adapter import OpenAILLMInvocationParams
|
||||||
from pipecat.frames.frames import LLMTextFrame
|
from pipecat.frames.frames import (
|
||||||
|
LLMTextFrame,
|
||||||
|
)
|
||||||
from pipecat.metrics.metrics import LLMTokenUsage
|
from pipecat.metrics.metrics import LLMTokenUsage
|
||||||
from pipecat.processors.aggregators.llm_context import LLMContext
|
from pipecat.processors.aggregators.llm_context import LLMContext
|
||||||
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
|
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
|
||||||
@@ -174,20 +176,16 @@ class SambaNovaLLMService(OpenAILLMService): # type: ignore
|
|||||||
# Keep iterating through the response to collect all the argument fragments
|
# Keep iterating through the response to collect all the argument fragments
|
||||||
arguments += tool_call.function.arguments
|
arguments += tool_call.function.arguments
|
||||||
elif chunk.choices[0].delta.content:
|
elif chunk.choices[0].delta.content:
|
||||||
await self.push_frame(
|
frame = LLMTextFrame(chunk.choices[0].delta.content)
|
||||||
LLMTextFrame(chunk.choices[0].delta.content, skip_tts=self._get_skip_tts())
|
frame.includes_inter_frame_spaces = True
|
||||||
)
|
await self.push_frame(frame)
|
||||||
|
|
||||||
# When gpt-4o-audio / gpt-4o-mini-audio is used for llm or stt+llm
|
# When gpt-4o-audio / gpt-4o-mini-audio is used for llm or stt+llm
|
||||||
# we need to get LLMTextFrame for the transcript
|
# we need to get LLMTextFrame for the transcript
|
||||||
elif hasattr(chunk.choices[0].delta, "audio") and chunk.choices[0].delta.audio.get(
|
elif hasattr(chunk.choices[0].delta, "audio") and chunk.choices[0].delta.audio.get(
|
||||||
"transcript"
|
"transcript"
|
||||||
):
|
):
|
||||||
await self.push_frame(
|
await self.push_frame(LLMTextFrame(chunk.choices[0].delta.audio["transcript"]))
|
||||||
LLMTextFrame(
|
|
||||||
chunk.choices[0].delta.audio["transcript"], skip_tts=self._get_skip_tts()
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
# if we got a function name and arguments, check to see if it's a function with
|
# if we got a function name and arguments, check to see if it's a function with
|
||||||
# a registered handler. If so, run the registered callback, save the result to
|
# a registered handler. If so, run the registered callback, save the result to
|
||||||
|
|||||||
@@ -275,7 +275,8 @@ class SarvamSTTService(STTService):
|
|||||||
await self._socket_client.translate(**method_kwargs)
|
await self._socket_client.translate(**method_kwargs)
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Error sending audio to Sarvam: {e}", exception=e)
|
logger.error(f"Error sending audio to Sarvam: {e}")
|
||||||
|
await self.push_error(ErrorFrame(f"Failed to send audio: {e}"))
|
||||||
|
|
||||||
yield None
|
yield None
|
||||||
|
|
||||||
@@ -331,11 +332,13 @@ class SarvamSTTService(STTService):
|
|||||||
logger.info("Connected to Sarvam successfully")
|
logger.info("Connected to Sarvam successfully")
|
||||||
|
|
||||||
except ApiError as e:
|
except ApiError as e:
|
||||||
await self.push_error(error_msg=f"Sarvam API error: {e}", exception=e)
|
logger.error(f"Sarvam API error: {e}")
|
||||||
|
await self.push_error(ErrorFrame(f"Sarvam API error: {e}"))
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to connect to Sarvam: {e}")
|
||||||
self._socket_client = None
|
self._socket_client = None
|
||||||
self._websocket_context = None
|
self._websocket_context = None
|
||||||
await self.push_error(error_msg=f"Failed to connect to Sarvam: {e}", exception=e)
|
await self.push_error(ErrorFrame(f"Failed to connect to Sarvam: {e}"))
|
||||||
|
|
||||||
async def _disconnect(self):
|
async def _disconnect(self):
|
||||||
"""Disconnect from Sarvam WebSocket API using SDK."""
|
"""Disconnect from Sarvam WebSocket API using SDK."""
|
||||||
@@ -348,9 +351,7 @@ class SarvamSTTService(STTService):
|
|||||||
# Exit the async context manager
|
# Exit the async context manager
|
||||||
await self._websocket_context.__aexit__(None, None, None)
|
await self._websocket_context.__aexit__(None, None, None)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(
|
logger.error(f"Error closing WebSocket connection: {e}")
|
||||||
error_msg=f"Error closing WebSocket connection: {e}", exception=e
|
|
||||||
)
|
|
||||||
finally:
|
finally:
|
||||||
logger.debug("Disconnected from Sarvam WebSocket")
|
logger.debug("Disconnected from Sarvam WebSocket")
|
||||||
self._socket_client = None
|
self._socket_client = None
|
||||||
@@ -370,7 +371,8 @@ class SarvamSTTService(STTService):
|
|||||||
# Messages will be handled via the _message_handler callback
|
# Messages will be handled via the _message_handler callback
|
||||||
await self._socket_client.start_listening()
|
await self._socket_client.start_listening()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Sarvam receive task error: {e}", exception=e)
|
logger.error(f"Error in Sarvam receive task: {e}")
|
||||||
|
await self.push_error(ErrorFrame(f"Sarvam receive task error: {e}"))
|
||||||
|
|
||||||
async def _handle_message(self, message):
|
async def _handle_message(self, message):
|
||||||
"""Handle incoming WebSocket message from Sarvam SDK.
|
"""Handle incoming WebSocket message from Sarvam SDK.
|
||||||
@@ -425,7 +427,8 @@ class SarvamSTTService(STTService):
|
|||||||
await self.stop_processing_metrics()
|
await self.stop_processing_metrics()
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Failed to handle message: {e}", exception=e)
|
logger.error(f"Error handling Sarvam message: {e}")
|
||||||
|
await self.push_error(ErrorFrame(f"Failed to handle message: {e}"))
|
||||||
await self.stop_all_metrics()
|
await self.stop_all_metrics()
|
||||||
|
|
||||||
@traced_stt
|
@traced_stt
|
||||||
|
|||||||
@@ -195,6 +195,15 @@ class SarvamHttpTTSService(TTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that Sarvam TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that Sarvam's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
def language_to_service_language(self, language: Language) -> Optional[str]:
|
def language_to_service_language(self, language: Language) -> Optional[str]:
|
||||||
"""Convert a Language enum to Sarvam AI language format.
|
"""Convert a Language enum to Sarvam AI language format.
|
||||||
|
|
||||||
@@ -254,7 +263,8 @@ class SarvamHttpTTSService(TTSService):
|
|||||||
async with self._session.post(url, json=payload, headers=headers) as response:
|
async with self._session.post(url, json=payload, headers=headers) as response:
|
||||||
if response.status != 200:
|
if response.status != 200:
|
||||||
error_text = await response.text()
|
error_text = await response.text()
|
||||||
yield ErrorFrame(error=f"Sarvam API error: {error_text}")
|
logger.error(f"Sarvam API error: {error_text}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"Sarvam API error: {error_text}"))
|
||||||
return
|
return
|
||||||
|
|
||||||
response_data = await response.json()
|
response_data = await response.json()
|
||||||
@@ -263,7 +273,8 @@ class SarvamHttpTTSService(TTSService):
|
|||||||
|
|
||||||
# Decode base64 audio data
|
# Decode base64 audio data
|
||||||
if "audios" not in response_data or not response_data["audios"]:
|
if "audios" not in response_data or not response_data["audios"]:
|
||||||
yield ErrorFrame(error="No audio data received")
|
logger.error("No audio data received from Sarvam API")
|
||||||
|
await self.push_error(ErrorFrame(error="No audio data received"))
|
||||||
return
|
return
|
||||||
|
|
||||||
# Get the first audio (there should be only one for single text input)
|
# Get the first audio (there should be only one for single text input)
|
||||||
@@ -284,7 +295,8 @@ class SarvamHttpTTSService(TTSService):
|
|||||||
yield frame
|
yield frame
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Error generating TTS: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
finally:
|
finally:
|
||||||
await self.stop_ttfb_metrics()
|
await self.stop_ttfb_metrics()
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
@@ -455,6 +467,15 @@ class SarvamTTSService(InterruptibleTTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that Sarvam TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that Sarvam's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
def language_to_service_language(self, language: Language) -> Optional[str]:
|
def language_to_service_language(self, language: Language) -> Optional[str]:
|
||||||
"""Convert a Language enum to Sarvam AI language format.
|
"""Convert a Language enum to Sarvam AI language format.
|
||||||
|
|
||||||
@@ -557,7 +578,8 @@ class SarvamTTSService(InterruptibleTTSService):
|
|||||||
await self._disconnect_websocket()
|
await self._disconnect_websocket()
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
finally:
|
finally:
|
||||||
# Reset state only after everything is cleaned up
|
# Reset state only after everything is cleaned up
|
||||||
self._started = False
|
self._started = False
|
||||||
@@ -581,9 +603,8 @@ class SarvamTTSService(InterruptibleTTSService):
|
|||||||
|
|
||||||
await self._call_event_handler("on_connected")
|
await self._call_event_handler("on_connected")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(
|
logger.error(f"{self} exception: {e}")
|
||||||
error_msg=f"Error connecting to Sarvam TTS Websocket: {e}", exception=e
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
)
|
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
await self._call_event_handler("on_connection_error", f"{e}")
|
await self._call_event_handler("on_connection_error", f"{e}")
|
||||||
|
|
||||||
@@ -599,7 +620,8 @@ class SarvamTTSService(InterruptibleTTSService):
|
|||||||
await self._websocket.send(json.dumps(config_message))
|
await self._websocket.send(json.dumps(config_message))
|
||||||
logger.debug("Configuration sent successfully")
|
logger.debug("Configuration sent successfully")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
raise
|
raise
|
||||||
|
|
||||||
async def _disconnect_websocket(self):
|
async def _disconnect_websocket(self):
|
||||||
@@ -611,7 +633,8 @@ class SarvamTTSService(InterruptibleTTSService):
|
|||||||
logger.debug("Disconnecting from Sarvam")
|
logger.debug("Disconnecting from Sarvam")
|
||||||
await self._websocket.close()
|
await self._websocket.close()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error closing websocket: {e}", exception=e)
|
logger.error(f"{self} error closing websocket: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
finally:
|
finally:
|
||||||
self._started = False
|
self._started = False
|
||||||
self._websocket = None
|
self._websocket = None
|
||||||
@@ -635,7 +658,7 @@ class SarvamTTSService(InterruptibleTTSService):
|
|||||||
await self.push_frame(frame)
|
await self.push_frame(frame)
|
||||||
elif msg.get("type") == "error":
|
elif msg.get("type") == "error":
|
||||||
error_msg = msg["data"]["message"]
|
error_msg = msg["data"]["message"]
|
||||||
await self.push_error(error_msg=f"TTS Error: {error_msg}")
|
logger.error(f"TTS Error: {error_msg}")
|
||||||
|
|
||||||
# If it's a timeout error, the connection might need to be reset
|
# If it's a timeout error, the connection might need to be reset
|
||||||
if "too long" in error_msg.lower() or "timeout" in error_msg.lower():
|
if "too long" in error_msg.lower() or "timeout" in error_msg.lower():
|
||||||
@@ -697,11 +720,13 @@ class SarvamTTSService(InterruptibleTTSService):
|
|||||||
await self._send_text(text)
|
await self._send_text(text)
|
||||||
await self.start_tts_usage_metrics(text)
|
await self.start_tts_usage_metrics(text)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
await self._disconnect()
|
await self._disconnect()
|
||||||
await self._connect()
|
await self._connect()
|
||||||
return
|
return
|
||||||
yield None
|
yield None
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
|
|||||||
@@ -48,14 +48,12 @@ class SimliVideoService(FrameProcessor):
|
|||||||
"""Input parameters for Simli video configuration.
|
"""Input parameters for Simli video configuration.
|
||||||
|
|
||||||
Parameters:
|
Parameters:
|
||||||
enable_logging: Whether to enable Simli logging.
|
|
||||||
max_session_length: Absolute maximum session duration in seconds.
|
max_session_length: Absolute maximum session duration in seconds.
|
||||||
Avatar will disconnect after this time even if it's speaking.
|
Avatar will disconnect after this time even if it's speaking.
|
||||||
max_idle_time: Maximum duration in seconds the avatar is not speaking
|
max_idle_time: Maximum duration in seconds the avatar is not speaking
|
||||||
before the avatar disconnects.
|
before the avatar disconnects.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
enable_logging: Optional[bool] = None
|
|
||||||
max_session_length: Optional[int] = None
|
max_session_length: Optional[int] = None
|
||||||
max_idle_time: Optional[int] = None
|
max_idle_time: Optional[int] = None
|
||||||
|
|
||||||
@@ -86,10 +84,6 @@ class SimliVideoService(FrameProcessor):
|
|||||||
Please use 'api_key' and 'face_id' parameters instead.
|
Please use 'api_key' and 'face_id' parameters instead.
|
||||||
|
|
||||||
use_turn_server: Whether to use TURN server for connection. Defaults to False.
|
use_turn_server: Whether to use TURN server for connection. Defaults to False.
|
||||||
|
|
||||||
.. deprecated:: 0.0.95
|
|
||||||
The 'use_turn_server' parameter is deprecated and will be removed in a future version.
|
|
||||||
|
|
||||||
latency_interval: Latency interval setting for sending health checks to check
|
latency_interval: Latency interval setting for sending health checks to check
|
||||||
the latency to Simli Servers. Defaults to 0.
|
the latency to Simli Servers. Defaults to 0.
|
||||||
simli_url: URL of the simli servers. Can be changed for custom deployments
|
simli_url: URL of the simli servers. Can be changed for custom deployments
|
||||||
@@ -141,22 +135,15 @@ class SimliVideoService(FrameProcessor):
|
|||||||
|
|
||||||
config = SimliConfig(**config_kwargs)
|
config = SimliConfig(**config_kwargs)
|
||||||
|
|
||||||
if use_turn_server:
|
|
||||||
warnings.warn(
|
|
||||||
"The 'use_turn_server' parameter is deprecated and will be removed in a future version.",
|
|
||||||
DeprecationWarning,
|
|
||||||
stacklevel=2,
|
|
||||||
)
|
|
||||||
|
|
||||||
self._initialized = False
|
self._initialized = False
|
||||||
# Add buffer time to session limits
|
# Add buffer time to session limits
|
||||||
config.maxIdleTime += 5
|
config.maxIdleTime += 5
|
||||||
config.maxSessionLength += 5
|
config.maxSessionLength += 5
|
||||||
self._simli_client = SimliClient(
|
self._simli_client = SimliClient(
|
||||||
config=config,
|
config,
|
||||||
latencyInterval=latency_interval,
|
use_turn_server,
|
||||||
|
latency_interval,
|
||||||
simliURL=simli_url,
|
simliURL=simli_url,
|
||||||
enable_logging=params.enable_logging or False,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
self._pipecat_resampler: AudioResampler = None
|
self._pipecat_resampler: AudioResampler = None
|
||||||
@@ -181,7 +168,7 @@ class SimliVideoService(FrameProcessor):
|
|||||||
self._audio_task = self.create_task(self._consume_and_process_audio())
|
self._audio_task = self.create_task(self._consume_and_process_audio())
|
||||||
self._video_task = self.create_task(self._consume_and_process_video())
|
self._video_task = self.create_task(self._consume_and_process_video())
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unable to start connection: {e}", exception=e)
|
logger.error(f"{self}: unable to start connection: {e}")
|
||||||
|
|
||||||
async def _consume_and_process_audio(self):
|
async def _consume_and_process_audio(self):
|
||||||
"""Consume audio frames from Simli and push them downstream."""
|
"""Consume audio frames from Simli and push them downstream."""
|
||||||
@@ -259,7 +246,7 @@ class SimliVideoService(FrameProcessor):
|
|||||||
await self._simli_client.send(audioBytes)
|
await self._simli_client.send(audioBytes)
|
||||||
return
|
return
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error sending audio: {e}", exception=e)
|
logger.exception(f"{self} exception: {e}")
|
||||||
elif isinstance(frame, TTSStoppedFrame):
|
elif isinstance(frame, TTSStoppedFrame):
|
||||||
try:
|
try:
|
||||||
if self._previously_interrupted and len(self._audio_buffer) > 0:
|
if self._previously_interrupted and len(self._audio_buffer) > 0:
|
||||||
@@ -267,7 +254,7 @@ class SimliVideoService(FrameProcessor):
|
|||||||
self._previously_interrupted = False
|
self._previously_interrupted = False
|
||||||
self._audio_buffer = bytearray()
|
self._audio_buffer = bytearray()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error stopping TTS: {e}", exception=e)
|
logger.exception(f"{self} exception: {e}")
|
||||||
return
|
return
|
||||||
elif isinstance(frame, (EndFrame, CancelFrame)):
|
elif isinstance(frame, (EndFrame, CancelFrame)):
|
||||||
await self._stop()
|
await self._stop()
|
||||||
|
|||||||
@@ -194,7 +194,7 @@ class SonioxSTTService(STTService):
|
|||||||
self._websocket = await websocket_connect(self._url)
|
self._websocket = await websocket_connect(self._url)
|
||||||
|
|
||||||
if not self._websocket:
|
if not self._websocket:
|
||||||
await self.push_error(error_msg=f"Unable to connect to Soniox API at {self._url}")
|
logger.error(f"Unable to connect to Soniox API at {self._url}")
|
||||||
|
|
||||||
# If vad_force_turn_endpoint is not enabled, we need to enable endpoint detection.
|
# If vad_force_turn_endpoint is not enabled, we need to enable endpoint detection.
|
||||||
# Either one or the other is required.
|
# Either one or the other is required.
|
||||||
@@ -327,7 +327,8 @@ class SonioxSTTService(STTService):
|
|||||||
# Expected when closing the connection
|
# Expected when closing the connection
|
||||||
logger.debug("WebSocket connection closed, keepalive task stopped.")
|
logger.debug("WebSocket connection closed, keepalive task stopped.")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
|
|
||||||
async def _receive_task_handler(self):
|
async def _receive_task_handler(self):
|
||||||
if not self._websocket:
|
if not self._websocket:
|
||||||
@@ -403,8 +404,13 @@ class SonioxSTTService(STTService):
|
|||||||
if error_code or error_message:
|
if error_code or error_message:
|
||||||
# In case of error, still send the final transcript (if any remaining in the buffer).
|
# In case of error, still send the final transcript (if any remaining in the buffer).
|
||||||
await send_endpoint_transcript()
|
await send_endpoint_transcript()
|
||||||
|
logger.error(
|
||||||
|
f"{self} error: {error_code} (_receive_task_handler) - {error_message}"
|
||||||
|
)
|
||||||
await self.push_error(
|
await self.push_error(
|
||||||
error_msg=f"Error: {error_code} (_receive_task_handler) - {error_message}"
|
ErrorFrame(
|
||||||
|
error=f"{self} error: {error_code} (_receive_task_handler) - {error_message}"
|
||||||
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
finished = content.get("finished")
|
finished = content.get("finished")
|
||||||
@@ -419,4 +425,5 @@ class SonioxSTTService(STTService):
|
|||||||
# Expected when closing the connection.
|
# Expected when closing the connection.
|
||||||
pass
|
pass
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error receiving message: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
|
|||||||
@@ -467,7 +467,8 @@ class SpeechmaticsSTTService(STTService):
|
|||||||
await self._client.send_audio(audio)
|
await self._client.send_audio(audio)
|
||||||
yield None
|
yield None
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
await self._disconnect()
|
await self._disconnect()
|
||||||
|
|
||||||
def update_params(
|
def update_params(
|
||||||
@@ -513,7 +514,8 @@ class SpeechmaticsSTTService(STTService):
|
|||||||
self._client.send_message(payload), self.get_event_loop()
|
self._client.send_message(payload), self.get_event_loop()
|
||||||
)
|
)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
raise RuntimeError(f"error sending message to STT: {e}")
|
raise RuntimeError(f"error sending message to STT: {e}")
|
||||||
|
|
||||||
async def _connect(self) -> None:
|
async def _connect(self) -> None:
|
||||||
@@ -579,7 +581,8 @@ class SpeechmaticsSTTService(STTService):
|
|||||||
logger.debug(f"{self} Connected to Speechmatics STT service")
|
logger.debug(f"{self} Connected to Speechmatics STT service")
|
||||||
await self._call_event_handler("on_connected")
|
await self._call_event_handler("on_connected")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Error connecting to Speechmatics: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
self._client = None
|
self._client = None
|
||||||
|
|
||||||
async def _disconnect(self) -> None:
|
async def _disconnect(self) -> None:
|
||||||
@@ -593,9 +596,8 @@ class SpeechmaticsSTTService(STTService):
|
|||||||
except asyncio.TimeoutError:
|
except asyncio.TimeoutError:
|
||||||
logger.warning(f"{self} Timeout while closing Speechmatics client connection")
|
logger.warning(f"{self} Timeout while closing Speechmatics client connection")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(
|
logger.error(f"{self} exception: {e}")
|
||||||
error_msg=f"Error disconnecting from Speechmatics: {e}", exception=e
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
)
|
|
||||||
finally:
|
finally:
|
||||||
self._client = None
|
self._client = None
|
||||||
await self._call_event_handler("on_disconnected")
|
await self._call_event_handler("on_disconnected")
|
||||||
|
|||||||
@@ -105,6 +105,15 @@ class SpeechmaticsTTSService(TTSService):
|
|||||||
"""
|
"""
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates that Speechmatics TTSTextFrames include necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True, indicating that Speechmatics's text frames include necessary inter-frame spaces.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
@traced_tts
|
@traced_tts
|
||||||
async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]:
|
async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]:
|
||||||
"""Generate speech from text using Speechmatics' HTTP API.
|
"""Generate speech from text using Speechmatics' HTTP API.
|
||||||
@@ -163,7 +172,7 @@ class SpeechmaticsTTSService(TTSService):
|
|||||||
|
|
||||||
# Report error frame
|
# Report error frame
|
||||||
yield ErrorFrame(
|
yield ErrorFrame(
|
||||||
error=f"Service unavailable [503] (attempt {attempt}, retry in {backoff_time:.2f}s)"
|
error=f"{self} Service unavailable [503] (attempt {attempt}, retry in {backoff_time:.2f}s)"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Wait before retrying
|
# Wait before retrying
|
||||||
@@ -174,13 +183,16 @@ class SpeechmaticsTTSService(TTSService):
|
|||||||
|
|
||||||
except (ValueError, ArithmeticError):
|
except (ValueError, ArithmeticError):
|
||||||
yield ErrorFrame(
|
yield ErrorFrame(
|
||||||
error=f"Service unavailable [503] (attempts {attempt})",
|
error=f"{self} Service unavailable [503] (attempts {attempt})",
|
||||||
|
fatal=True,
|
||||||
)
|
)
|
||||||
return
|
return
|
||||||
|
|
||||||
# != 200 : Error
|
# != 200 : Error
|
||||||
if response.status != 200:
|
if response.status != 200:
|
||||||
yield ErrorFrame(error=f"Service unavailable [{response.status}]")
|
yield ErrorFrame(
|
||||||
|
error=f"{self} Service unavailable [{response.status}]", fatal=True
|
||||||
|
)
|
||||||
return
|
return
|
||||||
|
|
||||||
# Update Pipecat metrics
|
# Update Pipecat metrics
|
||||||
@@ -222,7 +234,7 @@ class SpeechmaticsTTSService(TTSService):
|
|||||||
break
|
break
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Error generating TTS: {e}")
|
yield ErrorFrame(error=f"{self}: Error generating TTS: {e}", fatal=True)
|
||||||
finally:
|
finally:
|
||||||
# Emit the TTS stopped frame
|
# Emit the TTS stopped frame
|
||||||
yield TTSStoppedFrame()
|
yield TTSStoppedFrame()
|
||||||
|
|||||||
@@ -38,7 +38,7 @@ class STTService(AIService):
|
|||||||
|
|
||||||
Event handlers:
|
Event handlers:
|
||||||
on_connected: Called when connected to the STT service.
|
on_connected: Called when connected to the STT service.
|
||||||
on_disconnected: Called when disconnected from the STT service.
|
on_connected: Called when disconnected from the STT service.
|
||||||
on_connection_error: Called when a connection to the STT service error occurs.
|
on_connection_error: Called when a connection to the STT service error occurs.
|
||||||
|
|
||||||
Example::
|
Example::
|
||||||
@@ -329,4 +329,4 @@ class WebsocketSTTService(STTService, WebsocketService):
|
|||||||
|
|
||||||
async def _report_error(self, error: ErrorFrame):
|
async def _report_error(self, error: ErrorFrame):
|
||||||
await self._call_event_handler("on_connection_error", error.error)
|
await self._call_event_handler("on_connection_error", error.error)
|
||||||
await self.push_error_frame(error)
|
await self.push_error(error)
|
||||||
|
|||||||
@@ -231,6 +231,23 @@ class TTSService(AIService):
|
|||||||
CHUNK_SECONDS = 0.5
|
CHUNK_SECONDS = 0.5
|
||||||
return int(self.sample_rate * CHUNK_SECONDS * 2) # 2 bytes/sample
|
return int(self.sample_rate * CHUNK_SECONDS * 2) # 2 bytes/sample
|
||||||
|
|
||||||
|
@property
|
||||||
|
def includes_inter_frame_spaces(self) -> bool:
|
||||||
|
"""Indicates whether TTSTextFrames include necesary inter-frame spaces.
|
||||||
|
|
||||||
|
When True, the TTSTextFrame objects pushed by this service already
|
||||||
|
include all necessary spaces between subsequent frames. When False,
|
||||||
|
downstream processors (like the assistant context aggregator) may need
|
||||||
|
to add spacing.
|
||||||
|
|
||||||
|
Subclasses should override this property to return True if their text
|
||||||
|
generation process already includes necessary inter-frame spaces.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
False by default. Subclasses can override to return True.
|
||||||
|
"""
|
||||||
|
return False
|
||||||
|
|
||||||
async def set_model(self, model: str):
|
async def set_model(self, model: str):
|
||||||
"""Set the TTS model to use.
|
"""Set the TTS model to use.
|
||||||
|
|
||||||
@@ -425,16 +442,10 @@ class TTSService(AIService):
|
|||||||
# pause to avoid audio overlapping.
|
# pause to avoid audio overlapping.
|
||||||
await self._maybe_pause_frame_processing()
|
await self._maybe_pause_frame_processing()
|
||||||
|
|
||||||
pending_aggregation = self._text_aggregator.text
|
aggregate = self._text_aggregator.text
|
||||||
|
|
||||||
# Reset aggregator state
|
|
||||||
await self._text_aggregator.reset()
|
await self._text_aggregator.reset()
|
||||||
self._processing_text = False
|
self._processing_text = False
|
||||||
|
await self._push_tts_frames(AggregatedTextFrame(aggregate.text, aggregate.type))
|
||||||
if pending_aggregation.text:
|
|
||||||
await self._push_tts_frames(
|
|
||||||
AggregatedTextFrame(pending_aggregation.text, pending_aggregation.type)
|
|
||||||
)
|
|
||||||
if isinstance(frame, LLMFullResponseEndFrame):
|
if isinstance(frame, LLMFullResponseEndFrame):
|
||||||
if self._push_text_frames:
|
if self._push_text_frames:
|
||||||
await self.push_frame(frame, direction)
|
await self.push_frame(frame, direction)
|
||||||
@@ -443,7 +454,6 @@ class TTSService(AIService):
|
|||||||
elif isinstance(frame, TTSSpeakFrame):
|
elif isinstance(frame, TTSSpeakFrame):
|
||||||
# Store if we were processing text or not so we can set it back.
|
# Store if we were processing text or not so we can set it back.
|
||||||
processing_text = self._processing_text
|
processing_text = self._processing_text
|
||||||
# Assumption: text in TTSSpeakFrame does not include inter-frame spaces
|
|
||||||
await self._push_tts_frames(AggregatedTextFrame(frame.text, AggregationType.SENTENCE))
|
await self._push_tts_frames(AggregatedTextFrame(frame.text, AggregationType.SENTENCE))
|
||||||
# We pause processing incoming frames because we are sending data to
|
# We pause processing incoming frames because we are sending data to
|
||||||
# the TTS. We pause to avoid audio overlapping.
|
# the TTS. We pause to avoid audio overlapping.
|
||||||
@@ -534,10 +544,8 @@ class TTSService(AIService):
|
|||||||
|
|
||||||
async def _process_text_frame(self, frame: TextFrame):
|
async def _process_text_frame(self, frame: TextFrame):
|
||||||
text: Optional[str] = None
|
text: Optional[str] = None
|
||||||
includes_inter_frame_spaces: bool = False
|
|
||||||
if not self._aggregate_sentences:
|
if not self._aggregate_sentences:
|
||||||
text = frame.text
|
text = frame.text
|
||||||
includes_inter_frame_spaces = frame.includes_inter_frame_spaces
|
|
||||||
aggregated_by = "token"
|
aggregated_by = "token"
|
||||||
else:
|
else:
|
||||||
aggregate = await self._text_aggregator.aggregate(frame.text)
|
aggregate = await self._text_aggregator.aggregate(frame.text)
|
||||||
@@ -547,18 +555,11 @@ class TTSService(AIService):
|
|||||||
|
|
||||||
if text:
|
if text:
|
||||||
logger.trace(f"Pushing TTS frames for text: {text}, {aggregated_by}")
|
logger.trace(f"Pushing TTS frames for text: {text}, {aggregated_by}")
|
||||||
await self._push_tts_frames(
|
await self._push_tts_frames(AggregatedTextFrame(text, aggregated_by))
|
||||||
AggregatedTextFrame(text, aggregated_by), includes_inter_frame_spaces
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _push_tts_frames(
|
async def _push_tts_frames(self, src_frame: AggregatedTextFrame):
|
||||||
self, src_frame: AggregatedTextFrame, includes_inter_frame_spaces: Optional[bool] = False
|
|
||||||
):
|
|
||||||
type = src_frame.aggregated_by
|
type = src_frame.aggregated_by
|
||||||
text = src_frame.text
|
text = src_frame.text
|
||||||
|
|
||||||
# Skip sending to TTS if the aggregation type is in the skip list. Simply
|
|
||||||
# push the original frame downstream.
|
|
||||||
if type in self._skip_aggregator_types:
|
if type in self._skip_aggregator_types:
|
||||||
await self.push_frame(src_frame)
|
await self.push_frame(src_frame)
|
||||||
return
|
return
|
||||||
@@ -578,7 +579,7 @@ class TTSService(AIService):
|
|||||||
|
|
||||||
await self.start_processing_metrics()
|
await self.start_processing_metrics()
|
||||||
|
|
||||||
# Process all filters.
|
# Process all filter.
|
||||||
for filter in self._text_filters:
|
for filter in self._text_filters:
|
||||||
await filter.reset_interruption()
|
await filter.reset_interruption()
|
||||||
text = await filter.filter(text)
|
text = await filter.filter(text)
|
||||||
@@ -616,7 +617,7 @@ class TTSService(AIService):
|
|||||||
# context and the context that IS added does not include TTS-specific tags
|
# context and the context that IS added does not include TTS-specific tags
|
||||||
# or transformations.
|
# or transformations.
|
||||||
frame = TTSTextFrame(text, aggregated_by=type)
|
frame = TTSTextFrame(text, aggregated_by=type)
|
||||||
frame.includes_inter_frame_spaces = includes_inter_frame_spaces
|
frame.includes_inter_frame_spaces = self.includes_inter_frame_spaces
|
||||||
await self.push_frame(frame)
|
await self.push_frame(frame)
|
||||||
|
|
||||||
async def _stop_frame_handler(self):
|
async def _stop_frame_handler(self):
|
||||||
@@ -743,8 +744,6 @@ class WordTTSService(TTSService):
|
|||||||
frame = TTSStoppedFrame()
|
frame = TTSStoppedFrame()
|
||||||
frame.pts = last_pts
|
frame.pts = last_pts
|
||||||
else:
|
else:
|
||||||
# Assumption: word-by-word text frames don't include spaces, so
|
|
||||||
# we can rely on the default includes_inter_frame_spaces=False
|
|
||||||
frame = TTSTextFrame(word, aggregated_by=AggregationType.WORD)
|
frame = TTSTextFrame(word, aggregated_by=AggregationType.WORD)
|
||||||
frame.pts = self._initial_word_timestamp + timestamp
|
frame.pts = self._initial_word_timestamp + timestamp
|
||||||
if frame:
|
if frame:
|
||||||
@@ -781,7 +780,7 @@ class WebsocketTTSService(TTSService, WebsocketService):
|
|||||||
|
|
||||||
async def _report_error(self, error: ErrorFrame):
|
async def _report_error(self, error: ErrorFrame):
|
||||||
await self._call_event_handler("on_connection_error", error.error)
|
await self._call_event_handler("on_connection_error", error.error)
|
||||||
await self.push_error_frame(error)
|
await self.push_error(error)
|
||||||
|
|
||||||
|
|
||||||
class InterruptibleTTSService(WebsocketTTSService):
|
class InterruptibleTTSService(WebsocketTTSService):
|
||||||
@@ -843,7 +842,7 @@ class WebsocketWordTTSService(WordTTSService, WebsocketService):
|
|||||||
|
|
||||||
async def _report_error(self, error: ErrorFrame):
|
async def _report_error(self, error: ErrorFrame):
|
||||||
await self._call_event_handler("on_connection_error", error.error)
|
await self._call_event_handler("on_connection_error", error.error)
|
||||||
await self.push_error_frame(error)
|
await self.push_error(error)
|
||||||
|
|
||||||
|
|
||||||
class InterruptibleWordTTSService(WebsocketWordTTSService):
|
class InterruptibleWordTTSService(WebsocketWordTTSService):
|
||||||
|
|||||||
@@ -246,7 +246,8 @@ class UltravoxSTTService(AIService):
|
|||||||
|
|
||||||
logger.info("Model warm-up completed successfully")
|
logger.info("Model warm-up completed successfully")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
|
logger.error(f"{self} exception: {e}")
|
||||||
|
await self.push_error(ErrorFrame(error=f"{self} error: {e}"))
|
||||||
|
|
||||||
def _generate_silent_audio(self, sample_rate=16000, duration_sec=1.0):
|
def _generate_silent_audio(self, sample_rate=16000, duration_sec=1.0):
|
||||||
"""Generate silent audio as a numpy array.
|
"""Generate silent audio as a numpy array.
|
||||||
@@ -376,7 +377,7 @@ class UltravoxSTTService(AIService):
|
|||||||
if arr.size > 0: # Check if array is not empty
|
if arr.size > 0: # Check if array is not empty
|
||||||
audio_arrays.append(arr)
|
audio_arrays.append(arr)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
# Handle numpy array data
|
# Handle numpy array data
|
||||||
elif isinstance(f.audio, np.ndarray):
|
elif isinstance(f.audio, np.ndarray):
|
||||||
if f.audio.size > 0: # Check if array is not empty
|
if f.audio.size > 0: # Check if array is not empty
|
||||||
@@ -436,11 +437,17 @@ class UltravoxSTTService(AIService):
|
|||||||
yield LLMFullResponseEndFrame()
|
yield LLMFullResponseEndFrame()
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
yield ErrorFrame(error=f"Unknown error occurred: {e}")
|
logger.error(f"{self} exception: {e}")
|
||||||
|
yield ErrorFrame(error=f"{self} error: {e}")
|
||||||
else:
|
else:
|
||||||
|
logger.error("No model available for text generation")
|
||||||
yield ErrorFrame("No model available for text generation")
|
yield ErrorFrame("No model available for text generation")
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
|
logger.error(f"{self} exception: {e}")
|
||||||
|
import traceback
|
||||||
|
|
||||||
|
logger.error(traceback.format_exc())
|
||||||
yield ErrorFrame(f"Error processing audio: {str(e)}")
|
yield ErrorFrame(f"Error processing audio: {str(e)}")
|
||||||
finally:
|
finally:
|
||||||
self._buffer.is_processing = False
|
self._buffer.is_processing = False
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user